eweekchat

Trends in Data Orchestration
JOIN US: This is a chat-based conversation about how batch processing, while still being used pretty extensively, is nonetheless being eclipsed by newer methods of making data work inside IT systems.
   4 years ago
#eweekchatNext-Gen Security TrendsJOIN US: This is a chat-based conversation about what SecOps and DataOps admins are looking at to fortify their systems following the huge Solarwinds and FireEye data breaches.
   4 years ago
#eweekchatNext-Gen Health Care ITJOIN US: This is a chat-based conversation about advancements in next-generation health-care IT, including new collaboration tools, AI and telemedicine.
David Wilmer
Q6: Kubernetes is very important in the evolution of data orchestration as organizations make their move to Cloud and containerization and platform agnostisity. (I think I just invented a new word!)
David Wilmer
Talend saw the value of Kubernetes very early on as the leader of containerized orchestration and now with Spark on Kubernetes we are experiencing the next evolution in advanced analytics.
Chris Preimesberger
Great! "Agnostisity" is indeed a new word; "agnosticity" is in the dictionary!
David Wilmer
@editingwhiz it's all about the spelling ;)
mbnewburgh
“We are the music makers, we are the dreamers of dreams”
David Wilmer
A5: I think it starts with the continued maturing of DataOps and the advancement of ML and AI into the orchestration process. And of course leverage (Talend's) data quality capabilities available through the integration processes.
David Wilmer
However, I would argue that more important than your orchestration tools would be your integration and data quality tools! (Shameless plug).
David Wilmer
You can automate and orchestrate the crap out of your processes, but if the data is poor quality its still going to be garbage in, garbage out – just like MDM systems from 15 years ago.
Chris Preimesberger
And, of course, data quality is a necessary constant throughout all these processes--otherwise bad stuff happens!
David Wilmer
A1: Talend is an Orchestration Enabler. Talend provides all the necessary components for proper data orchestration from scheduling and monitoring, to error handling and reporting. We are also seeing the importance of data orchestration as Microservices become more prevalent.
Chris Preimesberger
Welcome, David! Glad you're with us today.
David Wilmer
Glad for the opportunity!
Chris Preimesberger
Sidebar: Why are microservices important in new-gen orchestration?
David Wilmer
I think it gets organizations more modularized.. One microservice goes offline doesn't shut down the whole system. In proper orchestration, process can be re-routed until down system is back online.
David Wilmer
A2: The obvious advantage is minimize downtime. Smart orchestration can self correct or at least re-direct so one down process doesn't shut down the whole system. Future state, bring AI into the mix and orchestration could potentially predict failures.
Chris Preimesberger
Sidebar: Are you seeing a trend now toward this AI input?
David Wilmer
AI is trending across all of IT. Some orchestration solutions are already incorporating Machine Learning into their orchestration. AI would be the next logical step.
David Wilmer
AI is trending across all of IT. Some orchestration solutions are already bringing Machine Learning into the orchestration process. AI is the next logical progression.
David Wilmer
A4: Observability has to be a major part. just like security, automation, data quality, etc... They are all pieces to a larger puzzle. The question is, who is on the other end of the observability capabilities? If no one is watching or being notified, orchestration has failed.
David Wilmer
Put some AI on the other end of observability and actions can be taken immediately.
Chris Preimesberger
Efficient observability is in the eye of the beholder!
David Wilmer
This has been fun! Thanks @editingwhiz for letting me share some thoughts!!
David Wilmer
...and create new words in the process...
Nong Li
A3: Data orchestration and data security need to be designed hand in hand. Data orchestration helps us get data into different systems to do what they do best but those systems may not have the same security requirements.
Nong Li
The systems can sit in different parts of the network, accessible to different users and with different security capabilities. It's obviously critical to make sure the data is secure, no matter where it ends up.
Nong Li
One of the patterns we've seen again and again is that whatever the security management solution is, it needs to support dynamic access control. It's impossible to predict what kind of data will end up where and that naturally evolves.
Nong Li
Similarly, it's not great to restrict users from end systems. What we've seen work really well is to think about the data security aspects as a fundamental requirement for data orchestration systems and plan for it at the start!
Nong Li
A1: Hi, happy to be here! We help our users make the data in their platform more accessible, and the data orchestration processes are a critical requirement and capability of their platforms.
Nong Li
A lot of the value all of us are getting from the data investments is to be able to operationalize it and to make decisions on it continuously. We need to do this as more data, applications and use cases emerge.
Nong Li
Our focus at Okera is to make it easy for applications and data flows to access data with proper controls. We've seen getting the access capabilities right to really speed up all the data processes.
Chris Preimesberger
Welcome Nong! Talk about a last-minute addition! Thanks for joining us!
Chris Preimesberger
We've about 5 mins left, how about some final thoughts and perhaps some takeaways you got from this session?
mbnewburgh
personally, I think the new word of the day is "Agnostisity"
David Wilmer
@mbreingold I whole-heartedly agree!
Nong Li
A5: Deepen the integration with the enterprise data platform. Data orchestration can help with many parts of the data lifecycle and help connect them together. This includes security, cataloging, and other components in a typical platform.
Nong Li
Aside from the data itself, the metadata integration from the data orchestration processes can be very powerful.