JOIN US: This is a chat-based conversation about how batch processing, while still being used pretty extensively, is nonetheless being eclipsed by newer methods of making data work inside IT systems.
Q6: Kubernetes is very important in the evolution of data orchestration as organizations make their move to Cloud and containerization and platform agnostisity. (I think I just invented a new word!)
Talend saw the value of Kubernetes very early on as the leader of containerized orchestration and now with Spark on Kubernetes we are experiencing the next evolution in advanced analytics.
A5: I think it starts with the continued maturing of DataOps and the advancement of ML and AI into the orchestration process. And of course leverage (Talend's) data quality capabilities available through the integration processes.
You can automate and orchestrate the crap out of your processes, but if the data is poor quality its still going to be garbage in, garbage out – just like MDM systems from 15 years ago.
A1: Talend is an Orchestration Enabler. Talend provides all the necessary components for proper data orchestration from scheduling and monitoring, to error handling and reporting. We are also seeing the importance of data orchestration as Microservices become more prevalent.
I think it gets organizations more modularized.. One microservice goes offline doesn't shut down the whole system. In proper orchestration, process can be re-routed until down system is back online.
A2: The obvious advantage is minimize downtime. Smart orchestration can self correct or at least re-direct so one down process doesn't shut down the whole system. Future state, bring AI into the mix and orchestration could potentially predict failures.
AI is trending across all of IT. Some orchestration solutions are already incorporating Machine Learning into their orchestration. AI would be the next logical step.
AI is trending across all of IT. Some orchestration solutions are already bringing Machine Learning into the orchestration process. AI is the next logical progression.
A4: Observability has to be a major part. just like security, automation, data quality, etc... They are all pieces to a larger puzzle. The question is, who is on the other end of the observability capabilities? If no one is watching or being notified, orchestration has failed.
A3: Data orchestration and data security need to be designed hand in hand. Data orchestration helps us get data into different systems to do what they do best but those systems may not have the same security requirements.
The systems can sit in different parts of the network, accessible to different users and with different security capabilities. It's obviously critical to make sure the data is secure, no matter where it ends up.
One of the patterns we've seen again and again is that whatever the security management solution is, it needs to support dynamic access control. It's impossible to predict what kind of data will end up where and that naturally evolves.
Similarly, it's not great to restrict users from end systems. What we've seen work really well is to think about the data security aspects as a fundamental requirement for data orchestration systems and plan for it at the start!
A1: Hi, happy to be here! We help our users make the data in their platform more accessible, and the data orchestration processes are a critical requirement and capability of their platforms.
A lot of the value all of us are getting from the data investments is to be able to operationalize it and to make decisions on it continuously. We need to do this as more data, applications and use cases emerge.
Our focus at Okera is to make it easy for applications and data flows to access data with proper controls. We've seen getting the access capabilities right to really speed up all the data processes.
A5: Deepen the integration with the enterprise data platform. Data orchestration can help with many parts of the data lifecycle and help connect them together. This includes security, cataloging, and other components in a typical platform.
