eweekchat

DataOps and Data Management
JOIN US: Discuss trends in DataOps and Data Management.
   3 years ago
#eweekchatData Storage and ProtectionJOIN US: Discuss trends in Data Storage, Protection and Privacy.
   3 years ago
#eweekchatNew Tech to Expect for 2022JOIN US: Discuss trends in new technology in the year head.
James Maguire
Q5. Apart from the challenges listed above, is DataOps’s greatest challenge human or technological?
Radhika Krishnan
A5. It’s both and necessarily so. Challenge is how we effectively #collaborate in a large organization, how we create processes to add end-to-end visibility, what tech do we use for scalable #DataManagement. #UX #AI
Bas Kamphuis
A5: It’s a combination of both: technology hasn't quite advanced to do it all, and human capital is not infinite in its capacity. And so, between those two, the challenges are probably about equal. The answer is automation.
Sam Lakkundi
A5. Human element is DataOps’ greatest challenge. We need a different approach to DataOps where self-service and low-code/no-code data pipelines are a fundamental principle. Only practical solution is democratization of data integration enabled through automation
Chris Ehrlich
A5: In terms of DataOps’ greatest challenge being human or technical, it is human-related. DataOps must compete as another sub-discipline for internal budgets and tech talent and a field for pros to specialize in over other more mature data fields. #DataOps
Bruce Kornfeld
A5. Technological. It's just not easy to create work-flows for ingesting, processing, moving, and storing data for all customers' use cases - but the ones that solve this will have true market disruption.
Cognite
A5: The greatest challenge with all new technology remains the product maturity until that is overcome. Human, organisational, cultural, etc. topics are just more trendy to discuss. For example, all talk on the "cultural" issues with smartphones ended after iPhone @JamesMaguire
James Maguire
@RKs2cents I agree! Might have been a trick question :)
Sam Lakkundi
@samlakkundi A5. At our BMC Innovation Labs, we believe while technology challenges can be difficult to overcome, some of the human barriers around their DataOps initiatives are even more pressing. #BMCInnovationLabs
Cognite
@brucekornfeld Truth spoken. And not necessary to solve universally either, within specific verticals is a great place to start - and is in general the mega trend across enterprise software
Radhika Krishnan
@samlakkundi We agree with democratization point along with enablers of #DataGovernance at scale
James Maguire
Q4. What’s DataOps’s greatest challenge: Cohesion between the teams? Process efficiency? Diversity of technologies? Or...?
Radhika Krishnan
A4. The reality is 87% of #DataScience projects never make it into production and the majority of #DataLakes have become unmanaged and ungoverned. See the report: https://htchivantara.is/3DtKLXc. #DataOps provides better practices to bring back value to data.
https://htchivantara.is/3DtKLXc
451 Research: DataOps Unlocks the Value of Data
451 Research: DataOps Unlocks the Value of Data
A recent survey from 451 Research revealed that 100% of respondents are currently planning or actively pursuing initiatives to deliver more agile and automated data management to manage vast new data flows. The method for unlocking value from the vas...
Chris Ehrlich
A4: DataOps’ greatest challenge is leadership buy-in for an organizational commitment to specialized DataOps functions — and not pushing the discipline as an add-on function on data professionals with other core priorities. #DataOps #DataManagement
Sam Lakkundi
A4. Effective communication among key stakeholders, yet poor coordination can make building, deploying, and maintaining data pipelines more difficult than needed. Many #DataOps teams don't understand how data pipeline works.
Cognite
A4: Technology immaturity combined with very real change management is always a challenging combo... as much as many love talking about culture being the greatest challenge to all new things, genuine fit-for-purpose technology platform (SaaS) availability is alike @JamesMaguire
Radhika Krishnan
@Chris_Ehrlich Totally agree on the leadership buy-in. That's crucial to sustain the transformation required.
Bruce Kornfeld
A4. Generic vs. Specific. It's hard to build a data solution that works across all industries - it'll be too generic. But, building a solution for a particular industry is so narrow that it won't get widespread traction. I wonder if this is slowing innovation???
Cognite
A4: There's just something magical that happens when the right user is given access to the right product that actually makes their life meaningfully easier - it takes care of a lot by itself, incl. change management and plenty of executive leadership support need @JamesMaguire
Sam Lakkundi
@Chris_Ehrlich Couldn't agree more on leadership buy-in.
Cognite
@brucekornfeld There will be vertical specialization for sure within DataOps Platforms, simply as the needs in different markets are so very different, as is the data alike
Bas Kamphuis
A4: I think there are 3 major challenges: 1: Data silos, 2: proprietary technology stacks; and 3: the state of automation.
Bas Kamphuis
A4: Lastly the lack of intelligence within BI is a scaling challenge: there is not a whole lot of self-learning algorithms that are helping you interpret data at the BI layer of the architectural stack.
James Maguire
Q2. Do you think that DataOps is a mainstream approach in today’s enterprise?
Bruce Kornfeld
A2. Absolutely not! It's a great theory and there are some vendors that are talking about it, but we are a long way from mainstream.
Sam Lakkundi
A2. One of the biggest pain points companies experience today is getting the right data in the hands of the right people at the right time. DataOps is on the rise, especially in IT, because it unleashes the full power and potential of the business.
Radhika Krishnan
A2 .1) #DataOps practices are not yet mainstream in today’s enterprise. However, what is mainstream is the recognition amongst our customers that adopting and applying DataOps is required to become a #DataDriven organization.
Sam Lakkundi
A2. To become data-driven, real-time insights & agility are key. DataOps enables both as it allows orgs to move data as it’s changed in real time using automation. Flexible integration solutions let IT change a source/target without disrupting the infrastructure to stay agile.
Chris Ehrlich
A2: Yes, DataOps as a concept is mainstream at large enterprises. At a minimum, as practices, if not a defined team-based structure. #DataOps #DataManagement

(edited)

Radhika Krishnan
A2. 2) As IT increasingly becomes hybridized across #CloudComputing, #IoT and on-premise systems, #DataOps practices are rising to the forefront to serve how data is created, managed, governed and consumed.
Cognite
A2: Not quite :) There is a lot of market education to be done still, and we're happy to be joined by you all here today pulling your weight to grow awareness for the benefits of DataOps. For those in industry, we've written The Definitive Guide To Industrial DataOps in the topic
Cognite
A2: For those looking to understand more how DataOps plays a transformational role in Industry 4.0, we invite you to visit https://content.cognite.com/the-definitive-guide-t... @JamesMaguire
https://content.cognite.com/the-definitive-guide-to-industrial-dataops
The Definitive Guide to Industrial DataOps
The Definitive Guide to Industrial DataOps
The guide, written by Cognite, explains the emerging discipline of Industrial DataOps for business and IT professionals - a first for the industry.
Bas Kamphuis
A2: It’s not mainstream yet. We do see interesting examples that leverage automation more and more. It’s an evolution I think, not that different from autonomous vehicles: first we had cars, then we have cruise control in these cars that was quite good at keeping t
Sam Lakkundi
in my view, organizations are streamlining processes so that data moves along the pipeline much more quickly, continuously yielding actionable insights and demonstrable value to the business. #DataOps is key to this line of thinking thus mainstream.
Sam Lakkundi
#DataOps is also a key theme at our annual virtual event #BMCExchange this year. In fact, @BMCSoftware CTO @RamChakravarti2 will be diving into how to unlock DataOps to drive maximum business value.
Radhika Krishnan
@brucekornfeld, we are hearing about it more and more from our customers in helping them put #DataOps into practice.
Bas Kamphuis
A2 We continue to see this gap be underestimated at some of our customers: "If I just could get the data from this system into this platform, the world of analytics and smart insights opens up to me".
Bas Kamphuis
A2: And after this IT driven integration work it finds a lack of adoption, because the context of which data is used and the business purpose is missing and transparent in the analytic solution.
James Maguire
Q3 Why is DataOps important in today’s data-intensive world?
Radhika Krishnan
A3. To unlock value, applications need access to that data to contextualize it and correlate it across different datasets from 1000s of different sources coming in different formats all the time. #DarkData
Bruce Kornfeld
A3. There is just too much data being generated (video, IoT, etc...) and not all of it is needed. DataOps will help organizations accomplish "data thinning" so only the important stuff is kept.
James Maguire
@brucekornfeld We are drowning in data....
Bruce Kornfeld
Yes! I can't image what being an IT Pro is like these days...how do they keep up????
Chris Ehrlich
A3: DataOps is important, because organizations that don’t commit to a DataOps methodology will be at a competitive disadvantage and never truly view and leverage big data across departments. They will effectively mismanage data. #DataOps #DataManagement
Cognite
A3: Our data landscape is not getting any less overwhelming, and as about all enterprises are struggling with making data truly available and useful for their data consumers, continuing with former rigid data management to business handover is simply not an option@JamesMaguire

(edited)

Sam Lakkundi
A3. We’re living in a time when data is generated in enormous numbers - by us, our devices, and the networks that transport it. It's a data-driven world, and as such, the data-driven business is sure to follow. (1/2)
Sam Lakkundi
A3. Being data-driven is a tenet of an Autonomous Digital Enterprise, where manual effort is minimized to capitalize on human creativity, skills, and intellect. (2/2)
Bruce Kornfeld
@samlakkundi We've been talking about the "growth of data" for so many years now - I just think its really coming to a head.
Bas Kamphuis
@brucekornfeld Agree that there is a lot of talk about the 'grwoth of data' - but is that really the issue in the context of dataops and enterprises - My point being that companies are data rich for decades, just insights poor
Bas Kamphuis
A3:
dataops to me is trying to translate that data rich enterprise into a insights rich
enterprise.

(edited)

Bas Kamphuis
A3: Lets go back to ‘purpose’ – our desire and ability to analyze data serves a purpose – probably something like ‘enabling better decisions that ultimately create a competitive advantage for the organization at large’. And that means either increasing revenue or re
Bas Kamphuis
A3: The ability to make smarter decisions, to analyze better, is in a large part constrained by the amount of data scientists that you can hire, the amount of people that you can find that understand it all.
Bas Kamphuis
A3: And so therefore data output, data ops, in that context is a force multiplier. If I can automatically interpret, I can scale the number of insights that I can extract and the speed in which I can make these insights.