DataOps

   2 years ago
#DataOpsDataOps in ActionJoin DataOps experts and the IBM Chief Data Office as they share best practices and methodologies to deliver business-ready data fast to accelerate digital transformation and the journey to AI.
Siddharth
What do you refer to as #DataOps?

#DevOps
Dr. G
A powerful combination of #Data #Engineering, Data #Flow mgmt and #integration, Data #Quality, Data Security, Data Privacy, and Governance
Mark Peters
# devops is dataops, no feedback without data
Siddharth
#DevOps is about feature velocity whereas #DataOps is analytic velocity. Need to talk about devops in order to understand dataops better :)
Mark Peters
too many terms, data is data and analyzing data drives decisions. Just like dev, data must add value, velocity should be in line with feedback
Dianna Pieper
in my opinion DataOps, like DevOps, should be about the behavior of creating teams that collaborate to ensure what is being delivered is timely, uses common practice etc (but should be data focused)
Siddharth
What are the first 3 steps towards DataOps?
Siddharth
1) Data Gathering, 2) Cloud Adoption 3) Analytics Velocity
#DataOps
Dr. G
Alignment to business goals and purpose of the data being gathered. 2) Treat data as a value stream and reduce friction from capture to insights 3) Seamless integration - with existing systems, cloud, disparate data sources
Dianna Pieper
agree Dr. G - before solutioning we need to understand what data we are addressing the determine the method to treat.
𝓐𝓷𝓾𝓻π“ͺ𝓰 𝓒𝓱π“ͺ𝓻𝓢π“ͺ
Know your Why of Data using data, ensure alignment, target low hanging for experiment and bitter fruits to bring focus of organisation
Leonardo Murillo
Which tools are you using or are you missing to achieve your DataOps objectives?
Leonardo Murillo
I have found that the ecosystem of DataOps focused tools is still maturing and does still have some important gaps, do you agree?

(edited)

Siddharth
as I already called out. to start your journey you don't need much tool. Having data modelling any would work.
Mark Peters
starting with the right tool too often means not asking the right data question
Dr. G
Agree with you. I would say we should first evaluate our data pipeline and see what gaps we currently have and look for a tool that can easily integrate into the pipeline, has a quick time to implement and a low learning curve and that will increase data value
Leonardo Murillo
But as you progress, DataOps teams will want to automate, validate data consistency, etc. Consider for instance the context of automating the production readiness of your data.
Siddharth
What key learnings from DevOps can help DataOps?
Dr. G
Focusing on right culture, treating data as your second most valuable asset (first is people), Automation, data pipelines, observability, Value stream management are some of the areas from #DevOps that can help #DataOps
PaweΕ‚ Piwosz
agree with all you said. I'd like to add ownership as key element of culture and go boldly forward from observability to telemetry, structured logs approach, etc
Leonardo Murillo
collaboration and mindset are key, its a culture shift before a technology one
Dianna Pieper
in my experience, having a transition plan is key, messaging, supporting failure without finger pointing, and having top management support
𝓐𝓷𝓾𝓻π“ͺ𝓰 𝓒𝓱π“ͺ𝓻𝓢π“ͺ
Lean agile mindset, feedback loops, collaboration and automation can be take to start with
Garima Bajpai
teamup for data, dataops is not project a mindset shift
William Szepesi
Hello everybody I'm joining from Ottawa! Back to basics to start... why did IBM raise the topic and initiate discussion around DataOps back in 2014? What issues were they facing at that time?
Mark Peters
probably decreased value generation compared to initial devops
Mark Peters
dont newd advanced tools until initial gains stop, same as weightlifting #westsidepl
William Szepesi
@TinyCyber implicitly then, Value Stream Management is a good partner for DataOps
Mark Peters
value stream management should be thw underpinning of any strategic change. If no value in action, then change. Meaximize the work not done #agile
Dianna Pieper
@TinyCyber agree, but who is ensuring this is occuring? there needs to be governance that supports the organizational goals and direction. Enterprise data would ensure the underpinning and DataOps would provide the practice to support delivery (in my opinion)
Mark Peters
Disagree. Strategic alignment should be everyones mission. Collaborative ownership of the end product, external governance can decrease value and create resentment. Different than compliance
Mark Peters
not a who ensures governance but what are the teams goals for data? Data analytics should be clear. Maybe a data person to offer various options?
Dianna Pieper
@TinyCyber Like cybersecurity, it is everyone's mission but not everyone can be accountable. Having guidance and governance to ensure regulatory and industry best practice is essential. This needs to be overseen to review for consistency in application
Mark Peters
Uncertain...perhaps if we had more data
Mark Peters
We’ve established flow and now feedback, perhaps an experiment. If we are chatting than the chat has started
Dianna Pieper
point taken. I am seeing a trend, DevOps to DataOps, and now seeing MLOps. thoughts?
Mark Peters
Too many word for similar processes. Key is simplicity, afterall attachment is the root of sufferings
Mark Peters
Whichever ops, goal is reducing silos and accelerating value to the customer
Dianna Pieper
I attended last week, concerned with meshing methodologies rather than creating one that customly fits
Mark Peters
Data results are always unique, data frameworks not so much, measure what matters
Dr. G
Data storage or data hoarding - how do you separate yourself from data hoarders?
Dianna Pieper
I believe this is where enterprise data structure will help.
Mark Peters
transparency! Clone your own data copy to do unusual things, then share results with others
William Szepesi
good question and I feel Value Stream Management or even a basic cost/benefit analysis can provide the answer here.
Siddharth
within the Banking & Finance Industry we go by the #governance #compliance rules. for e.g. we need to save transactions for X years then. Easy to segregate :)
Mark Peters
@WSzepesi keep all the data for the current process and store old atuff in varying archive layers
Mark Peters
@pareeksiddharth those transactions are different than dev data, can use dev for compliance but frequently includes policy and other elements
Garima Bajpai
start with your business question
Dianna Pieper
thanks for sharing - create a great day!
Dianna Pieper
As I understand this more, Data needs something like enterprise data, like enterprise architecture. DataOps, which to me is the practice of it - as it relates to creating and using) and enterprise data for planning to ensure consistency, governance, etc.
Mark Peters
Depends on the question, early gains need leas data for growth, the closer to potential, the more difficult to close the gap
Dianna Pieper
I believe we need to understand what 'data' you are referring.
Mark Peters
Any data associated with the value stream strategically
Garima Bajpai
I do believe that is needed going forward from lean to enterprise #leandata