datasciencedevops

DevOps in Data Science
Discuss how developers are bringing DevOps practices into the data-science pipeline.
jameskobielus34
http://www.via-cc.at...

2 Votes Vote
Peter Burris data science!
2 Votes Vote
Peter Burris I know it sounds recursive, but devops is going to need a LOT of data science-like stuff to reach full potential.
4 Votes Vote
John Furrier #devops enablers are integrated "open toolchains"
1 Votes Vote
John Furrier #devops enable #2: use of analytics and cognitive/deep learning
1 Votes Vote
John Furrier #devops enablement #3: microservices and container adoption
1 Votes Vote
John Furrier #devops enabler #4: fully integrated #PrivateCloud and #PublicCloud
1 Votes Vote
jameskobielus I'll jump in here: source-control repository, data lake, and integrated collaboration environment that spans the entire pipeline.
2 Votes Vote
John Furrier Just did a #crowdchatstorm in a thread take that @pmarca
1 Votes Vote
jameskobielus @furrier The integrated toolchain needs to be embedded within the integrated DevOps collaboration environment that I alluded to.
2 Votes Vote
jameskobielus Question 6 coming.
2 Votes Vote
Kirk Borne #Microservices come to mind. Also, APIs... here are just a few: https://twitter.com/...
3 Votes Vote
Kirk Borne #Containers have really had a huge positive impact on productivity for #MachineLearning #DataScience teams in my organization
3 Votes Vote
Kirk Borne #DataLakes done right will definitely be a big plus in breaking down data silos, enabling rapid innovation, and zero-day discovery from new data sources: https://twitter.com/...
2 Votes Vote
Peter Burris @KirkDBorne How? I can see why it might, but can you offer specifics?
1 Votes Vote
jameskobielus @plburris Right. Automated ML-driven code-gen tools are coming along fast and furious. Microsoft, for example, is making great strides to use ML for rapid app development and iteration.
1 Votes Vote
Kirk Borne @plburris Schema-on-read is fast. Schema-on-write requires months of data modeling, design, development, testing,... i.e., DevOps in the database-build phase, yes, but which I am not seeing too much of.
2 Votes Vote
Kirk Borne See my article "Mining the #BigData Wheel" that mentions fast schema-on-read day-zero analytics here: https://mapr.com/blo... at @MapR #DataScience
2 Votes Vote
jameskobielus @KirkDBorne You are quite right. Precious little DevOps drives the database modeling process.
1 Votes Vote
Peter Burris2
How many DevOps-using companies are applying these tools and techniques to the development pipeline for machine learning and other data science apps?

How many DevOps-using companies are applying these tools and techniques to the development pipeline for machine learning and other data science apps?

2 Votes Vote
jameskobielus29
http://www.via-cc.at...

1 Votes Vote
jameskobielus Posting another poll. Look up and/or refresh your browsers.
2 Votes Vote
John Furrier I think this is developing now. ML gets people excited but the value will be in the automation of tasks
2 Votes Vote
John Furrier Some of the most successful organizations we see employ end to end orchestration solutions to automation the software delivery pipeline from developer check-in to production release and feedback loops from production monitoring
2 Votes Vote
Kirk Borne Yes, it is developing now, and it is about time. Very excited to see this upcoming conference: https://www.dataopss...
2 Votes Vote
John Furrier @KirkDBorne best practices as key as early adopter and pioneers are learning lots fast here
2 Votes Vote
Peter Burris By definition, how can an ML pipeline be run any way other than by a method that looks a lot like devops.
3 Votes Vote
Kirk Borne I suspect that many orgs who do #DataScience and #MachineLearning pipelines and #DataProduct development are using #DevOps, because it is natural to do so, and there is no need to broadcast something so obvious to the world.
2 Votes Vote
Kirk Borne @plburris Bingo! Right!!
2 Votes Vote
jameskobielus @furrier Yes. Cross-role orchestration of disparate processes in the ML pipeline--ingest, preparation, modeling, training, deployment, feedback, etc.
1 Votes Vote
jameskobielus Now for Question 5. Look above.
0 Votes Vote
David Floyer Until the outcome of data scientist is measured by automation of business processes, very little adoption of DevOps will happen
3 Votes Vote
jameskobielus @plburris You can run an ML pipeline through sneakernet, but that's an extraordinary waste of high-priced data scientists' workdays.
3 Votes Vote
jameskobielus @KirkDBorne That's right. "Something so obvious." We see the automation of the data-science pipeline in A/B testing that's going on in every org's ML-driven e-commerce, recommendation engine, mobile apps and other app infrastructures.
2 Votes Vote
Kirk Borne @dfloyer @jameskobielus Automation is so important and often so lost on folks who play in the #DataScience sandbox. https://xkcd.com/974...
2 Votes Vote
Peter Burris Yes! Waste! Hence, my equating of DevOps --> Lean.
0 Votes Vote
Kirk Borne5
Survey results on DevOps adoption in IT projects: http://www.via-cc.at...

4 Votes Vote
John Furrier great the new thread is where links expand. Images, slideshares, videos, and blog links. Thanks for sharing
1 Votes Vote
jameskobielus39
http://www.via-cc.at...

2 Votes Vote
John Furrier This is a loaded question but I'd say it depends where the conversation is started in organization or C-Level
2 Votes Vote
John Furrier it doesn't matter where the initiative starts but where it ends. Adoption should yield results
1 Votes Vote
Kirk Borne Okay, I am going to cheat. Here are the results of a recent survey on DevOps adoption: https://betanews.com...
3 Votes Vote
John Furrier I still think the chasm is being crossed as we speak and #devops pioneers still view devops as devops; mainstream call is #cloudOps
1 Votes Vote
jameskobielus @KirkDBorne Great article. Thanks for sharing, Kirk.
2 Votes Vote
John Furrier Company putting out a manifesto doesn't make it #devops real agility and proof points wins the day
2 Votes Vote
John Furrier 35% of some projects proves my thought on chasm crossing #cloudops is here which is #devops made easy more automation required
1 Votes Vote
Peter Burris Like most complex, social changes: It's selective.
2 Votes Vote
jameskobielus I have not seen any research pointing to DevOps adoption rates in enterprise data science. but this cited research gives numbers on DBA adoption of DevOps, which is interesting.
3 Votes Vote
jameskobielus Question #4 up above.
2 Votes Vote
John Furrier #devops challenge is scaling it beyond the small number of teams and projects
1 Votes Vote
Peter Burris Where Agile is the dev process, and ops process less driven by hardware (i.e., cloud), more likely to find devops.
2 Votes Vote
John Furrier main comment from #devops pros is: "devops is never finished"
2 Votes Vote
John Furrier I think incentives across siloed executive leadership are the largest inhibitors to the DevOps transformation; once executive incentives motivate collaboration over siloed transformations, then you win
2 Votes Vote
Kirk Borne There are various definitions of DataOps, but I prefer this one: #DevOps for #DataScience = #DataOps (IMHO). It's about #DataProduct design, development, deployment lifecycle.
1 Votes Vote
John Furrier This also might be good to talk value stream automation
1 Votes Vote
David Floyer Only when development perceive that the Ops is useful in getting code out faster.
1 Votes Vote
jameskobielus @furrier Another key challenge is scaling to handle the growing range of artifacts--code snippets, statistical models, metadata, schemas, etc.--in a complex app-dev pipeline.
2 Votes Vote
Kirk Borne The rate of change (actually, rate of acceleration) in digital transformation is really high and is jerking biz around (including tipping points and future shock): https://www.amazon.c... by @csurdak
2 Votes Vote
Peter Burris @KirkDBorne We equate digibiz = differential use of data. As more firms institutionalize work around data assets, more digibiz -- which amplifies the role of data assets.
1 Votes Vote
Peter Burris @KirkDBorne Hence, the acceleration.
1 Votes Vote