bigdataautomation

Ralph Finos
http://www.via-cc.at...

Krish Krishnan
They are two sides of a coin and need to be treated that way
MaribelLopez
I believe these could be powerful together. Not doing it and can say I have a few customers that I know are doing it. But they are progressive on multiple levels
Krish Krishnan
my developers love the fact they can develop, test, optimize and automate
Krish Krishnan
This is my team mantra that I teach across the world
Yves Mulkers
Strange to see that the Dev way of working seems to be unknown to a lot of #data people. Even some developers still don't manage the full software development lifecycle....
Basil Faruqui
This is an area where we see a lot of interest as organizations look to deliver at scale using CI/CD frameworks. The key is to ensure that the workload automation solution can seamlessly co-exist with the rest of the devops toolchain
Krish Krishnan
@BFaruqui i think the CxO generation is also fast changing and that is good news
MaribelLopez
The devops toolchain is still being developed for most so I think we could integrate them
Krish Krishnan
Strategy wise data workload needs to be designed based on platform, code will be independent but DevOps + KANBAN has provided success
Jennifer Shin
In one of my recent roles, I automated validating and testing #bigdata before releasing it to clients and shortened hte time required by over 80%
Krish Krishnan
great example here is implementation of BlockChain by companies including Disney
Yves Mulkers
@MaribelLopez It's an attitude towards delivering repeatable, sustainable solutions. And not a quick one time shot. It's like I've always seen... business starts developing something without the guidance.
Andrew Brust
Microsoft's new data prep tool (embedded into its AI Workbench, for now) allows you to archive data flows into GitHub. So there *is* momentum in this direction.
Robby Dick
@jennjshin that is impressive - how did you do it?
jameskobielus
From my research, I'm starting to see a closer DevOps alignment of data-app developers and traditional developers. The impetus is the increasing prevalence of machine learning in apps of all sorts. Data, model, code, rules etc in a single CI/CD
MaribelLopez
@YvesMulkers +1 business is running fast, groups are uncoordinated. In effect, we're recrafting how we think about integrating work or multiple groups
Jennifer Shin
Businesses often don't realize that #bigdata testing and validation are different processes - the latter is a good starting point for automation
Basil Faruqui
@jennjshin shortening the time to market is key as most #bigdata projects are intended to create competitive advantage
MaribelLopez
@datagenius Your developers sound bought in to the process. I still have some clients in Waterfall. Others are full on with 2 wk sprints
jameskobielus
@BFaruqui The DevOps toolchain is increasingly evolving to incorporate robust pipelines for ML model dev, training, and deploy. See my latest Wikibon note: https://wikibon.com/...
Yves Mulkers
@jennjshin WIth testing you mean integrating the data? the technical aspect?
Jennifer Shin
@robbydbmc when businesses are well behind the times, it is easier to improve the process with automation - the company used to have an analyst check the data manually...
jameskobielus
@YvesMulkers From what I can see, most developers don't manage the FULL dev lifecycle. Most are specialized to coding. Some only to QA.
Dana Gardner
DevOps complexity will only increase, factor multi-cloud, and so automation across all aspects, not just data, is essential. Also ML and AI may help in managing hybrid deployments, and so will need data assimilation and mgmt.
Jennifer Shin
@YvesMulkers If I'm testing the data, I will check for certain expected outputs or evaluate scenarios where I suspect there might be an issue. with a new data integration processing, testing is essential, but not as much once it's established
Robby Dick
@jennjshin unfortunately not all that surprising - too much manual effort spent on things that can be automated
Yves Mulkers
@jennjshin Nice view on that. Great that you make the distinction between the 2 phases.
Ralph Finos
http://www.via-cc.at...

Krish Krishnan
Data Science Evolution
Yves Mulkers
Time to Market. From #data to business Insights? Effort, budget, repeatable
Krish Krishnan
Business Collaboration on Governance
Andrew Brust
Ultimately, I think it will surface in project increased success rates.
Krish Krishnan
Reduction of Risk over Time
Krish Krishnan
Reduction of Silos
Basil Faruqui
In my experience customers who start big data projects with defined problem to solve are able to clearly measure success or failure vs many others who are starting from the point of view of let's build a data lake and then we will find the golden nuggets
MaribelLopez
@YvesMulkers +1 time to market, also time to deliver new insights
jameskobielus
IT overhead reduction.
Krish Krishnan
Big Data is NOT EQUAL to Big Success but it EQUALS Collaboration
Yves Mulkers
@jameskobielus Not only. It's more about speed, control and dependency...
Andrew Brust
Self-service adoption of the corp. data lake will rise with automation of data workloads, as well.
jameskobielus
Faster time-to-insight if data can be discovered, ingested, prepped, & accessed more rapidly by decision makers and analytics-infused process automation can be made continually agile and contextual.
Krish Krishnan
I also would like to refer to Monsanto and Novartis for their success case studies
Dana Gardner
Agility that abets business innovation, and the agility comes from data-driven refinements and increased automation. Show the tie-in between data-workload automation and resulting agility to deliver change.
Andrew Brust
Flywheel effect. Automation makes data lake more useful, encourages greater utilization, begets more discipline re automation.
jameskobielus
@andrewbrust For sure. Data lakes are being made self-replenishing and analytics insights are being harvested from them in an ML/DL/AI frame with less human intervention.
Jennifer Shin
I establish metrics at the start of every project. I need to estimate the benefits of automating a project before I can convince the business to commit to large #bigdata automation that requires resouces
Ralph Finos
http://www.via-cc.at...

Krish Krishnan
In our organization and clients, we are seeing big data drive the business transformation
Krish Krishnan
the transformation is data driven and empowered by robotic process automation
MaribelLopez
My clients are seeing big data enable them to understand the new dynamics of the business
Krish Krishnan
i see the emergence of a data kernel layer that will make this a differentiator
Basil Faruqui
we are seeing more and more companies looking at data driven analytics as a way to drive new business models and shaping new customer experiences
MaribelLopez
#bigdata isn't new I think the real issue is speed at which it needs to be turned into insight
George Gilbert
data-driven analytics anticipates and influences user interactions and informs or automates decisions in the form of business transactions
jameskobielus
In many organizations, big data comes in through scalable log analysis in IT infrastructure management. Here's a recent Wikibon report by George Gilbert on machine learning on big data for IT management: https://wikibon.com/...
Krish Krishnan
the rapid emergence of data driven transformation has opened new doors for innovation
George Gilbert
@MaribelLopez completely agree on speed. ever lower latency is becoming more crucial in more usage scenarios
MaribelLopez
@ggilbert41 I call these delivering right-time experiences - using #analytics to deliver #contextual business processes
Krish Krishnan
we also see the evolution of neural networks and advanced machine learning to empower data science
Dana Gardner
I'm seeing more that business processes are not yet taking advantage of the fruits of big data -- insights and learning.
Jennifer Shin
#bigdata offers insights into how our service is used by current customers, which is essential for data driven product development
Krish Krishnan
to me personally the whole move signifies the emergence of a real world data warehouse finally
jameskobielus
The big-data-driven digital business processes may be back-end (i.e., IT service management) or front-end (customer engagement, multichannel marketing). Increasingly, both front- and back-end driven by common data lake for machine learning etc.
Basil Faruqui
@MaribelLopez speed is definitely the new digital currency. In addition to speed what we find is crucial is the ability to scale.
Krish Krishnan
a tagline for this is chaos is a constant & change is inevitable
Dana Gardner
Apps are being harvested for the data, and its applied to analytics, but the feedback to optimize the workflows and processes is still lagging
George Gilbert
the bottleneck to applying big data to actionable analytics is the relative scarcity of people with skills to build ML/AI models.
Krish Krishnan
most recently in Paraguay I was interviewed on Live TV, their only question is how slow are we?
Jennifer Shin
by collecting #bigdata businesses can improve existing processes by quantifying their efficiency (or inefficiency)
Robby Dick
Good point @Dana_Gardner , existing business processes are left untouched and the #bigdata work is often standalone - sometimes that might be okay, but usually not!
jameskobielus
In terms of data-driven business processes, what's happening is more of those will be driven from the edge--data sources and consumers in mobile, IoT, and industrial endpoints. AI will be the driver of those processes. And that rides big data.
George Gilbert
@Dana_Gardner coudln't agree more. expect to see more ISV applications that "hook into" underlying legacy applications
MaribelLopez
@jennjshin quantifying processes efficiency is a big deal Most co need to build new processes to be successful in #futureofwork
jameskobielus
@datagenius Krish: What is robotic process automation and how are your clients using it?
George Gilbert
@jameskobielus typically these are "screen scraping" apps that try to do the equivalent of cross application scripting for enterprise applications. automation comes from eliminating human operators from some processes
jameskobielus
@datagenius What's a "data kernel layer"?
Jennifer Shin
@MaribelLopez agreed. quantifying process efficiency is not only essential for successful #futureofwork, it is becoming a differentiator for competition
jameskobielus
@ggilbert41 That bottleneck is increasingly being addressed through ML automation, but those tools are not entirely mature yet or widely adopted.
jameskobielus
@Dana_Gardner Dana: Is that due to anemic adoption of business intelligence tools, or to lack of a culture of evidence-driven decision making, or both? Or something else?