bigdataautomation

They are two sides of a coin and need to be treated that way

I believe these could be powerful together. Not doing it and can say I have a few customers that I know are doing it. But they are progressive on multiple levels

my developers love the fact they can develop, test, optimize and automate

This is my team mantra that I teach across the world

Strange to see that the Dev way of working seems to be unknown to a lot of #data people. Even some developers still don't manage the full software development lifecycle....

This is an area where we see a lot of interest as organizations look to deliver at scale using CI/CD frameworks. The key is to ensure that the workload automation solution can seamlessly co-exist with the rest of the devops toolchain

@BFaruqui i think the CxO generation is also fast changing and that is good news

The devops toolchain is still being developed for most so I think we could integrate them

Strategy wise data workload needs to be designed based on platform, code will be independent but DevOps + KANBAN has provided success

In one of my recent roles, I automated validating and testing #bigdata before releasing it to clients and shortened hte time required by over 80%

great example here is implementation of BlockChain by companies including Disney

@MaribelLopez It's an attitude towards delivering repeatable, sustainable solutions. And not a quick one time shot. It's like I've always seen... business starts developing something without the guidance.

Microsoft's new data prep tool (embedded into its AI Workbench, for now) allows you to archive data flows into GitHub. So there *is* momentum in this direction.

@jennjshin that is impressive - how did you do it?

From my research, I'm starting to see a closer DevOps alignment of data-app developers and traditional developers. The impetus is the increasing prevalence of machine learning in apps of all sorts. Data, model, code, rules etc in a single CI/CD

@YvesMulkers +1 business is running fast, groups are uncoordinated. In effect, we're recrafting how we think about integrating work or multiple groups

Businesses often don't realize that #bigdata testing and validation are different processes - the latter is a good starting point for automation

@jennjshin shortening the time to market is key as most #bigdata projects are intended to create competitive advantage

@datagenius Your developers sound bought in to the process. I still have some clients in Waterfall. Others are full on with 2 wk sprints

@BFaruqui The DevOps toolchain is increasingly evolving to incorporate robust pipelines for ML model dev, training, and deploy. See my latest Wikibon note: https://wikibon.com/...

@jennjshin WIth testing you mean integrating the data? the technical aspect?

@robbydbmc when businesses are well behind the times, it is easier to improve the process with automation - the company used to have an analyst check the data manually...

@YvesMulkers From what I can see, most developers don't manage the FULL dev lifecycle. Most are specialized to coding. Some only to QA.

DevOps complexity will only increase, factor multi-cloud, and so automation across all aspects, not just data, is essential. Also ML and AI may help in managing hybrid deployments, and so will need data assimilation and mgmt.

@YvesMulkers If I'm testing the data, I will check for certain expected outputs or evaluate scenarios where I suspect there might be an issue. with a new data integration processing, testing is essential, but not as much once it's established

@jennjshin unfortunately not all that surprising - too much manual effort spent on things that can be automated

@jennjshin Nice view on that. Great that you make the distinction between the 2 phases.

Data Science Evolution

Time to Market. From #data to business Insights? Effort, budget, repeatable

Business Collaboration on Governance

Ultimately, I think it will surface in project increased success rates.

Reduction of Risk over Time

Reduction of Silos

In my experience customers who start big data projects with defined problem to solve are able to clearly measure success or failure vs many others who are starting from the point of view of let's build a data lake and then we will find the golden nuggets

@YvesMulkers +1 time to market, also time to deliver new insights

IT overhead reduction.

Big Data is NOT EQUAL to Big Success but it EQUALS Collaboration

@jameskobielus Not only. It's more about speed, control and dependency...

Self-service adoption of the corp. data lake will rise with automation of data workloads, as well.

Faster time-to-insight if data can be discovered, ingested, prepped, & accessed more rapidly by decision makers and analytics-infused process automation can be made continually agile and contextual.

I also would like to refer to Monsanto and Novartis for their success case studies

Agility that abets business innovation, and the agility comes from data-driven refinements and increased automation. Show the tie-in between data-workload automation and resulting agility to deliver change.

Flywheel effect. Automation makes data lake more useful, encourages greater utilization, begets more discipline re automation.

@andrewbrust For sure. Data lakes are being made self-replenishing and analytics insights are being harvested from them in an ML/DL/AI frame with less human intervention.

I establish metrics at the start of every project. I need to estimate the benefits of automating a project before I can convince the business to commit to large #bigdata automation that requires resouces

In our organization and clients, we are seeing big data drive the business transformation

the transformation is data driven and empowered by robotic process automation

My clients are seeing big data enable them to understand the new dynamics of the business

i see the emergence of a data kernel layer that will make this a differentiator

we are seeing more and more companies looking at data driven analytics as a way to drive new business models and shaping new customer experiences

#bigdata isn't new I think the real issue is speed at which it needs to be turned into insight

data-driven analytics anticipates and influences user interactions and informs or automates decisions in the form of business transactions

In many organizations, big data comes in through scalable log analysis in IT infrastructure management. Here's a recent Wikibon report by George Gilbert on machine learning on big data for IT management: https://wikibon.com/...

the rapid emergence of data driven transformation has opened new doors for innovation

@MaribelLopez completely agree on speed. ever lower latency is becoming more crucial in more usage scenarios

@ggilbert41 I call these delivering right-time experiences - using #analytics to deliver #contextual business processes

we also see the evolution of neural networks and advanced machine learning to empower data science

I'm seeing more that business processes are not yet taking advantage of the fruits of big data -- insights and learning.

#bigdata offers insights into how our service is used by current customers, which is essential for data driven product development

to me personally the whole move signifies the emergence of a real world data warehouse finally

The big-data-driven digital business processes may be back-end (i.e., IT service management) or front-end (customer engagement, multichannel marketing). Increasingly, both front- and back-end driven by common data lake for machine learning etc.

@MaribelLopez speed is definitely the new digital currency. In addition to speed what we find is crucial is the ability to scale.

a tagline for this is chaos is a constant & change is inevitable

Apps are being harvested for the data, and its applied to analytics, but the feedback to optimize the workflows and processes is still lagging

the bottleneck to applying big data to actionable analytics is the relative scarcity of people with skills to build ML/AI models.

most recently in Paraguay I was interviewed on Live TV, their only question is how slow are we?

by collecting #bigdata businesses can improve existing processes by quantifying their efficiency (or inefficiency)

Good point @Dana_Gardner , existing business processes are left untouched and the #bigdata work is often standalone - sometimes that might be okay, but usually not!

In terms of data-driven business processes, what's happening is more of those will be driven from the edge--data sources and consumers in mobile, IoT, and industrial endpoints. AI will be the driver of those processes. And that rides big data.

@Dana_Gardner coudln't agree more. expect to see more ISV applications that "hook into" underlying legacy applications

@jennjshin quantifying processes efficiency is a big deal Most co need to build new processes to be successful in #futureofwork

@datagenius Krish: What is robotic process automation and how are your clients using it?

@jameskobielus typically these are "screen scraping" apps that try to do the equivalent of cross application scripting for enterprise applications. automation comes from eliminating human operators from some processes

@datagenius What's a "data kernel layer"?

@MaribelLopez agreed. quantifying process efficiency is not only essential for successful #futureofwork, it is becoming a differentiator for competition

@ggilbert41 That bottleneck is increasingly being addressed through ML automation, but those tools are not entirely mature yet or widely adopted.

@Dana_Gardner Dana: Is that due to anemic adoption of business intelligence tools, or to lack of a culture of evidence-driven decision making, or both? Or something else?