datafriction

doubling annually

growth is as fast as enterprises can capture the data. it's not the 40% CAGR IDC and EMC keep citing

40% is just Moore's Law

I'm repping a vendor, but I can say from our perspective hearing from clients, it's dramatically - esp. in retail, distribution, telco and managed services sectors.

Similar to Randy, but for our customers it's massive growth

@readyforthenet I've heard that as a general rule of thumb for annual data growth across enterprises of various sizes, industries, etc.

it volume is growing related to increased variety of metrics being captured

Management data is slower than data overall, but not by much. It's massive growth. Easily 40%+ a year.

Retail supply chain is driving enormous data sprawl.

@dorkninja The industries that are doing big data most avidly tend to show greatest growth. Especially those, such as retail and media, that are making huge investments in machine learning and AI and need huge training data sets.

one way to think about growth: number of entities in the application and infrastructure landscape (ever more fine-grained) emitting ever more telemetry per entity

IT Ops is becoming a data-driven practice, hence growth. Need drives growth!

@mcauth Can you give an order of magnitude on that? 10x annual growth? 100x?

@mcauth It's the "DevOps-ification" if IT!

@jameskobielus More anecdotal, but for many customers it's 2-5x annually

@mcauth Is the volume of infrastructure data growing in direct proportion to the size and complexity of the IT infrastructure itself? Or is the amount of infrastructure data growing faster (or slower) than the infrastructure?

@colin_walker I'm curious why management data is growing more slowly (albeit slightly) than application data. Are IT management tools growing more sophisticated in deriving analytic insights for the tasks they perform?

I just think there's a naturally offset correlation. You deal with more bytes in/out as things like HD vid, audio streams, higher graphic content etc. become the norm. But those things don't automatically drive more mgmt data.

@ggilbert41 exactly

I see scale-out app architectures and associated service policies driving the need for increased measurements

Big issue. Integrated ITOM data and tooling implies more integrated roles and orgs. Scares a lot of folks.

One CIO told me: This is the biggest challenge to his business's adopting new technology.

@plburris Yep, people are afraid of change, afraid of retooling not only the compute side, but the human side. Understandable but also necessary.

@ggilbert41 Great framework for sizing IT management data.

@mcauth IT ops is also becoming an ML/AI driven process. Anomaly detection amid petabytes+ absolutely demands precision "pattern-sniffing" in real-time.

@jameskobielus IT Ops will begin to look more like IoT as more intelligence and decision ability is put into end systems

@readyforthenet Or perhaps it's the other way around :) IoT looks more like ops. Either way, fascinating perspective.

@jameskobielus ...and effective feature extraction.

with that approach you can create a data lake for IT which can be further refined into machine learning models of how specific domains work

Agreed, although challenging from an integration and orchestration perspective sometimes.

you can create a #datalake for the entire org - not just IT. Which is how it should be

@dorkninja exactly - that's the trade-off

Agreed, assuming you can actually make use of that data lake in am meaningful way. So often people get huge cess pools of unrefined data that isn't useful, doesn't have appropriate ML features, and they can garner no insight from. #SadMLFails

here too I'd say that different data sources warrant different treatment, though ultimately they can be unified

see cess pools being created and access to data through same old stove piped approach

@mcauth that's the reality today but also the goal

+1. Hurts my data loving soul, but it's a song all too common. Companies know they "need" big data, but have no idea how to implement/refine/use/benefit. So they end up with a non useful solution, are mired in it for years, and fall behind.

@Jshoc I've heard about data lakes that became swamps but first time I've heard cesspool

@colin_walker Yup. And that inhibits future innovation and the appetite to experiment.

Nice to have you on the chat.

Precisely. Suddenly the message is "We tried that big data thing. It's bogus. Why should we try it again?" when new, innovative, powerful approaches surface. Same issue "cloud" and many new techs have faced, frankly.

@colin_walker some vendors build in the data/ML smarts for a particular domain so their application collects and organizes only the data they need in the data lake

@jameskobielus happy to be here - see you in NYC in a few weeks?

What's the "SAME way" that all data should be managed and integrated? A data lake? Data warehouse? Data refinery?

Truth. Unfortunately I think the biggest value of ML won't come from silo driven lakes. Correlation is *money* when it comes to trends/patterns.

akin to to dashboard exhuast

@ggilbert41 Specific IT management domains? Such as incident response? etc.

@jameskobielus data lake with governance - I like Steve Smith's piece https://www.eckerson...

@colin_walker that's right and you often won't know what's correlated up front

I'll be at Strata at the end of the month. Let's talk. You guys doing the Cube?

Data lakes only become cesspools if they do not have a good ILM process in place from the outset

Precisely. Process/analyze/correlate the best you can, and store for further analysis? Yes please. #DataNirvana

and value data

@colin_walker It's all about getting to the data with minimal friction, max velocity

Great let's do that - @2thebeach coordinating our Cube participation

@jameskobielus i think incident response is more horizontal. i was thinking about intrusion detection or management of an application's full stack such as SAP, etc

@colin_walker We're the emergence lineage, provenance, governance and security capabilities for data lakes, but what is still laking is a clear vision of the desired outcome.

@colin_walker yes please

@NeilRaden Oof, that hits home hard. 100% agree. People don't know what they want to know, yet. They're looking for not only the answers, but the questions they're supposed to be asking. Makes it tough to set up appropriate ILM, practices, etc.

@mcauth Exactly! How fast can I get to insights? How little time can I waste digging through what I don't need, or care about?

@colin_walker There are some emerging autodiscovery and pattern sniffing techniques that can help here I think. Akin to pharma saving all clinical trial and vector data forever, in case a future superbug appears.

@EvansBI yes but - you don't always know what you're looking for so it needs to be flexible - historically ILM hasn't been

@NeilRaden ...and perhaps a commitment from the organization to tap into the potential. It'll provide returns if the commitment is there.

@NeilRaden problem I believe is confusion from vendors about what outcome is best for the data you have ingested - specific use cases should be designed by industry to enable governance to work and be compliant with regulatory rules GDPR etc

@dorkninja Agreed. Lots of options coming that use ML to ask the questions of the data without humans having to know what to ask. Definitely the way forward, IMO. #OneMansOpinion. Part of why we use this technique in our tech. #DontKnowWhatYouDontKnow

@colin_walker Spent a little time with Jeff Jonas while at IBM - he would vehemently agree!

ILM based on regulatory compliance and industry methodology can be tuned to be both flexible and encompassing - problem with Big Data technologies is that they primarily did not start out that way so difficult to enact ILM correctly

@dorkninja patterns for sure...and relevant features extracted from the operational data. This is key to the ML equation, to put it mildly

@mcauth What's the role of machine learning in unifying the treatment of different data sources while also enabling differentiated task-specific IT management insights?

@jameskobielus Increasingly large. Putting data sources aside for a moment, the growth alone warrants machine assist. Human analyst error rates are high vs. machines for a huge swath of ops intel.

@EvansBI yup that's the challenge but also the opportunity - we're working on it at @unifisoftware

@jameskobielus ML has a substantial role in demand forecasting/management

@mcauth Exactly. Forget sources or silos. The sheer growth and volume of inbound to be processed means ML is either currently or soon 100% required, full stop. Otherwise what's your option? Call center sized buildings full of data analysts? #Pass

Funny that so do we at @solixbigdata :-)

@colin_walker Industrial-grade feature discovery on that infrastructure data can call out the predictors from unstructured data. Perhaps cluster analysis algorithms.

@jameskobielus Precisely. Proper Discovery -> ML -> Analysis -> Reporting chains are the way of the future. #NotKickBoxing

@EvansBI we should probably chat offline!