datafriction

doubling annually

growth is as fast as enterprises can capture the data. it's not the 40% CAGR IDC and EMC keep citing

40% is just Moore's Law

I'm repping a vendor, but I can say from our perspective hearing from clients, it's dramatically - esp. in retail, distribution, telco and managed services sectors.

Similar to Randy, but for our customers it's massive growth

@readyforthenet I've heard that as a general rule of thumb for annual data growth across enterprises of various sizes, industries, etc.

it volume is growing related to increased variety of metrics being captured

Management data is slower than data overall, but not by much. It's massive growth. Easily 40%+ a year.

Retail supply chain is driving enormous data sprawl.

@dorkninja The industries that are doing big data most avidly tend to show greatest growth. Especially those, such as retail and media, that are making huge investments in machine learning and AI and need huge training data sets.

one way to think about growth: number of entities in the application and infrastructure landscape (ever more fine-grained) emitting ever more telemetry per entity

IT Ops is becoming a data-driven practice, hence growth. Need drives growth!

@mcauth Can you give an order of magnitude on that? 10x annual growth? 100x?

@mcauth It's the "DevOps-ification" if IT!

@jameskobielus More anecdotal, but for many customers it's 2-5x annually

@mcauth Is the volume of infrastructure data growing in direct proportion to the size and complexity of the IT infrastructure itself? Or is the amount of infrastructure data growing faster (or slower) than the infrastructure?

@colin_walker I'm curious why management data is growing more slowly (albeit slightly) than application data. Are IT management tools growing more sophisticated in deriving analytic insights for the tasks they perform?

I just think there's a naturally offset correlation. You deal with more bytes in/out as things like HD vid, audio streams, higher graphic content etc. become the norm. But those things don't automatically drive more mgmt data.

@ggilbert41 exactly

I see scale-out app architectures and associated service policies driving the need for increased measurements

Big issue. Integrated ITOM data and tooling implies more integrated roles and orgs. Scares a lot of folks.

One CIO told me: This is the biggest challenge to his business's adopting new technology.

@plburris Yep, people are afraid of change, afraid of retooling not only the compute side, but the human side. Understandable but also necessary.

@ggilbert41 Great framework for sizing IT management data.

@mcauth IT ops is also becoming an ML/AI driven process. Anomaly detection amid petabytes+ absolutely demands precision "pattern-sniffing" in real-time.

@jameskobielus IT Ops will begin to look more like IoT as more intelligence and decision ability is put into end systems

@readyforthenet Or perhaps it's the other way around :) IoT looks more like ops. Either way, fascinating perspective.

@jameskobielus ...and effective feature extraction.

Glad to be here. I've been covering this topic for years under various contexts, including data latency, data virtualization, etc.