datafriction

I'm in!

Sign me up :)

100%. Let's do this.

Generally speaking? Harder. Rapidly. Exponentially. With the right platforms/tools? Definitely still possible and perhaps "easy". But with tools of yesteryear? Bordering on impossible soon.

Depends on the use case environment and again, industry segment I think. Analytics are becoming very broad based and flexible - good and bad. Target rich, accuracy poor in many cases.

@RalphFinos Expectations for value are increasing, so once you engage in it, you find it to be harder because you begin to discover what data is valuable and what isn't and what sources you actually need

...and extracting value depends on use case much of the time

2/ meaning: incident response or outage: need data *now*

3/ @JasonJohnsonPMP forensics / capacity planning / etc...historical is needed.

@mcauth uses cases can be built in in the form of models about how the applications and infrastructure interact - deeper for narrower domains, broadly for more end-to-end visibility

Good Ops data solutions meet the use case "where it's at"

There's seems to be a developing movement toward gathering/staging data en masse, and figuring out what to do with it later in some cases.

@dorkninja Interesting. When you say IT management analytics are "target rich, accuracy poor," is there some underlying issue with IT mgt data accuracy, or analytic tool sophistication?

Regarding use cases: Stream processing systems extract value from data in motion (need data *now*) while data lake approaches are better for trending, planning, analytics.

@ggilbert41 What sort of models are you referring to? Data models? Statistical models? Orchestration models explicating iinteractions among IT infrastructure models? The more "end-to-end" and "up the stack" their span, powerful but unwieldy

doubling annually

growth is as fast as enterprises can capture the data. it's not the 40% CAGR IDC and EMC keep citing

40% is just Moore's Law

I'm repping a vendor, but I can say from our perspective hearing from clients, it's dramatically - esp. in retail, distribution, telco and managed services sectors.

Similar to Randy, but for our customers it's massive growth

@readyforthenet I've heard that as a general rule of thumb for annual data growth across enterprises of various sizes, industries, etc.

it volume is growing related to increased variety of metrics being captured

Management data is slower than data overall, but not by much. It's massive growth. Easily 40%+ a year.

Retail supply chain is driving enormous data sprawl.

@dorkninja The industries that are doing big data most avidly tend to show greatest growth. Especially those, such as retail and media, that are making huge investments in machine learning and AI and need huge training data sets.

one way to think about growth: number of entities in the application and infrastructure landscape (ever more fine-grained) emitting ever more telemetry per entity

IT Ops is becoming a data-driven practice, hence growth. Need drives growth!

@mcauth Can you give an order of magnitude on that? 10x annual growth? 100x?

@mcauth It's the "DevOps-ification" if IT!

@jameskobielus More anecdotal, but for many customers it's 2-5x annually

@mcauth Is the volume of infrastructure data growing in direct proportion to the size and complexity of the IT infrastructure itself? Or is the amount of infrastructure data growing faster (or slower) than the infrastructure?

@colin_walker I'm curious why management data is growing more slowly (albeit slightly) than application data. Are IT management tools growing more sophisticated in deriving analytic insights for the tasks they perform?

I just think there's a naturally offset correlation. You deal with more bytes in/out as things like HD vid, audio streams, higher graphic content etc. become the norm. But those things don't automatically drive more mgmt data.

@ggilbert41 exactly

I see scale-out app architectures and associated service policies driving the need for increased measurements

Big issue. Integrated ITOM data and tooling implies more integrated roles and orgs. Scares a lot of folks.

One CIO told me: This is the biggest challenge to his business's adopting new technology.

@plburris Yep, people are afraid of change, afraid of retooling not only the compute side, but the human side. Understandable but also necessary.

@ggilbert41 Great framework for sizing IT management data.

@mcauth IT ops is also becoming an ML/AI driven process. Anomaly detection amid petabytes+ absolutely demands precision "pattern-sniffing" in real-time.

@jameskobielus IT Ops will begin to look more like IoT as more intelligence and decision ability is put into end systems

@readyforthenet Or perhaps it's the other way around :) IoT looks more like ops. Either way, fascinating perspective.

@jameskobielus ...and effective feature extraction.