bigdata

in the mid- to late-90's #ERP, internet infrastructure sw revenue grew triple digits

but they all crashed into a wall starting in '98 because customers had bought much faster than they could deploy - some in anticipation of #Y2K

there is a similar sense today of "fear of missing out" driving #bigdata purchases while the technology is still maturing

that's what our survey data and research in the #bigdata ecosystem tells us

for example - between spring '14 and fall '15, those describing their deployments as "successful" has grown from 41% then to 45% now.

So do you see the Hadoop market crashing @hortonworks is releasing earnings today

part of the push for triple digit but unsustainable growth in the late '90s came from the vendors themselves

enterprise #software could cost well over 50% of every $ in revenue because of cost of paying expensive sales reps

the problem of the cost of sales is worse today because #opensource #software doesn't come with the huge upfront licenses that came with software in the '90s

I think hadoop is still messy but the foundational elements are in place now to extend to other systems that should propel growth; what do you think?

so vendors have an even greater motivation to sell more to the customer than they can absorb in any near term period

#hadoop isn't a product but rather an ecosystem. parts of that like #yarn and #hdfs are ubiquitous and will be a foundation as the #analytics stack grows and matures

for customers to get past complexity, we are likely to see convergence of more components

for example, most #hadoop vendors are placing #apachespark as the core of their #analytics frameworks while keeping the more mature management and admin bits from hadoop

other vendors like #aws #azure #googlecloudplatform are further simplifying #hadoop and #bigdata #analytics

public cloud is going to big the big wildcard. most customers and vendors regard them as direct competitors to #hadoop right now

but in our survey an equal % used #hadoop and native #aws #azure #googlecloudplatform services - both with roughly 37%

this question specifically asked about the use of public #cloud for #bigdata #analytics

the reasons for use of public #cloud were 58% data already there and 54% for operational simplicity #TCO

44.5% said public #cloud gave developers #analytics tools that were easier to build an end-to-end application with versus on-premises

AWS is crushing it bc they have the app marketplace; IBM is right behind them building fast

if customers can get the data to the public clouds (either natively or thru transfer), the path to get started and maintain the systems is much simpler. tons of working big data examples on public clouds.

. @bgracely totally agree but those apps are not #hadoop but other underlying tech #integratedstacks

AWS EMR is Hadoop.

@bgracely is EMR considered an app?

EMR is foundational, whereas Kinesis or Redshift would be considered apps. either way, they are all simplifying (or operationalizing) the part customers have problems getting to work.

is CrowdChat a Hadoop or Big Data application?

@bgracely CrowdChat is a dynamoDB app and a big data app on AWS

@bgracely yes, and a key word is "if". To @ggilbert41's point, it makes a lot of sense if the data is already there

the #bigdata revenue numbers we see today from individual vendors aren't as big as the late '90s, so we may have headroom but...

when there is rampant and steep discounting in a market as part of a land grab, that's often an indicator that we're close to a correction

in addition, there are way too many vendors - public and private - that are spending so much on sales & marketing in this land grab that they are consuming significant amounts of the cash on their balance sheets

that provides a potential vendor-induced retrenchment if the capital markets no longer accept heavy operating losses without evidence of future profitability

it's not going to be easy to find many canaries in the coal mine because most of the vendors are private...

many thanks this makes sense. i sometimes hear commentary that cloudera has deep pockets but its all become too complicated & the code base has become too messy. its not as simple & clean cut offering like it was a while back - does any of that ring true?

thanks for your commentary?

but maybe #hortonworks which reports today can provide some insight into what's going on in the #bigdata ecosystem more broadly

#cloudera started life with the objective of making #hadoop easier to manage and they've done a good job. but the #cloud competition may be most intense because those vendors live to simplify and have the infrastructure built-in to do it

we surveyed 300 practitioners in U.S. this fall and also spring 18 months ago

we surveyed enterprises that had deployed or were evaluating a #bigdata #analytics project

we defined #bigdata #analytics to include technologies and data that traditional #sql #databases couldn't address

full methodology and data at #wikibon http://wikibon.com/w...

what are customers saying? will hadoop blowup or grow?

#hadoop isn't going away anytime soon - it has bright prospects

. @ggilbert41 can hadoop pure play vendors survive or do they have to do more?

the problem with #bigdata software currently is that vendors are selling it and customers are buying it faster than they can possibly deploy

@ggilbert41 do vendors see that as a problem, because they can't monetize the apps/customizations yet, or just viewed as a customer problem?

@bgracely great question: i think we are likely to see both sides recognizing this as a problem

@bgracely customers are essentially piling up "inventory" of undeployed licenses and vendors are facing a "channel" that's filling up

@bgracely the problem is that customers may start wanting to deploy more in the #cloud for simplified admin and overall TCO

@bgracely in addition - they may see the cloud native services such as #aws #kinesis #redshift #machinelearning as more accessible to developers and even simpler to operate

. @bgracely to me there are not enough big data apps out there on hadoop imho that is the big issue

this is why you're seeing things like AWS QuickSite (https://aws.amazon.c...) get created because companies have existing "Data Scientists on Excel" that could use simpler applications and shareable tools/data.

also #ibm #watson #expertstorybooks are "notebooks" like we're seeing spring up in the #apachespark community and lately #hadoop too

these go a long way toward simplifying #datascience and application building. #ibm calls them citizen #datascientists