BigData

the Hadoop adoption curve has not peaked yet. Still room for growth. Use case will define the best tool.

Spark is about fast processing. Hadoop is about distributed data (files) access for processing. They co-exist.

There is a school of thought that if you go all in on #ApacheSpark, you don't even need #Hadoop core: #HDFS and #YARN. but storage layer is key for hand-offs between jobs when Spark isn't the end-to-end processing engine

You want to use the right "tool" for the job, depending on data type, batch vs. real-time, etc.

This was an interesting article I read a few weeks back. High level, but makes the point of co-existence. http://www.forbes.co...

@vGazza Hadoop is becoming a key part of enterprise infrastructure. Some part of it - at least #HDFS and #YARN should be common foundation - for a while. #ApacheSpark will have its own take on storage integration at some time

@ggilbert41 what other object stores can #ApacheSpark use as persistence layer other than #HDFS?

@ggilbert41 all in on Spark to some means Databricks - which has its own stack all the way down to the metal

Spark can be MUCH faster than Hadoop processing, so Spark is red hot right now. That's fine, but Hadoop is not going away anytime soon.

@ggilbert41 George & all - where are you seeing Spark take off? What use cases and/or industries

@ggilbert41 Great comment about the Databricks implementation, which you get on @MapR. I would buy that for just that reason.

@jscheplick in the press!

@jscheplick there are a growing number of data tools ISVs who are using it as their data management back-end - #Zoomdata is a great example. also Internet services vendors who have the skills are working with ML and streaming

Start with a truck (Hadoop), move to a pick-up (Spark), end with an max plaid Tesla (streams)

Spark's in-memory efficiency great for complex network analysis across heterogeneous data sources

#ApacheSpark makes a lot of analytics easier and faster by running different workloads on the same engine. Still needs more performance improvements. #IBM contributions could really change things

Horses for courses; Hadoop is batch and most efficient/greatest throughout. Spark is microbatch, gets an answer quicker, but less efficient/slower throughout.

@dfloyer Is Hadoop the new tape :-)

Means you can develop for scale but still work on tab files

@dfloyer Project Tungsten for #ApacheSpark should get them to pure streaming but that will never completely replace need for batch and #Hadoop

@dvellante HDFS is the new long term storage media Mapreduce and Spark can read from

Spark (fast, in-memory) vs Hadoop (batch) = the TWIN TOWERS of the Lord of the Things (#IoT): my new @MapR blog will discuss this

@ggilbert41 Completely agree. There is a role for both

Infostreams is real-time, can be supported in development by Hadoop and Spark

@KirkDBorne loved the Twin Towers analogy! #LOTR

@KirkDBorne the debate about running all analytics on fast, in-memory (i.e. Spark) is likely misleading

@KirkDBorne the analysis around throughput / volume is likely to have different logic than analysis around per event updates

@ggilbert41 Memory x Flash x Disk is a matter of cost per capacity and potential revenue from insight

@ggilbert41 Great SoI deck. Can you elaborate on why Spark is at "slow" end of innovation axis on Slide 24? Would think it is high innovation level

@dvellante > Hadoop = Tape - Love the analogy. What is "spinning rust?" in this model?

@KevinPetrieTech good question: it was a tough call - but having the wild west ecosystem of databases or even the Hadoop ecosystem means each component can evolve independently at own pace. Spark libraries must evolve to integrate with each other

@ggilbert41 Got it. Spark has arisen quickly, but nature of Spark libraries throttles future innovation vs. other platforms

what about the #dataocean ?

See my article on the data lake, sea, ocean, flood, tsunami, stream,... https://www.mapr.com...

#datalake is repository for unrefined, uncurated data that data scientists and biz analysts can explore. repeatable analytics can go to DW or a production #Hadoop cluster

it's about the schema. Data Lakes are flexible, Data Warehouses are rigid.

Data lakes should be easier to adapt to changing business & infrastructure needs vs. EDW

@vGazza also correct. the flexible schema is part of making the #datalake a self-service environment - you add the schema as you explore

Diverse multi-source heterogeneous data sets are the norm in Data Lakes, but not in Data Warehousing.

Data Lakes support schemas on demand, so you can improve the models continuously

Data warehouses and data lakes are slow and sound so old..what's new & different with Systems of Intelligence models ?

Data Warehouse = schema on write.. Data Lake = schema on read.

@CrowdCaptain Data Lakes support flexible ingestion and insight layers, so they can be fast also

Data Lake allows easy updates (data "columns"). DW requires new schema and index builds when adding new

Data lakes retain all (more) data vs. DW

ok it's all about the #dataocean bc oceans have currents and are always highly dynamic so the real time intelligence algos and tech are in the #dataoceans

DW are archived hierarchially... Data Lakes are mostly object-based

@vGazza flexible ingestion enables self-service, but repeatability invites structure and therefore performance

@bigdataryan care to chime in?

We are talking a lot about technical differences, but what is the difference for business value?

@CrowdCaptain #SystemsOfIntelligence should derive their value from its apps (smart Machine Learning) not its data model (DW or Data Lake)

data lakes are just a cheaper and bigger version of the failed data warehouses model

@KirkDBorne it's all about flexibility vs. performance trade-off

a Data Lake can be a source of data for a Data Warehouse once you know what to ask

@KirkDBorne Kirk, i couldn't have said it better. Data Lakes are training wheels. #MachineLearning is what drives Systems of Intelligence

@ggilbert41 systematic queries can be optimised and run on Data Warehouses for performance

data streams, data rivers power things along. Data lakes fester.

@vGazza that's why DataLakes coexist with DataWarehouses - exploration vs. production performance

Completely new methods needed for Data Lakes. #Machine Learning and stats miss many of the possibilities

@CrowdCaptain > Getting out front in the decision process.