SparkInsight

Please post your replies here

for sure the need to build ever more sophisticated and lower latency analytic pipelines that inform operational apps at the transaction level

good old fashioned COMPETITION

systems of record could only deliver historical performance reporting - like steering a ship by looking backwards at its wake

There are a number of trends: a move to real-time analytics is starting, there are new devices generating new streams of voluminous data, there are computing devices in the hands of even relatively impoverished people, raw network speeds.

Real-world challenges requiring smarter apps are simply the volume and velocity of data one should take into account, which is beyond what humans can correctly process

Also, for many real-world problems, algorithms have improved to outperform human experts

smarter apps should come with easy to use UI and flexibility to exercise different combinations of analysis

Complexity, we are moving to a new level of maturity with analytics and the problems are tough

there are challenges and room of improvements across all industries. Healthcare, Biotechnology.. now we got potential and capabilities of using huge amount of data to make our apps smarter.

there was always the need to augment end-user decision making. now there is the ability to do it in near real-time and in many cases without a human in the loop - ad-tech, etc

Another thing that is big is the ACA. Now medical records are electronic and human error can be caught, treatments can be judged statistically.

The core real-world challenges are always speed, hence the need for real-time in-memory and stream computing; agility, hence the need for interactive statistical modeling; scalability, hence the need for MPP data analytic platforms.

Healthcare, pharma and cyber are verticals that are now being transformed by big data and data science. Just a few years behind advertising & finance.

We did a demo based on a real-world scenario using Spark and NLP see here http://sparknlp.mamm... source here soon https://github.com/m...

: Startups to giant corporations need data, design and speed to improve their business process through smarter apps . Probably design and speed are the main challenges.

Heard a cool story from a data scientist at UC Berkeley, eliminate blood tests with wearables but culture shock in medical community resulted

@acoliver, thanks for the link! Where did you get the anonymized EMR records for this demo?

The convergence of mobile, social, IoT, streaming, and cloud analytics has driven the dev and adoption of Spark. The ubiquity of these converged apps in e-commerce and marketing has spurred need for smarter, game-changing Spark apps.

@davidtalby The negex page has a zip you can download.

@davidtalby Anonymized EMR records came from the Chapman Negex corpus.

@madiakc Culture shock is something we live every day... change is hard and this tech affects many people. Any tips on how to deal with it?

Time for Question #4, please look at the top of your screen for question #4

The chief challenges in the Insight Economy are changing shapes of engagement, influence, marketing, transactions, and collaboration. The chief enabler is contextualized analytics drawing on machine learning and predictive modeling.

@davidtalby Also I should note that Ian Pointer @carsondial did all the hard work. I just stood around saying "nope not there yet" :-)

: We have a webcast that throws light on how to develop smarter apps fueled by data. This might be a good place to start . http://ibm.co/1eTiuN...

@carsondial and @acoliver - much appreciated!

How do I retweet a LinkedIn post??

Please post your replies to question #4 here

Helps explore previously untapped data like the streaming API

Oh good gosh can NVidia, AMD and Intel get together and commoditize APIs for dealing with GPUs and that all get tied up into Spark?

Lowering the barrier of entry and also helping to improve iteration velocity in Big Data development

Standard APIs are the key to simplicity on the development side. With standards, Spark apps only need to be written to those APIs in order to execute on any platform that runs Spark engines and can, via Spark SQL, tap into HDFS data.

I'd like to see a big focus on making ETL as relatively transparent as possible. Legacy shouldn't be an impairment to adoption.

Helps foster community #IBMStreams integrates with Spark MLlib Toolkit

Having simple, easy to learn & use API's to critical to develop productivity, and also to adoption because that's the first impression for developers trying out a platform.

Spark API's started very simple, and now new layers of abstraction like DataFrame and KeyStoneML are being added as the core gains complexity over time

Developers always more productive when can rely on a single IDE, single set of APIs, single storage/execution platform, and single pool of data and library of algoirthms. Can then focus on statistical/predictive explorations.

Interesting article in HBR https://hbr.org/2015... APIs making business news

@jameskobielus but it is important that this IDE not be slow, not be tied to a single vendors overall platform play and evolve with the times (i.e. git vs other tools...cough..clearcase...cough)

Simplicity must, of necessity, gave way to greater complexity as the Spark open-source code evolves, a la "Tungsten" and adds APIs. When will Spark APIs become too complex to fathom? Hard to say at this juncture in the tech's evolution.

Thank you all! Take a look at question #5 on the evolution of Spark

Look at the top of your screen for question #5

Spark API's are already becoming complex; this is why @Atigeo built and open sourced xFrame: https://github.com/A...

Look at the top of your screen for question #5

Please post your response here

This is the 1st question for our chat today and we request you to post your replies under each question.

I don't fundamentally believe in data scientists as people imagine them. There are good mathematicians who write bad python. There are bad mathematicians with a distributed computing background and there are business SMEs.

To build smarter Spark apps, data scientists need to have domain knowledge of what real-world scenario they're modeling: marketing, customer service, fiinance, HR, or whatever.

is the emergence of notebooks democratizing access to working with spark?

fundamentally I think these are teams of people and not this mythical mad data scientist who knows business, your business, math, AI, distributed computing, codes and walks on water..

To build smarter Spark apps, data scientists need to do their work in collaborative, interactive environments that involve various specialists--domain specialists, statistical analytics, visualization experts, etc--pooling knowledge.

To build smarter Spark apps, data scientists need to build their models from data in "lakes" that include all relevant sources, use appropriate algorithms from common libraries, and establish tight governance over data & models.

in addition to the domain knowledge, spark developers or their colleagues maintaining the operational apps have to build the pipeline that informs the transaction in near real-time

In addition to the hard skills like math and statistics softer skills like communication helps

@jameskobielus To be honest, I actually think the #datalake is a stopgap. In the future we'll have better integration tools that distribute data automatically.

To build smarter Spark apps, data scientists should engage with their peers throughout the open-source community to learn from best practices, identify best tools, and bring the best people to their projects.

@madiakc good observation Kimberly and I agree. Many projects fail due to mis-communication not lack of expertise.

Thank you all for you replies to question #1, now let us go to question #2 at the top of your screen.

Data science consists of a few steps: understand the business goal (isn’t that always true, but often it is overlooked), profile and explore data and prepare data, consult with experts, build/deploy applications, valid & refresh often

1. ask the right question 2. understand the data 3. understand the difference big data vs small data 4. evaluate if Spark is the right option for this specific use case 5. utilise parallelism and in-memory power of Spark 6. code

The smartest Spark app of all is one that addresses a key business problem. Engage with business stakeholders in identifying what sort of analytic apps you wish to build out with Spark tools. What scratches the most urgent business itch?

Thank you all for you replies to question #1, now let us go to question #2 at the top of your screen.