SparkMachineLearning

I'd like to see even more and richer feature transformations. This is 80% of the work in predictive modeling.

Are there any hurdle? Really? Just kidding.

In my view these are: usability, scalability and deployment. Spark ML pipelines are great but still need improvements in areas of usability (APIs etc) and scalability (especially in terms of "wide" datasets with many features)...

I guess main hurdle is that there isn't enough built-in feature engineer and algorithms....

As @JFPuget said: Spark is king when your problem is distributeable. One of my co-workers is looking at an NMF problem right now, wher "locality is king". This makes using Spark trickier

One of the big issues in Spark ML is deciding which machine learning algorithms to include. Adding more algorithms can be useful, but each one adds a large amount of code maintenance overhead to the project.

... the fix is to add many more at each Spark release, including 2.0

... a constant issue coming up with Spark ML users is deploying pipelines to production. While Spark supports save/load in Spark, this does not solve the problem of pushing your trained pipeline to a real-time, low-latency serving environment

No, @JFPuget and I didn't coordinate our answers beforehand.-)

We need a standard to store pipelines and models directly from Spark and other ML frameworks.

@JFPuget Like PMML "2.0" ?

another hurdle in the big data frameworks(not specific to ML) is the challenge of efficiently test the jobs the developer is designing before applying to a huge dataset. You want tools to quickly identify the potential problems in the job

@dusenberrymw I'd worry about Spark "code bloat" where you're adding yet another ML algorithm to an existing app. Here's a column I wrote a few years ago on this topic: http://www.ibmbigdat...

@alexlang11 or PFA (Portable Framework For Analytics), a Json format

@MLnick Yeah "wide" datasets seem to be a more general problem with Spark.

@johngouf Spot on! We face this with a GraphX app we're building right now...

@dusenberrymw I'd like to see all scikit-learn algorithms...

@dusenberrymw And while you're at it, start with matrix operations.-)

@JFPuget In terms of stadnards for storing pipelines and models directly form Spark and other ML frameworks, Ben Lorica had a good article a few years ago here: https://www.oreilly....

@dusenberrymw Could you elaborate on the problems with "wide" datasets you've seen?

@alexlang11 Curious to have another open discussion about GraphX to hear your thoughts!

Here's a column I wrote on standardized ML pipelines: http://www.ibmbigdat...

@alexlang11 Yeah I agree! Making it easier to write distributed ML algorithms based on matrix operations is the goal with the Apache SystemML project that sits on top of Spark.

@alexlang11 I for one have seen issues with the OneHotEncoder on high-cardinality categorical features. Also the fact that many transformers only operate on one column is an issue.

@johngouf GraphX works, but it's the "works well on the developers laptop, runs into OOM for a very large dataset on the server" problem. We have to get better at "debuggability" of that...

@dusenberrymw I actually think that the right approach is for that NOT to live in Spark core - I think the SystemML idea of building matrix math on top of the core primitives and focusing on that is making more and more sense

@alexlang11 The issue I've found with "wide" datasets is that the size of each row is much larger, and so the number of partitions that the data is split into needs to be much higher to prevent hitting Spark limits.

@dusenberrymw Additionally, I've found that having larger rows incurs a much higher cost to several DataFrame/DataSet operations.

@alexlang11 It was what I imagined as well....Encountered the same issues for SparkSQL, but in version 1.6.x

@MLnick Speaking of scaling, here's a link to Nick's upcoming talk at Spark Summit Europe on scaling factorization machines on Spark using parameter servers: https://spark-summit...

To learn how expert Spark developers are addressing these ML hurdles, attend the meetup in Brussels on 27 Oct: http://bit.ly/2ecmY7...

It's a pretty big "niche" to be sure...

Machine learning's optimal niche is as the core approach for automating more of the distillation of patterns from unstructured and complex data, both structured and streaming.

Full ack with @jameskobielus: By applying ML models, you can operationalize complex insights from data patterns: rules can't express that

Machine learning's niche is to support development of apps that automatically "learn" from fresh data--in other words, it's the intelligence behind cognitive computing.

it is a big "niche" - but essentially ML should be applied whenever (i) humans won't or can't scale to the problem (speed, data volume, complexity etc) and/or (ii) when the ML model can perform as well or better than the human.

Machine Learning is all about making predictions on new data, and learn from wrong predictions.

Machine learning's niche is to detect the data-driven environmental "signals" that drive artificial intelligence applications. Autonomous vehicles, for example, would be impossible without ML.

Other analytics may stop once a model of input data is built.

Saw this today: http://idlewords.com...
"ML is like a deep-fat fryer. If you’ve never deep-fried something before, you think to yourself: "This is amazing! I bet this would work on anything!”

Here's a recent blog I wrote on the "power apps of machine learning": https://www.linkedin...

@JFPuget "and learn from wrong predictions" is absolutely key! Otherwise, you get stuck in a negative reinformcement loop

@alexlang11 I'd be careful. Often one combines rules and machine learning to take action.

Another untapped opportunity that I see, is how to support ML algorithms expressed with the same APIs to support a diverse variety of underlying nodes/slaves, like RaspPi, Nvidia boards, android devices...

ML is also particularly nice when the problem has a larger amount of variability that is perhaps difficult to encapsulate in a simple rule-based system.

@MLnick .. and (iii) you have eexampels of what needs to be learned.

@JFPuget We actually do that as well! Let me officially retract from my statement above

@JFPuget yes true! Though anomaly detection and some unsupervised techniques can still be applied :)

@alexlang11 I'm sure you do ;)

@JFPuget Yeah that's key in a more general sense. ML itself won't solve problems without definition; rather the application of ML to a problem is when it becomes useful.

@MLnick Agreed, but these are not mainstream use of ML IMHO.

@MLnick supervised learning is easier to leverage than unsupervised on to make decisions.

@dusenberrymw Exactly.

@JFPuget I think it's also easier for non-ML users to spot supervised problems -- essentially I want to be able to "predict this" given some data.

@dusenberrymw yes, supervised problems are easier to define.