SparkDeepLearning

Deep learning isn’t for free: there’s a ton of parameters that have to be tuned: number of layers, number of neurons, optimization strategy,… . Using Spark, you can run several of these models in parallel, and pick the best one.

Deep learning libraries are being open-sourced. Check out this recent article: http://www.technewsw...

It is great that a lot of innovation in deep learning is happening in the open. Several key frameworks like Torch, Caffe and TensorFlow. Builds community and makes it accessible

Deep learning has already been open sourced in the Spark and Java communities. DeepLearning4J: http://www.spark.tc/...

There's still work to do to run a single DL model in parallel...deeplearning4j does that somewhat by splitting up the data

this is arguably an area where Apache Spark is relatively weak. However it still provides a great base for large-scale training, and other open-source libraries (such deeplearning4j) build on Spark to scale out learning.

#Spark excels in distributed computing DL training process itself is not naturally distributed, but designing ways for #DeepLearning to train and calculate in parallel is an active R&D area as DL permeates industry applications

It is extremely hard to scale out the training of DL models, however, it is common to use the distributed computation for the trained model evaluation and real-world production usage.

Apache Spark represents a fast start for people and organizations to embrace deep learning and machine learning. #SparkDeepLearning

The open-source community : plenty. All the major frameworks (Keras, Tensorflow, Torch, DeepLearning4J, mxnet) are open-source today. Spark has a lot of potential for helping with building deep learning models, but many things to improve still.

however, the field is moving super fast so there is not a governance model yet

Check out Francois Garrilot's talk on DeepLearning4J and Apache Spark: https://www.youtube....

Even though DL does a lot of feature creation and selection "under the covers", Spark has still a place for preprocessing the data

Open Source embraces the collective intelligence and minds of thought leaders and experts on Deep Learning to the benefit of the data science community

Check out IBM's distributions of leading Deep Learning software frameworks: https://download.bou...

@jameskobielus Thanks for the plug !

Deeplearning4j is the first commercial-grade, open-source, distributed deep-learning library written for Java and Scala. Integrated with Hadoop and Spark, DL4J is designed to be used in business environments on distributed GPUs and CPUs.

There's a lot of existing work for parallelizing deep learning training, gathered under the moniker of 'parameter server'. There's some specific work to do to bring Spark towards cooperating with this model.

Here's the link to DeepLearning4J.org: https://deeplearning...

- one of the common challenges is software development FrameWorks, Spark provides one we've fallen in love with, because we grew to loath MapReduce ;-)

@jameskobielus It's also used in "Deep Learning" by Josh Patterson and Adam Gibson, which I find a very good introduction from real practitioners

- opensource.. well there's a religious long running debate on Open vs Closed source software, but the single greatest value proposition with OpenSource is "you can get the code and modify it yourself for free"

- IBM has done an amazing job of re-inventing itself since the days of being a Hardware company, to being not just a Software company, but an OpenSource software company, and the Right sets of OpenSource, i.e. Apache Spark.

Databricks recently added support for integrating TensorFlow via TensorFrames. That wouldn't have happened without both being open source

- if you take the recently announced and now generally available ( for less than lunch money ) Data Science Experience #DSX it's OpenSource like Jupiter Notebooks + Spark "done right" and made easy for everyone to gain access to..

but Google and Facebook too. Example: TensorFlow and Natural Language Analysis API

- I also love the idea that IBM's taken the right blend of OpenSource and built "solutions" which were beyond the reach of most #CitizenDataScientists and let folk do great things without having to "compile the platform" ;-)

Seeing much great adoption - which implies acceptance.

#DeepLearning applications in Natural Language Understanding techniques as well as Q-learning in robotics are changing the game. Coupled with advances in image recognition with ConvoNets, it is no wonder there is so much interest in AI now.

It's being embedded in so many applications. Consumers are not even aware they are using it in their every day interactions #SparkDeepLearning

The rapid improvement and inexorable commoditization of low-cost hardware platforms--especiallly GPU and neurosynaptic chips--to execute compressed deep learning artificial neural nets inside everything.

Customer sentiment, NL Processing, Visual Recognition

it's tough to pick one as the pace of advancement is so fast. But for me the success of DL in reinforcement learning (e.g. AlphaGo, game-playing bots etc) is one of the most promising for moving closer to the "AI" that everyone talks about.

Transfer learning, the ability to re-tool a deep neural network trained to learn a task as part of a network working on another question. http://www.theregist...

@jameskobielus Embedded on chips - yes.

Deep learning's embedding into the fabric of every human artifact will proceed apace as new frontiers in nanotechnology, perhaps quantum computing, come to market.

Are we moving toward an IOT of collective conciousness #SparkDeepLearning

Reinforcement learning, made fast for deep learning through e.g. Actor-critic methods, lets you teach a model how to act in a "game" of your invention, just by telling it its score on its plays.

@MikeTamir Definitely robots

Also seeing in massive online gaming

GPU acceleration for Apache Spark is essential for deep learning's broad adoption. See here: http://www.spark.tc/...

OpenSource platforms (software and hardware) and APIs for AI development

IBM's deep learning GPU partnership with NVIDIA for Power System: https://www-03.ibm.c...

@huitseeker In the DeepLearning4J world, that's called RL4J https://github.com/d... (using A3C)

IBM's work with Lawrence Livermore Labs on the TrueNorth synaptic chip tech: http://www-03.ibm.co...

That there's so much interest in using it for language understanding, moving off the "classic" image / video analysis tasks

a more subtle transformational point is the potential for fast, *open* experimentation, collective learning, the ability to repeat and build on each other's experiments and results. DL+Cloud+Open

That reminds me of Airbnb's recent move to put more of their data, as well as analysis notebooks, on github. Not DL, but certainly open beyond "open data"

- for me, the greatest "trend" has been toward Human focused value adds, ways to make life better for humans, and not just making more "things" but solving real world "problems", Health, Life, Big Challenges to help Humanities as a whole

#DeepLearning4J (which is getting some nice plugs) is a great option especially for Java developers. #TensorFlow has of course become very popular, #Keras is nice for data scientists getting started.

I'm biased - but Watson of course https://www.ibm.com/...

Not to put too fine a point on it, but I recommend PowerAI for OpenPower: http://www-03.ibm.co...

Deeplearning4J is on the JVM, and it's young but it already has a small open-source community around it and an upcoming book to help onboard new users http://shop.oreilly....

Currently there are many competing open-source frameworks - TensorFlow, Keras, mxnet, Caffee, Torch, DL4j and so on. Most use one or two of these for their work. GPUs are more important for scaling than clusters (up to a certain problem size)

Caffe: https://download.bou...

Theano: https://download.bou...

@huitseeker I like that book a lot, currently reading it on Safari...

Torch: https://download.bou...

There are lots as @MLnick notes. Different ones now are better at different domains or appear to different programming model preferences

Another cool lib is Keras, a simple front-end to other frameworks. You can reimplement a fresh paper in a few dozen lines https://github.com/u... ... did you know that in the DL4J ecosystem, Keras is known as ScalNet https://github.com/d... ?

Good article here http://www-01.ibm.co...

Agree with @NxtGenAnalytics - if you want to analyze images, you can use the Watson API right away

@NxtGenAnalytics - ha.. yea ok, we'll let you have that one ;-)

Deep Learning is super heavy in amount of data and power of compute. So you need accelerators. The cloud also brings exciting scale-out options and ease of use and experimentation

OpenBLAS: http://www.openblas....

@dez_blanchfield LOL

Those with truly large problems tend to have their own in-house frameworks custom built. For the rest, TF, mxnet & DL4J are ones that offer parallel training using clusters of CPU/GPU

DeepMind: https://deepmind.com...

@MLnick - that's the magic of OpenSource really isn't it, lots of us get ideas, we test those ideas in code, regardless of if it works for us or not we can #share that code and make it #opensource to let others build on and move faster ;-)

Also if you plan to train on a lot of data, note that I'd pay special focus on libraries that have a built-in solution for distributed training. So far, I'm aware of CNTK, mxnet, DeepLearning4J, and Tensorflow.

Watson Personality Analyzer is interesting - just don't show your spouse the analysis!

For good measure, though it's not a library/tool for development, here's OpenAI: https://openai.com/b...

TensorFlow: https://www.tensorfl...

- a great thing about the OpenSource work is that so many folk come from different coding backgrounds and build in their favourite or native language, i.e. Python, Java, C, C++ / C#, Scala, and you end up with lots of tools & lots of lanuages

@NxtGenAnalytics - ha.. laugh out loud, that's funny..

- i.e. Caffe's a deep learning library with Python & MATLAB bindings, it's quickly becoming a popular "go to" for a rapidly growing community..

I don't think that most people would consider spark for deep learning. Machine learning, yes, but deep learning support is barely there.

I worry that unless it is consumable deep learning will remain out of reach for the majority. Need to bring it to the masses with consumability, convergence, collaboration in mind

@huitseeker it is possible to distribute others with a parameter server based approach. that's what mxnet uses within (CMU ps-lite). for caffe, it has been done by petuum.

- lots of people new to the game forget that Wolfram Mathematica has been in this space for a lifetime ;-)

- anyone here using Torch ( written in Lua )

- Python nuts like me love Theano as well ;-)

- I'm also a fan of Keras ( yea yea Python nut again ).. Keras uses both TensorFlow and Theano

- one I've started playing with recently is a JavaScript implementation of neural networks called ConvNetJS

- TensorFlow is another certain Python nuts are crazy for because of it's BIG brand origins and proven successes thus far..

And what about FBLearner Flow (Facebook AI Backbone)?

@huitseeker I also have to mention that there are some completely different choices depending on the situation. Keras is awesome for experimenting on a rainy weekend. DeepLearning4J aims at production models, and working well with your Hadoop / Spark.