Overview of Machine Learning in the Wolfram Language

Etienne Bernard, Wolfram Research

Join the Conversation #WolframTechConf

Machine Learning?

Computer “learns” a model from data

⟶

⟶

{ ->cat, ->dog,...}	⟶

Learning types

Supervised learning: input -> output from examples

Unsupervised learning: only inputs

Reinforcement learning: agent acting in an environment

Typical tasks

Recognition of images, sound, text, etc.

->cat, ->dog,...

Prediction of a variable in a dataset

(6 yo,female)->1.15 m,(11 yo,male)->1.48 m,...

Prediction of a time series

Text translation

"the house is blue"->"la maison est bleue"

Missing values imputation

->

Anomaly detection

Data clustering

Compression

Error correction

"the housse is blue"->"the house is blue"

Learn to evolve in an environment

Data mining tasks

In the Wolfram Language

Learning functions

Predict & Classify

FindDistribution & FindFormula

DimensionReduce

...

Learned functions (built-in models)

Built-in classifiers and predictors

ImageIdentify

TextCases, TextStructure

...

Classify & Predict

Supervised learning functions

Input can be images, text, sound, numerical data, etc.

Output can be:

Categorical variable (class) -> Classify

Numerical variable -> Predict

Automatic

Classify & Predict - Examples

Classify

c=Classify[
{->2,->5,->8,->0,->2,->7,->5,->1,->3,->0,->3,->9,->6,->2,->8,->2,->0,->6,->6,->1,->1,->7,->8,->5,->0,->4,->7,->6,->0,->2,->5,->3,->1,->5,->6,->7,->5,->4,->1,->9,->3,->6,->8,->0,->9,->3,->0,->3,->7,->4,->4,->3,->8,->0,->4,->1,->3,->7,->6,->4,->7,->2,->7,->2,->5,->2,->0,->9,->8,->9,->8,->1,->6,->4,->8,->5,->8,->0,->6,->7,->4,->5,->8,->4,->3,->1,->5,->1,->9,->9,->9,->2,->4,->7,->3,->1,->9,->2,->9,->6}]

{,,,,,,,,,}

Predict

p = Predict[{->40.2,->8.9,->11.,->4.9,->13.6,->15.6,->14.7,->3.8,->34.7,->10.8,->4.,->16.1,->3.3,->8.3,->12.6}]

{,,}

More about:

Example of a complete machine learning project:

http://blog.wolfram.com/2014/06/20/predicting-who-will-win-the-world-cup-with-wolfram-language/

http://blog.wolfram.com/2014/06/26/world-cup-follow-up-update-of-winning-probabilities-and-betting-results/

Workshop - Thursday at 3:30 pm - “Building Smart Applications That Learn”

Formula discovery

FindDistribution

d = MixtureDistribution[{0.3,0.7},{ExponentialDistribution[1],NormalDistribution[5,0.8]}];

data=RandomVariate[d, 1000];

Histogram[data,30,"PDF"]

e= FindDistribution[data]

Show[Histogram[data,30,"PDF"],Plot[PDF[e,x], {x,0,10}]]

FindFormula

data =Table[{x, Cos[x]+Sin[2x]+RandomReal[{-.2,.2}]}, {x, RandomReal[{0, 10}, 200]}];

ListPlot[data]

f=FindFormula[data,x]

Show[ListPlot[data], Plot[f,{x,0,10}]]

More about:

Talk of Giorgia Fortuna  - Thursday 1pm - “Automatic Formula Discovery in the Wolfram Language”

Dimensionality reduction

DimensionReduce

Load the Fisher iris dataset:

iris=ExampleData[{"MachineLearning","FisherIris"}, "Data"];
RandomSample[iris,5]

Reduce the dimension of the features:

features=DimensionReduce[iris[[All,1]],2];
features[[;;10]]

Visualize the reduced dataset:

byspecies=GroupBy[Thread[features->iris[[All,2]]],Last->First];
ListPlot[Values[byspecies], PlotLegends->Keys[byspecies]]

DimensionReduction

Load the MNIST dataset:

digits=ExampleData[{"MachineLearning","MNIST"}, "TestData"][[All,1]];
digits=Flatten[ImageData[#]]&/@RandomSample[digits];
Dimensions[trainingset]

Separate the dataset into a training set and a test set:

trainingset=digits[[;;9000]];
testset=digits[[9001;;]];

Create a dimension reducer from the training set with a target dimension of 50:

dr= DimensionReduction[trainingset,50]

Reduce a vector from the test set:

vector=RandomChoice[testset];

dr[vector]

Visualize the original vector and its reconstructed version:

toimage=Image[Partition[#,28], ImageSize->Tiny] &;

toimage /@ {vector,dr[vector,"ReconstructedVectors"]}

Replace some values of the vector by Missing[] and visualize it:

vectormissing=vector;
vectormissing[[309;;364]]=Missing[];
imagemissing = toimage[Replace[vectormissing, _Missing-> .5,{1}]]

Impute missing values with the reducer function:

imputedimage=toimage[dr[vectormissing, "ImputedVectors", PerformanceGoal->"Quality"]];

{toimage[vector],imagemissing,imputedimage}

Computer vision

ImageIdentify

Deep Convolutional Neural Network

Trained on ~(10^7) images

More about:

Talk of Sebastian Bodenstein - Thursday 11 am - “A Shallow Tour of Deep Learning”

Natural Language Processing

Built-in classifiers

TextCases - “NLP Swiss-army knife”

TextCases["Mrs. Fox is the CEO of the company.  She co-founded the company with Mrs. Smith.", "Sentence"]

TextCases["John and Mary went to the store and bought a table.", "Noun"]

TextCases["John and Mary went to Italy.", "Country"]

TextStructures

struc=TextStructure["The house is blue. My dog is fast."]

TextStructure["The house is blue. My dog is fast.", "PartOfSpeech"]

Current projects

“Deep-learned” functions

Specialized image classifier

Speech recognition

-> "How are you?"

ImageCases

image-net.org

Neural Network Framework

Train and evaluate neural networks

CPU & GPU computation

Should be there in V11...

Unsupervised Learning

ClusterClassify

-> (CopyTag[ClassifierFunction[]])

FeatureExtract

FeatureExtract[{,...}] -> {{1.3,5.6, ...}, ...}

Anomaly detection

Missing data imputation

Density estimation

Error correction

Bayesian optimization

“Model-based” optimization

Good for computationally-intensive functions

Automation tool

(Brochu et al. 2010)

Text functions

TextCases

Topic identification

Language translation

Future projects

Multi-output predictions

Supervised learning with various output type (numerical & categorical vector, sequence, text, etc.)

Any combination of output types

Predict

Probabilistic Graphical Models

Function that learns structure + parameters automatically

Tool to define and play with models

Koller & Friedman 2009

Probabilistic programming

Wolfram Language restricted to basic functions

Need a fast inference engine

Automatic data analysis

Interpretable model

Natural language report

Lloyd et al. 2014

Natural Language Processing

Text summarization

Unsupervised topic modeling

Fact extraction

Other learning types

Active learning

Reinforcement learning

Scalability & Deployment

GPU computation

Online algorithm for out-of-memory computation

Exportation of models to other languages