Workshop: Machine Learning

Etienne Bernard

Wolfram Research - Advanced Research Group

MNIST

Load the MNIST dataset from ExampleData

Plot a learning curve to figure out how much data we need

Plot the confusion matrix of the classifier on the ExampleData testset

dataset =ExampleData[{"MachineLearning","MNIST"}, "Data"];

dataset = RandomSample[dataset];

Length[dataset]

ByteCount[dataset]

RandomSample[dataset, 10]

dataset[[;;10]]

trainingset = dataset[[;;50000]];
validationset = dataset[[50001;;60000]];
testset = dataset[[60001;;]];

accuracies=Table[{i, ClassifierMeasurements[Classify[trainingset[[;;i]]],validationset, "Accuracy"]},{i,{10,20,50,100,200,500,1000,2000}}]

ListLogLinearPlot[{#1,1-#2}&@@@accuracies, PlotTheme->"Business", Joined->True]

c=Classify[trainingset[[;;10000]]]

cm = ClassifierMeasurements[c, testset]

cm["Properties"]

cm["ConfusionMatrixPlot"]

cm[{"Examples",5->6}]

House prices in Boston

Generate a PredictorFunction with the BostonHomes dataset from ExampleData

Analyse its performances on a testset

Try other performance goals, methods and compare performances

trainingset = ExampleData[{"MachineLearning","BostonHomes"}, "TrainingData"];

RandomSample[trainingset, 2]

testset = ExampleData[{"MachineLearning","BostonHomes"}, "TestData"];

p=Predict[trainingset, PerformanceGoal->"Quality"]

PredictorInformation[p]

PredictorInformation[p, "Properties"]

PredictorInformation[p, "LeafSize"]

PredictorInformation[p, "Options"]

pm = PredictorMeasurements[p, testset]

pm["Properties"]

pm["ComparisonPlot"]

pm["ComparisonPlot"]

pm["StandardDeviation"]

pm["StandardDeviation"]

Create a classifier to predict the outcome of football matches

Can we outperform betting odds?

http://blog.wolfram.com/2014/06/26/world-cup-follow-up-update-of-winning-probabilities-and-betting-results/

http://www.oddsportal.com/soccer/world/
http://www.11v11.com

http://wolfr.am/ml-workshop

dataset = Get["~/Dropbox/WorkshopData/dataset.m"];

odds = Get["~/Dropbox/WorkshopData/odds.m"];

keys = Get["~/Dropbox/WorkshopData/keys.m"];

dataset[[-1]]

Length[dataset]

trainingset = dataset[[;;-1000]];

testset = dataset[[-999;;]];

c=Classify[trainingset]

ClassifierInformation[c]

cm=ClassifierMeasurements[c,testset]

cm["Accuracy"]

cm["ConfusionMatrixPlot"]

testset[[1]]

c[testset[[1,1]], "Probabilities"]

c[testset[[1,1]],"Probabilities", ClassPriors-><|"Draw"->0|>]

ClassifierInformation[c,UtilityFunction]

cm["LogLikelihoodRate"]

odds[[{1}]]

bettingUtilityFunction[team1odd_, team2odd_, drawodd_] := 
	Module[{classes},
		classes = {"Team1", "Team2", "Draw"};
		Map[
			Association[Thread[classes -> #]] &,  
			DiagonalMatrix[{team1odd, team2odd, drawodd}],
			{0, 1}
		]
	];

indexeddata=AssociationThread[keys,dataset];

matches=Lookup[indexeddata,Keys[odds]];

Length[matches]

Length[odds]

matches[[1,1]]

odds[[1]]

bets=MapThread[c[#1, UtilityFunction->bettingUtilityFunction[#2["Team1"],#2["Team2"],#2["Draw"]]]&,{matches[[All,1]], Values[odds]}];

wining = SameQ@@@Transpose[{matches[[All,2]],bets}];

gains = MapThread[#1[#2]&,{Values[odds],bets}];

wininggains = Pick[gains, wining]

Length[wininggains]

Total[wininggains]

Length[bets]

202.8-231

-28.19999999/231.

During the workshop, some odds were accidentaly in the wrong order. With the correct odds, it appears that we lost 12% of our stake... However, the betting strategy is not optimal.

Loading

Quit[]

PacletDirectoryAdd["~/Work/eclipse/Mathematica/ARG"];
PacletDirectoryAdd["~/Work/eclipse/Mathematica/ARG/Classifiers"];
PacletDirectoryAdd["~/Work/eclipse/Mathematica/ARG/Predictors"];

Quiet[Get["MachineLearning`"]]