Image Processing Applications:
Classifying images

Giulio Alessandrini

WRI

Color-Based Season Classifier

We try to build a classifier to separate images of the four seasons according to their colors.

Color-Based Season Classifier

The images we need can be acquired through a simple Internet search.
We can write a small function to let us import them from a Google Images result page.

ClearAll[ImportImages]
ImportImages[query_,n_Integer?Positive]:=
Import/@Take[Join@@(StringCases[URLFetch["https://www.google.com/search?tbm=isch&q="<>query<>"&start="<>ToString[#]],"src=\"https"~~x:Except["\""]..~~"\"":>"https"<>x]&/@Range[1,n,20]),n]

The list of images is imported, conformed, and stored locally.

n=100;
seasonsNames={"spring","summer","autumn","winter"};

seasons=ImportImages[#,n]&/@seasonsNames;
seasons=Partition[ConformImages[Flatten@seasons[[All,;;n]],{150},ColorSpace->"LAB"],n];
DumpSave["seasons.mx",seasons];

We can immediately see how these images have different colors’ distributions.
We will try to leverage on this fact in order to build our classifier.

GraphicsRow[MapThread[Labeled[ChromaticityPlot3D[#,"RGB",Ticks->None,ImageSize->Scaled[.8]],#2]&,{seasons,seasonsNames}],ImageSize->800]

Machine Learning—Classify

Our very first try is with the new Wolfram Language super-function: Classify.

Classify can handle images so we could be temped to train the classifier directly on the dataset.

trainingSetSize=Round[2/3 n];

trainingSet=AssociationThread[seasonsNames,Take[seasons,All,trainingSetSize]];
testSet=AssociationThread[seasonsNames,Drop[seasons,None,trainingSetSize]];

SeasonClassifier=Classify[trainingSet,Method->"NearestNeighbors"]

ClassifierMeasurements[SeasonClassifier,trainingSet,"Accuracy"]

ClassifierMeasurements[SeasonClassifier,testSet,"Accuracy"]

Machine Learning—Classify

The dataset is too small, the noise is too much, ...
For some reason the brute force approach did not work right away.

A sensible thing would be to identify some key color information in order to reduce the dimensionality of the phase space.
We can see that the ChromaticityPlot of the images in the LAB color space is well localised in the (a*,b*) subspace—i.e. the color components.

GraphicsRow[MapThread[Labeled[ChromaticityPlot3D[#,"LAB",Ticks->None,ViewPoint->Right,ImageSize->Scaled[.8]],#2]&,{seasons,seasonsNames}],ImageSize->800]

Our new observables are going to be the DominantColors of each image, identified by their (a*,b*) values.

LABDominantColors[image_Image,n_Integer?Positive]:=LABDominantColors[image,n]=List@@@DominantColors[image,n,"LABColor"]

Four colors classifier

nColors=4;
trainingSet=AssociationThread[seasonsNames,
Map[Flatten@LABDominantColors[#,nColors][[All,{2,3}]]&,Take[seasons,All,trainingSetSize],{2}]];
testSet=AssociationThread[seasonsNames,
Map[Flatten@LABDominantColors[#,nColors][[All,{2,3}]]&,Drop[seasons,None,trainingSetSize],{2}]];

SeasonClassifier=Classify[trainingSet,Method->"NearestNeighbors"]

ClassifierMeasurements[SeasonClassifier,trainingSet,"Accuracy"]

ClassifierMeasurements[SeasonClassifier,testSet,"Accuracy"]

Six colors classifier

nColors=6;
trainingSet=AssociationThread[seasonsNames,
Map[Flatten@LABDominantColors[#,nColors][[All,{2,3}]]&,Take[seasons,All,trainingSetSize],{2}]];
testSet=AssociationThread[seasonsNames,
Map[Flatten@LABDominantColors[#,nColors][[All,{2,3}]]&,Drop[seasons,None,trainingSetSize],{2}]];

SeasonClassifier=Classify[trainingSet,Method->"NearestNeighbors"]

ClassifierMeasurements[SeasonClassifier,trainingSet,"Accuracy"]

ClassifierMeasurements[SeasonClassifier,testSet,"Accuracy"]

Adding more colors does not help!

Flat distribution—Region

A convex hull is defined from the distribution of the dominant colors for each season with the function ConvexHullMesh.

nColors=4;
trainingSet=Map[LABDominantColors[#,nColors][[All,{2,3}]]&,Take[seasons,All,trainingSetSize],{2}];

seasonRegions=Table[ConvexHullMesh@Select[season,Norm[#-Median@season[[1]]]<5MeanDeviation[Norm/@season]&],{season,Flatten[trainingSet,{{1},{2,3}}]}];
regionDistances=SignedRegionDistance/@seasonRegions;
seasonColors={RGBColor[0.5,0.76,0.15],RGBColor[1,0.75,0],Brown,RGBColor[0.67,0.87,1]};
RegionPlot[Evaluate@seasonRegions,PlotLegends->seasonsNames,PlotStyle->(Directive[#,Opacity[.5]]&/@seasonColors),BoundaryStyle->Thread[Rule[Range[4],seasonColors]]]

Define a distance function from a set of colors to a region using the new RegionDistance function.

(Subscript[⟨α (1/2)(1+sgn[D(s,c)])D(s,c)+β (1/2)(1-sgn[D(s,c)])D(s,c)⟩, c])

EnsembleColorDistance[colors_]:=
With[
{α=1,β=1/5},
Mean[Map[((1+Sign[#])/2α-(1-Sign[#])/2β)#&,Through[regionDistances[#]]&/@colors,{2}]]
]

The predicted season is the one that minimizes this distance.

ClearAll[SeasonClassifier]
SeasonClassifier[image_,n_]:=With[{colors=LABDominantColors[image,n][[All,{2,3}]]},
MinimalBy[Transpose[{seasonsNames,EnsembleColorDistance[colors]}],Last][[All,1]]]

Test the classifier on the remaining images.

trainingSet=Take[seasons,All,trainingSetSize];
testSet=Drop[seasons,None,trainingSetSize];
nColors=6;

res1=MapThread[({image,season}|->If[SeasonClassifier[image,nColors][[1]]===season,1,0] ), {trainingSet,Transpose[ConstantArray[seasonsNames,Length[trainingSet[[1]]]]]}, 2];

Mean[Flatten[res1]]//N

Grid[{seasonsNames,N[Mean/@res1,3]}]

res2=MapThread[({image,season}|->If[SeasonClassifier[image,nColors][[1]]===season,1,0] ), {testSet,Transpose[ConstantArray[seasonsNames,Length[testSet[[1]]]]]}, 2];

Mean[Flatten[res2]]//N

Grid[{seasonsNames,N[Mean/@res2,3]}]

Full distribution—PDF

nColors=10;
trainingSet=Map[LABDominantColors[#,nColors][[All,{2,3}]]&,Take[seasons,All,trainingSetSize],{2}];

seasonDistributions=SmoothKernelDistribution[Select[Flatten[#,1],Norm[#]>.05&],"Oversmooth","Gaussian"]&/@trainingSet//Quiet

MapThread[Labeled[Plot3D[PDF[#,{x,y}],{x,-1.3,1.3},{y,-1.3,1.3},PlotRange->All,PlotPoints->30,PlotStyle->#3],#2]&,{seasonDistributions,seasonsNames,seasonColors}]//Quiet

DensityPlot[PDF[#,{x,y}],{x,-1.3,1.3},{y,-1.3,1.3},PlotRange->All,PlotPoints->50,PlotLegends->Automatic]&/@seasonDistributions

We can now estimate the season probability for a given color

𝒫((Subscript[season, j])\[Conditioned](Subscript[color, i]))
=((𝒫((Subscript[color, i])\[Conditioned](Subscript[season, j]))*𝒫((Subscript[season, j])))/(𝒫((Subscript[color, i]))))=((𝒫((Subscript[color, i])\[Conditioned](Subscript[season, j]))*𝒫((Subscript[season, j])))/((∑_m)𝒫((Subscript[color, i])\[Conditioned](Subscript[season, m]))𝒫((Subscript[season, m]))))∼𝒫((Subscript[color, i])\[Conditioned](Subscript[season, j]))

And—assuming for the sake of simplicity that each color is independent—the season probability for a set of colors

𝒫((Subscript[season, j])\[Conditioned](Subscript[{(Subscript[color, i])}, i]))=(∏_i)𝒫((Subscript[season, j])\[Conditioned](Subscript[color, i]))

EnsembleColorDistance[colors_]:=
Times@@@Outer[PDF,seasonDistributions,colors,1]

ClearAll[SeasonClassifier]
SeasonClassifier[image_,n_:10]:=With[{colors=LABDominantColors[image,n][[All,{2,3}]]},
MaximalBy[Transpose[{seasonsNames,EnsembleColorDistance[colors]}],Last][[All,1]]
]

trainingSet=Take[seasons,All,trainingSetSize];
testSet=Drop[seasons,None,trainingSetSize];

res1=MapThread[({image,season}|->If[SeasonClassifier[image,nColors][[1]]===season,1,0] ), {trainingSet,Transpose[ConstantArray[seasonsNames,Length[trainingSet[[1]]]]]}, 2];

Mean[Flatten[res1]]//N

Grid[{seasonsNames,N[Mean/@res1,3]}]

res2=MapThread[({image,season}|->If[SeasonClassifier[image,nColors][[1]]===season,1,0] ), {testSet,Transpose[ConstantArray[seasonsNames,Length[testSet[[1]]]]]}, 2];

Mean[Flatten[res2]]//N

Grid[{seasonsNames,N[Mean/@res2,3]}]

Final results

Let's see how our last classifier performs!

set=RandomChoice[#,4]&/@seasons;
Grid[
MapThread[({image,season}|->Labeled[Image[image,ImageSize->150], Style[#,If[#===season,Black,Red]]&@SeasonClassifier[image][[1]]] ), {set,Transpose[ConstantArray[seasonsNames,Length[set[[1]]]]]}, 2]]

Thank you for your attention

Questions?

Initialization

Get["seasons.mx"]
Get["LABcolors.mx"]

seasonsPictures={
{,,,,,,,,,,,,,,,,,,},
{,,,,,,,,,,,,,,,,,,},
{,,,,,,,,,,,,,,,,,,},
{,,,,,,,,,,,,,,,,,,}};

n=100;
seasonsNames={"spring","summer","autumn","winter"};
trainingSetSize=Round[2/3 n];
seasonColors={RGBColor[0.5,0.76,0.15],RGBColor[1,0.75,0],Brown,RGBColor[0.67,0.87,1]};