Support Vector Classifier  via Mathematica

B. Paláncz
 
 Department of Photogrammetry and Geoinformatics
 Budapest University of Technology  and Economics, 
 H-1111 Budapest, Hungary, Müegyetem rkp. 3.
 
 e-mail:  palancz@epito.bme.hu

Abstract

In this case study, a  Support Vector Classifier function has been developed in Mathematica. Starting with a brief summary of support vector classification method, the step by step implementation of the classification algorithm in  Mathematica  is presented and explained. To check our function, two test problems, learning a chess board and classification of two intertwined spirals are solved. In addition an application to image recognition is demonstrated using wavelet transformation as feature extraction method.

1. Introductio(n^1)

Kernel Methods are relatively new family of algorithms that presents a series of useful features for pattern analysis in datasets. KMs combine the simplicity and computational efficiency of linear algorithms, such as the perception algorithm or ridge regression, with flexibility of nonlinear systems, such as for example neural networks, and rigor of statistical approaches such as regularization methods in multivariate statistics. As a result of the special way they represent functions, these algorithms typically reduce the learning step to convex optimization problem, that can always be solved in polynomial time, avoiding the problem of local minima typical of neural networks, decision trees and other nonlinear approaches.

2. Support Vector Classificatio(n^(2, 3))

2.1 Binary classification

In case of binary classification, we try to estimate a real-valued function  f : X⊆ (R^n)->R using training data , that is n - dimensional patters (Subscript[x, i]) and class labels (Subscript[y, i])

(((Subscript[x, 1]),(Subscript[y, 1])),...,((Subscript[x, m]),(Subscript[y, m])))∈ (R^n) × {-1,  1}

such that f will correctly classify new examples (x, y)-that is, f(x) = y for examples (x, y), which were generated from the same underlying probability distribution P(x,y) as the training data. If we put no restriction on the class of functions that we choose our estimate f from, however, even a function that does well on the training data -for example by satisfying f((Subscript[x, i])) = (Subscript[y, i]) for i = 1...m- need not generalize well to unseen examples. Suppose we know nothing additional about f (for example about its smoothness), then the values on the training patterns carry no information whatsoever about values on novel patterns. Hence learning is impossible, and minimizing the training error does not imply a small expected test error.
	Statistical learning theory, or VC (Vapnik-Chervonenkis) theory, shows that is crucial to restrict the class of functions that the learning machine can implement to one with capacity that is suitable for the amount of available training data.

2.2 Optimal hyperplane classifier

To design learning algorithms, we thus must come up with a class of  functions whose capacity can be computed. SV classifiers are based on the class of hyperplanes

⟨w, x⟩ + b = 0   w∈(R^n), b∈R

corresponding to decision functions

f(x) = sign(⟨w,x⟩ + b )

We can show that the optimal hyperplane, defined as the one with the maximal margin of separation between the two classes (see Figure 1), has the lowest capacity, which ensuring that the classifier learned from training samples will misclassify the less elements of the test samples originated from the same probability distribution.

Figure.1  A separable classification problem. The optimal hyperplane is orthogonal to the shortest line connecting the convex hulls of the two classes, and intersects it half way. There is a weight vector w and a threshold b such that (Subscript[y, i])(⟨ w,(Subscript[x, i])⟩ + b > 0. Rescaling w and b such that the point(s) closest to the hyperplane satisfy | ⟨ w,(Subscript[x, i])⟩ + b | =1, we obtain a form (w, b) of the hyperplane with  (Subscript[y, i])(⟨ w,(Subscript[x, i])⟩ + b ≥ 1. Note that the margin, measured perpendicularly to the hyperplane, equals \(TraditionalForm\`1\/\(\(∥\)\(\ \)\(w\)\(∥\)\)\). To maximize the margin, we thus have to minimize ∥ w∥ subject to (Subscript[y, i])(⟨ w,(Subscript[x, i])⟩ + b ≥ 1.

2.3 Maximal margin  classifier

The optimization problem to find the optimal w vector and the threshold b is the following,

given a set of  linearly separable training samples

S = (((Subscript[x, 1]),(Subscript[y, 1])),...((Subscript[x, m]),(Subscript[y, m])))

the hyperplane (\(TraditionalForm\`\(w\^*\)\),\(TraditionalForm\`\(b\^*\)\)) that maximizes the geometric margin

(Subscript[minimize, w, b]) ⟨w, w⟩

subject to (Subscript[y, i ])⟨w,(Subscript[x, i])⟩ +b ≥1, i = 1.  .. m

Then the geometric margin can be computed considering that

⟨\(w\^*\),(Subscript[x, 1])⟩ +\(b\^*\) = 1

⟨\(w\^*\),(Subscript[x, 2])⟩ +\(b\^*\) = - 1

then

⟨\(TraditionalForm\`\(w\^*\)\), ((Subscript[x, 1])-(Subscript[x, 2]))⟩ = 2

rescaling

⟨\(TraditionalForm\`\(w\^*\)\/\(\(∥\)\(\ \)\(w\^*\)\(∥\)\)\), ((Subscript[x, 1])-(Subscript[x, 2]))⟩ = \(TraditionalForm\`2\/\(\(∥\)\(\ \)\(w\^*\)\(∥\)\)\)

therefore the margin is

γ = (1/(∥ \(w\^*\)∥))

The traing patterns lie closest to the hyperplane  (see Figure 1. two red balls and one blue diamond) are called support vectors, carrying all relevant information about the classification problem. The number of support vectors, sv are equal or less than the number of the training patterns, m.
This minimization problem can be transformed into a dual maximization problem leading to a quadratic programming task, whose solution w has an expansion

w=(∑_(i = 1)^sv)(Subscript[ν, i])(Subscript[x, i])

Consequently, the final decision function is

f(x) = sign((∑_(i = 1)^sv)(Subscript[ν, i])⟨x,(Subscript[x, i])⟩+b)

which depends only on dot products between pattern. This lets us generalize to the nonlinear case.

2.4 Feature spaces and kernels

The Figure 2 shows the basic idea of SV machines, which is to map the data into some other dot space, called the feature space F via nonlinear map,

Φ :(R^n) -> F

and perform the above linear algorithm in F. This is only requires the evaluation of dot products,

K (u,v) = ⟨Φ(u), Φ(v)⟩

Figure 2. The idea of SV machines: map the training data nonlinearly into a higher dimensional feature space via Φ, and construct a separating hyperplane with maximum margin there. This yields a nonlinear decision boundary in input space. By the use of kernel function, it is possible to compute the separating hyperplane without explicitly carrying out the map into the feature space.

Clearly, if F is high dimensional, the dot product on the righthand side will be very expensive to compute. In some cases, however there is a simple kernel that can be evaluated efficiently. For instance, the polynomial kernel

K(u, v)=((⟨ u, v⟩)^d)

can be shown to correspond to a map Φ  into the space spanned by all products of exactly d dimensions of  (R^n). For d = 2 and u, v ∈(R^2), for example, we have

((⟨u, v⟩)^2)=(((⟨(({{(Subscript[u, 1])}, {(Subscript[u, 2])}})),(({{(Subscript[v, 1])}, {(Subscript[v, 2])}}))⟩))^2)=⟨(({{((Subscript[u, 1])^2)}, {(Sqrt[2])(Subscript[u, 1])(Subscript[u, 2])}, {((Subscript[u, 2])^2)}})),(({{((Subscript[v, 1])^2)}, {(Sqrt[2])(Subscript[v, 1])(Subscript[v, 2])}, {((Subscript[v, 2])^2)}}))⟩=⟨Φ(u),Φ(v)⟩

defining Φ(x) =(((Subscript[x, 1])^2),(Sqrt[2])(Subscript[x, 1]) (Subscript[x, 2]),((Subscript[x, 2])^2)).

\(More\ generally, \ 
    we\ can\ prove\ that\ for\ every\ kernel\ that\ gives\ rise\ to\ a\ positive\ matrix\ \((kernel\ matrix)\)\)

(Subscript[M, i,j ])= K((Subscript[x, i]),(Subscript[x, j])) ,  we can construct a map such that  K(u, v)=⟨Φ(u),Φ(v)⟩  holds.

2.5 Optimization as a dual quadratic programming problem

Now the dual maximization  problem is the following,

consider classifying a set of training samples,

S = (((Subscript[x, 1]),(Subscript[y, 1])),...((Subscript[x, m]),(Subscript[y, m])))

using the feature space implicitly defined by the kernel K(x, z) and suppose the parameters \(TraditionalForm\`\(α\^*\)\) solve the following quadratic optimization problem,

maximize     W(α) = (∑_(i  = 1)^m)(Subscript[α, i])-(1/2)(∑_(i, j = 1)^m)(Subscript[y, i])(Subscript[y, j])(Subscript[α, i])(Subscript[α, j])(K((Subscript[x, i]),(Subscript[x, j]))+(1/c)(Subscript[δ, i, j]))

subject to     (∑_(i  = 1)^m)(Subscript[y, i])(Subscript[α, i])=0, (Subscript[α, i])  ≥ 0, i =1  .. m

Let f(x) =(∑_(i  = 1)^m)(Subscript[y, i]) \(α\_i\^*\) K((Subscript[x, i]),x)+(SuperStar[b]), where (SuperStar[b]) is chosen so that (Subscript[y, i]) f((Subscript[x, i])) = 1 -((\(α\_i\^*\))/c) for any i with 
\(\(α\_i\^*\) ≠ \ 0. \)

Then the decision rule given by sign(f(x)) is equivalent to the hyperplane in the feature space implicitly defined by the kernel K(x, z), which solves the optimization problem, where the geometric margin is

γ=((((∑_(i∈ sv))\(α\_i\^*\)-(1/c)\(⟨\(α\^*\), \(α\^*\)⟩\)))^(-(1/2)))

where set sv corresponds to indexes i, for which \(\(\(\(α\_i\^*\) ≠ \ 0\)\(,\)\(\ \)\)\)

sv ={i: \(\(α\_i\^*\) ≠ \ 0\); i=1.  .. m}

Training samples, (Subscript[x, i]), for which i∈sv are called support vectors giving contribution
 to the definition  of  f(x).

3. Implementation of SVR in Mathematica

3.1 Steps of implementation

The dual optimization problem can be solved conveniently using Mathematica. In this section, the steps of  the implementation of  SVC algorithm are shown by solving XOR problem. The truth table of XOR, using bipolar values for the output, is

Table 1. Truth table of XOR problem

({{x1, x2, y}, {0, 0, -1}, {0,  1,    1}, {  1, 0,    1}, {   1,   1, -1}})

The input and output data lists are

xym = {{0, 0}, {0, 1}, {1, 0}, {1, 1}};

zm = {-1, 1, 1, -1};

Let us employ Gaussian kernel with β gain

β = 10. ;

K[u_, v_] := Exp[-β (u - v) . (u - v)]

The number of the data pairs in the training set, m  is

m = Length[zm]

Create the objective function W (α) to be maximized, with regularization parameter,  c = 5

c = 5. ;

First, we prepare an matrix M, which is an extended form of the kernel matrix,

M = (Table[
              N[K[xym[[i]], xym[[j]]]], {i, 1, m}, {j, 1, 
                m}] + (1/c) IdentityMatrix[m]);

then the objective function can be expressed as,

W = (∑_(i = 1)^m) (Subscript[α, i]) - (1/2) (∑_(i = 1)^m)(∑_(j = 1)^m)(zm[[i]] zm[[j]] (Subscript[α, i]) (Subscript[α, j]) M[[i, j]]);

The constrains for the unknown variables are

g = 
        Apply[And, 
          Join[Table[(Subscript[α, i]) ≥ 0, {i, 1, 
                m}], {(∑_(i = 1)^m) zm[[i]] (Subscript[α, i]) == 
                0}]];

However the maximization problem  is a convex quadratic problem, from practical reasons to maximize the objective function the built in function NMaximize is applied. NMaximize implements several algorithms for finding constrained global optima. The methods are flexible enough to cope with functions that are not differentiable or continuous, and are not easily trapped by local optima. Possible settings for the Method option include "NelderMead", "DifferentialEvolution", "SimulatedAnnealing" and "RandomSearch". 
Here we use  DifferentialEvolution, which is a genetic algorithm that maintains a population of specimens, (Subscript[x, 1]), …, (Subscript[x, n]), represented as vectors of real numbers (“genes”). Every iteration, each (Subscript[x, i]) chooses random integers a, b, and c and constructs the mate (Subscript[y, i]) = (Subscript[x, i]) + γ((Subscript[x, a]) + ((Subscript[x, b]) - (Subscript[x, c]))), where γ is the value of ScalingFactor. Then (Subscript[x, i]) is mated with (Subscript[y, i]) according to the value of CrossProbability, giving us the child (Subscript[z, i]). At this point (Subscript[x, i]) competes against (Subscript[z, i]) for the position of (Subscript[x, i]) in the population. The default value of SearchPoints is Automatic, which is Min[10*d, 50], where d is the number of variables.

We need the list of unknown variables α,

vars = Table[(Subscript[α, i]), {i, 1, m}];

Then the solution of the maximization problem is,

sol = 
      NMaximize[{W, g}, vars, 
        Method -> DifferentialEvolution]

The consistency of this solution can be checked by computing values of  b  for every data points. Theoretically, these values should be same for any data points, however, in general, this is only approximately true.

bdata = 
      Table[(((1 - ((Subscript[α, j])/c))/zm)[[j]] - (∑_(i = 1)^m) 
                    zm[[i]] (Subscript[α, i]) K[xym[[i]], 
                    xym[[j]]]) /. sol[[2]], {j, 1, m}]

The value of b can be chosen as the average of these values

b = Apply[Plus, bdata]/m

Then the classifier function is,

f[w_] := (((∑_(i = 1)^m) zm[[i]] (Subscript[α, i]) K[w, 
                    xym[[i]]]) + b) /. sol[[2]]

In symbolic form

Clear[x, y]

f[{x, y}]

Let us display the contour lines of the continuous classification function,

<< ExtendGraphics`LabelContour`

p=ContourPlot[f[{x,y}], {x, 0, 1},{y,0,1},Contours->{-0.5,-0.05,0,0.05,0.5},ContourShading->False,PlotPoints->50,DisplayFunction->Identity];

p1 = 
        LabelContourLines[p, LabelPlacement -> Automatic, 
          LabelFont -> {"Courier", 10}, 
          DisplayFunction -> Identity];

<<Graphics`MultipleListPlot`

p2 = 
        MultipleListPlot[{{0, 0}, {1, 1}}, {{0, 1}, {1, 0}}, 
          SymbolStyle -> {Hue[ .0], Hue[0.8]}, 
          SymbolShape -> {PlotSymbol[Box, 3], PlotSymbol[Star, 7]}, 
          Frame -> True, Axes -> False, AspectRatio -> 1, 
          DisplayFunction -> Identity];

Show[{p1, p2}, 
        DisplayFunction -> $DisplayFunction];

Figure 3. The contour lines of  the continuous classification function f(x1, x2) for XOR problem

The discrete classifier, the decision rule using signum function is

Plot3D[Sign[f[{x, y}]], {x, 0, 1}, {y, 0, 1}, 
        PlotRange -> All];

Figure 4. The decision rule,  sign(f(x1, x2)) for XOR problem

Let us check the classifier

Map[Sign[f[#]] &, xym]

and

zm == %

3.2 Mathematica module for SVR

These steps can be collected in a module, where the vector xm contains the input vectors (the training set), and the  vector ym contains the corresponding scalar output values ( the labels of the training set),

SupportVectorClassifier[xm_, ym_, K_, c_] := 
        Module[
{m, n, M, i, j, W, g, vars, sol, bdata, 
            b}, 
m = Length[ym]; 
          n = Length[xm[[1]]]; 
M = 
            Table[K[xm[[i]], xm[[j]]], {i, 1, m}, {j, 1, 
                  m}] + (1/c) 
                IdentityMatrix[
                  m]; 
W = (∑_(i = 1)^m) (Subscript[α, i]) - (1/2) (∑_(i = 1)^m)(∑_(j = 1)^m)(ym[[i]] ym[[j]] (Subscript[α, i]) (Subscript[α, j]) M[[i, 
                            j]]); 
g = 
            Apply[And, 
              Join[Table[(Subscript[α, i]) ≥ 0, {i, 1, 
                    m}], {(∑_(i = 1)^m) 
                          ym[[i]] (Subscript[α, i]) == 
                    0}]]; 
vars = 
            Table[(Subscript[α, i]), {i, 1, 
                m}]; 
sol = NMaximize[{W, g}, vars, 
                Method -> 
                  DifferentialEvolution][[2]]; 
bdata = Table[(((1 - ((Subscript[α, j])/c))/ym)[[j]] - (∑_(i = 1)^m) 
                          ym[[i]] (Subscript[α, i]) K[xm[[i]], 
                          xm[[j]]]) /. sol[[2]], {j, 1, 
                m}]; 
b = 
            Apply[Plus, bdata]/
              m; 
{(((∑_(i = 1)^m) 
                          ym[[i]] (Subscript[α, i]) K[Table[(Subscript[x, j]), {j, 1, n}], 
                          xm[[i]]]) + b) /. sol, 
            vars /. sol}];

The results of this module are the analytic form of the continuous classifier function and the values of  (Subscript[α, i])'s. Let us check the solution of the XOR problem

SupportVectorClassifier[xym, zm, K, c]

%[[1]] == f[{(Subscript[x, 1]), (Subscript[x, 2])}] // Chop

4. Test problems

4.1 Learning a chess board

Let us consider a 2 × 2 chess board. The training points are generated by uniformly distributed random numbers from the interval [-1,1] × [-1, 1]. 
The chess board matrix

M = {{1, -1}, {-1, 1}};

p1 = 
        ListDensityPlot[M, MeshRange -> {{-1, 1}, {-1, 1}}, 
          DisplayFunction -> Identity];

Creating the training set using 50 random samples,

xym = {}; zm = {};

Do[x1 = Random[Real, {-0.99, 0.99}]; 
        x2 = Random[Real, {-0.99, 0.99}]; 
If[
          x1 x2 > 0, z = 1, z = -1]; AppendTo[xym, {x1, x2}]; 
        AppendTo[zm, z], {k, 1, 50}];

Preparation data to display them

data = Transpose[Join[Transpose[xym], {zm}]];

data1 = Map[Drop[#, -1] &, Select[data, #[[3]] > 0 &]]; 
    data2 = Map[Drop[#, -1] &, 
        Select[data, #[[3]] < 0 &]];

p2=MultipleListPlot[data1,data2,SymbolShape -> {PlotSymbol[Triangle,5], PlotSymbol[Box,2]},SymbolStyle->{Hue[.7],Hue[.0]},Frame->True,AspectRatio->1,DisplayFunction->Identity];

Let us employ the same Gaussian kernel , but now with gain  β = 15.

β = 15;

employing parameter c = 100,

c = 100;

The solution is

F = SupportVectorClassifier[xym, zm, K, c];

This run could take some minutes.

p3= ContourPlot[F[[1]],{(Subscript[x, 1]),-1,1},{(Subscript[x, 2]),-1,1},Contours->{0},PlotPoints->100,ContourShading->False,DisplayFunction->Identity];

p4=DensityPlot[Sign[F[[1]]],{(Subscript[x, 1]),-1,1},{(Subscript[x, 2]),-1,1},PlotPoints -> 100,Mesh -> False,DisplayFunction -> Identity];

Show[GraphicsArray[{Show[{p2, p3}], p1, p4}], 
        DisplayFunction -> $DisplayFunction];

Figure 5. SVC result from 60 random samples: the functions  f(x) with the random samples, the ideal chess board and sign(f(x))

The SV solution of a larger chess board problem can be found in [3].

4.2 Two intertwined spirals

The two intertwined spirals is a challenging classification benchmark originated from the field of neural networks [4].

Parametric equations of the spirals are

x1[t_] := 2 Cos[ t] (E^(t/10))

y1[t_] := 1.5 Sin[ t] (E^(t/10))

x2[t_] := 2.7  Cos[ t] (E^(t/10))

y2[t_] := 2.025  Sin[ t] (E^(t/10))

Generating 26 discrete points for each spiral,

s1 = 
        Table[{ x1[t], y1[t]}, {t, π, 
            3.5 π, ((2.5 π)/25)}];

s2 = 
        Table[{ x2[t], y2[t]}, {t, -(π/2), 
            2.5 π, ((3.  π)/25)}];

then displaying these points,

S1 = 
        ListPlot[s1, PlotStyle -> {RGBColor[1, 0, 0], PointSize[0.02]}, 
          AspectRatio -> 1, 
          DisplayFunction -> Identity];

S2 = 
        ListPlot[s2, PlotStyle -> {RGBColor[0, 0, 1], PointSize[0.015]}, 
          AspectRatio -> 1, 
          DisplayFunction -> Identity];

pspiral = 
        Show[{S1, S2}, 
          DisplayFunction -> $DisplayFunction];

Figure 6. Two intertwined spirals represented by 41 points each

Creating the teaching set, putting these points into one list

xym = Join[s1, s2];

Generating the labels of the samples,

zm = Join[Table[1. , {26}], Table[-1. , {26}]];

Length[zm]

Dimensions[xym]

Applying  wavelet kernel [5] with parameter a = 1.8,  in case of  dimension  n = 2

n = 2; a = 1.8;

K[u_, 
        v_] := (∏_(i = 1)^n)(Cos[
              1.75 ((u[[i]] - v[[i]])/a)] Exp[-(( (((u[[i]] - v[[i]]))^2))/(2 (a^2)))])

and with parameter c = 100

c = 100. ;

The solution is

F = SupportVectorClassifier[xym, zm, K, c];

This run could take some minutes.

psvc1=ContourPlot[F[[1]],{(Subscript[x, 1]),-7,7},{(Subscript[x, 2]),-7,7},Contours->{0},PlotPoints->50,AspectRatio->1,ContourShading->False,DisplayFunction->Identity];

Show[{pspiral, psvc1}, 
        DisplayFunction -> $DisplayFunction];

Figure 7. Classification results of SVC with the nonlinear decision boundary

The continuous classification function and the decision rule can be displayed in 3D, too

P1=Plot3D[F[[1]],{(Subscript[x, 1]),-7,7},{(Subscript[x, 2]),-7,7},PlotRange -> All,PlotPoints -> {30, 30},DisplayFunction -> Identity];

P2=Plot3D[Sign[F[[1]]],{(Subscript[x, 1]),-7,7},{(Subscript[x, 2]),-7,7},PlotRange -> All,PlotPoints -> {40, 40},DisplayFunction -> Identity];

Show[GraphicsArray[{P1, P2}], 
        DisplayFunction -> $DisplayFunction];

Figure 8. Classification function and the corresponding decision rule

5. Image classification

We should like to classify  images into two different categories, snowman and dice, see Figures 9 and 10.

Figure 9. Representatives of snowman category

Figure 10. Representatives of dice category

The feature extraction of these pictures  represented by  matrices of 128 × 128 pixels,  was carried out by wavelet transform using second order Daubechies filter [6], then employing averaging technique [ 7] for seven nonoverlapping bands of the spectrum. Consequently the dimension of these feature vectors is seven,  n = 7.
The feature vectors of  snowmen, from left to right,

fvh1 = {17.987862406345197`, 15.339852345251783`, 
          13.405059986913816`, 11.39844147326539`, 9.147096417803048`, 
          7.033082666195232`, 4.798303619676565`};

fvh2 = {17.766474068532943`, 15.353169727671743`, 
          13.583147305884832`, 11.442932257229275`, 9.491679304635152`, 
          7.326023584142842`, 5.161496090621752`};

fvh3 = {18.103411342767437`, 15.371635513578214`, 
          13.202523449937722`, 11.062797391406976`, 9.368963494909677`, 
          7.057866279976232`, 5.077124049177121`};

fvh4 = {18.1709394935011`, 15.315294077374773`, 13.6041114027883`, 
          11.4958063381276`, 9.579804299143635`, 7.345586733409343`, 
          5.141453727483033`};

fvh5 = {18.348408722947845`, 15.600433067236114`, 13.45926587596142`, 
          11.298982304250389`, 9.234589734745336`, 6.935839853987073`, 
          4.6874515141320625`};

fvh6 = {18.55417730221509`, 15.405886197703628`, 12.994549889958655`, 
          11.108367738308896`, 9.292707430500966`, 6.919296621727446`, 
          4.511708592363226`};

fvh7 = {18.314635037628157`, 15.438902136555589`, 
          13.772985606947227`, 11.077006077418556`, 9.39215624869045`, 
          7.088943106162201`, 4.740247276938691`};

fvh8 = {18.422579068846456`, 15.44269453449702`, 13.513840962700638`, 
          11.428495120182927`, 9.148425413113836`, 7.046848261278439`, 
          4.694119093310168`};

fvh9 = {18.43513010369331`, 15.454169597007873`, 13.708644791672018`, 
          10.940891580889742`, 9.209006556341908`, 6.843915401182678`, 
          4.034620107512246`};

fvh10 = {18.436640348880456`, 15.34733995320342`, 
          13.609786709283107`, 11.450781255719415`, 8.952378078329414`, 
          6.624937026635307`, 3.713739193450354`};

The feature vectors of dices, from left to right,

fvk1 = {18.513206366376362`, 14.449874299970174`, 
          13.544739232822373`, 12.40092757444905`, 10.189337384845338`, 
          7.3292375831297685`, 4.736174157402171`};

fvk2 = {18.50419064824187`, 14.501955683529022`, 13.897585347039538`, 
          12.315411421576888`, 10.380098171932897`, 7.680862954130695`, 
          5.113633812677783`};

fvk3 = {18.35998649741921`, 14.325007487010204`, 13.891109482798795`, 
          12.041493245625402`, 10.245624068294767`, 7.635283116835789`, 
          4.892220409840139`};

fvk4 = {18.74725921957245`, 14.481206026532117`, 13.586894883522623`, 
          12.159670019275667`, 9.7073431909908`, 7.113947248457536`, 
          4.320217058606431`};

fvk5 = {18.578001547534335`, 13.56106484453565`, 13.939818234922745`, 
          12.275461106255154`, 10.339064713550757`, 7.645968182660811`, 
          5.0034125687198125`};

fvk6 = {18.734938375431323`, 13.426310886783215`, 
          13.848149559149393`, 12.51036473995895`, 10.038140671950991`, 
          7.478539842000284`, 4.364368004657445`};

fvk7 = {18.704207897344432`, 14.1043321595848`, 13.816500082559502`, 
          12.186540780365704`, 10.09035041547276`, 7.3090549357115`, 
          4.517843532536042`};

fvk8 = {18.62102884531058`, 13.85093633669147`, 13.564163871814355`, 
          12.054010558850377`, 9.881985429568406`, 7.000473177572209`, 
          4.252312638574372`};

fvk9 = {18.665170732704688`, 14.811226132783473`, 
          13.902084134865797`, 11.924150150415487`, 9.892352518265293`, 
          6.741462829622055`, 4.235952266871399`};

fvk10 = {18.83160012850086`, 14.559912101933348`, 
          13.518379854185252`, 12.281168961677198`, 9.892557265288342`, 
          7.087886591871221`, 4.344411525543565`};

The first 7 -7 pictures are considered as teaching set and the last 3 -3 pictures are considered as test samples.

xym = {fvh1, fvh2, fvh3, fvh4, fvh5, fvh6, fvh7, fvk1, fvk2, fvk3, 
          fvk4, fvk5, fvk6, fvk7};

Let us labeling the snowmen category with  +1  and the dice category with   - 1.

zm = {1, 1, 1, 1, 1, 1, 
          1, -1, -1, -1, -1, -1, -1, -1};

Employing wavelet kernel with parameters

n = 7; a = 2. ;

K[u_, 
        v_] := (∏_(i = 1)^n)(Cos[
              1.75 ((u[[i]] - v[[i]])/a)] Exp[-(( (((u[[i]] - v[[i]]))^2))/(2 (a^2)))])

and

c = 150;

The classification function is

F = SupportVectorClassifier[xym, zm, K, c];

Short[F[[1]], 20]

Defining the decision rule we get,

PictureClassifier[u_] := 
      Sign[F[[1]]] /. {(Subscript[x, 1]) -> 
            u[[1]], 
          (Subscript[x, 2]) -> u[[2]], 
          (Subscript[x, 3]) -> u[[3]], 
          (Subscript[x, 4]) -> u[[4]], 
          (Subscript[x, 5]) -> u[[5]], 
          (Subscript[x, 6]) -> u[[6]], 
          (Subscript[x, 7]) -> 
            u[[7]]}

Let us classify the test samples

Map[
      PictureClassifier[#] &, {fvh8, fvh9, fvh10, fvk8, fvk9, 
        fvk10}]

The classification is correct for every test picture.

References

1. Berthold M. and Hand D.J. (Eds.) "Intelligent Data Analysis, An Introduction", Springer, 2003.
 
2. Hearst M. A. Support Vector Machines, IEEE Intelligent Systems, pp. 18 -28, July/August, 1998.

3. Cristianini N. and Shawe-Taylor J. "An introduction to Support Vector Machines and other kernel - based learning methods", Cambridge, University Press, 2003.

4.Juillé H.,Pollack J.B. Co-evolving intertwined spirals, in Proc. of the 5th. Ann.Conf. on Evolutionary Programming, San Diego, USA, pp.461-468, 1996.

5. Zhang L, Zhou W. and Jiao L. Wavelet Support Vector Machine, IEEE Trans. Systems, Man and Cybernetics - Part B: Cybernetics, Vol. 4. No.1. pp. 34 -39, Febr. 2004  

6. Wavelet Explorer with Mathematica, Mathematica Application Package, Wolfram Research Inc. 2003.

7. Goswani J.C. and Chan A.K. "Fundamentals of wavelets", J.Wiley, 1999.