Supervised Analysis

machine learning framework for Mathematica

The most common task in data analysis is to find any relationships of a set of input parameters to one or more output parameters. This kind of analysis is called supervised, as the goal of the analysis is given by the user. In the case that no explicit goal parameter is available, the analysis is called unsupervised. We will describe unsupervised analysis in another example.

Initialization of the system

Needs["Graphics`Graphics`"]
Needs["mlf`"]

Basics—iris data set

In this example we will analyze a simple data file that contains data from about 150 iris flowers of different types. Each flower is described by five attributes. The first four characterize the blossom of the flower: sepal length, sepal width, petal length, and petal width. The fifth attribute specifies the category of irises to which the flower belongs. The goal of our analysis is to find out how we can decide to which category a new, unclassified flower belongs.

For that purpose we will create a decision tree and a set of rules, which allow us to classify new flowers by simply asking a few questions about its blossom.

In our data set we have four numerical attributes and one textual attribute. So we have four attributes describing the blossom of each flower and three classes of flowers. The data is stored in a text file where the first row contains the column headers. The textual attribute is in the last column.

Setup data

\(\(dataRaw = 
        Import[$TopDirectory <> "/AddOns/Applications/mlf/data/iris.txt", "Table"];\)\)
headers = dataRaw[[1]];
data = 
        dataRaw[[1 + 
              Range[Length@dataRaw - 1]]];
irisData = 
        Def["IrisData", 
          DataSet[data, headers]];
Clear[dataRaw, data]

Now we can compute appropriate predicates with the CreatePredicates[] and DefPredicateVars[] commands for all input attributes and define the predicates for the goal attribute accordingly.

testPreds = 
        CreatePredicates[irisData, {1, 2, 3, 4}, 
          5];
testVars = DefPredicateVars[testPreds];

goalPreds = 
        CreatePredicatesIs[irisData, 5];
goalVars = DefPredicateVars[goalPreds];

Decision tree

Compute a decision tree

The decision tree rule induction method FS-ID3 is a generalization of Quinlan's ID3 method. A decision tree can be thought of as a series of if-then-else statements and can be used to solve arbitrary classification problems.

To generate a decision tree, we have to specify a desired goal predicate. In this example we will use the class attribute, which is the fifth column in our input file. Then we can compute the desired decision tree and finally plot the resulting tree.

Now we can generate the decision tree using the CreateID3[] command. The algorithm uses the data stored previously and tries to find rules to separate the three classes.

id3tree = CreateID3[irisData, testVars, goalVars];

With the command PlotID3[] it is very easy to plot the decision tree obtained.

Show[
        Graphics[{PlotID3[id3tree], PlotClassLegend[goalVars]}, 
          AspectRatio -> 0.4]];

In this plot we can see the decision tree computed. Each node consists of three bars. The first one indicates the class distribution of samples belonging to this node. The second bar indicates the number of samples belonging to this node with respect to the total number of samples. The third bar indicates the entropy gain to the next level. A high entropy gain indicates important decisions. We can see that the first predicate petal_length_IsAtLeast_L separates the iris versicol flowers in the second branch. We can read this rule like if petal length is not at least low (which is equivalent to the statement petal length is very low) then class is iris versicol. The first branch contains only samples of the other two classes. The remaining samples are separated by a new predicate petal_width_IsAtLeast_H. This separates the remaining samples into the desired goal classes.

Reviewing the results

To check if the computed decision tree is able to describe the data, we can now use it to forecast the classification of the flowers and compare the results with the original classification.

RecallID3[] applies a decision tree to a new data set. Afterwards, the classifications obtained can be compared with the original ones by using the CompEvalMatrix[] command. The evaluation matrix contains the cross-validation of input and recall in percent. With the PlotEvalChart3D[] command this matrix can also be shown in a graphical manner.

recalledID3 = RecallID3[irisData, id3tree];

evalMatrix = 
        CompEvalMatrix[GetData[irisData, All, 5], 
          recalledID3];
evalMatrix // MatrixForm
PlotEvalChart3D@evalMatrix;

We can see that 86.57 percent of the first class have been classified correctly. However, 13.43 percent have been predicted to be from the third class.

To extract those samples that have not been classified correctly, the command ExtractNonMatchingRecs[] is used.

failedS = 
        Extract[GetData[irisData], 
          ExtractNonMatchingRecs[irisData, 5, recalledID3]];
failedS // TableForm

headers

Show@
        Graphics[
          Join[
PlotAttributes[failedS, Dims -> {3, 4}, 
              PointSize -> 0.02], 
PlotAttributes[
              irisData, Dims -> {3, 4}, Goal -> 5, 
              PointSize -> 0.01]
], Axes -> True, 
          PlotRange -> All];

Rule base

Compute a set of rules

To create a set of rules that can be used to forecast the classification of a new flower, a modification of Quinlan's FOIL algorithm, called FS-FOIL, is implemented in machine learning framework (mlf).

To generate a rule base we have to specify a desired goal predicate. In this example we can use the class attribute, which is the fifth column in our input file.

Now we can generate the rule base using the CreateFOIL[] command. The algorithm uses the data stored previously and tries to find rules to forecast the three classes.

foilRules = 
        CreateFOIL[irisData, testVars, goalVars, 
          MinConf -> 0.9];

The command PrintFOILRules[] can be used to print out a set of rules computed by FS-FOIL.

PrintFOILRules[foilRules]

In the output each, line corresponds to a rule, and its elements are connected by means of conjunction. The rules belonging to the same goal predicate can be interpreted as being disjointed.

With the option Info->True it is possible to display additional information about the accuracy of the rules found.

PrintFOILRules[foilRules, Info -> True]

The additional output can be read as follows.
({{tt, tf, rt, conf, supp}, {33.7154, 1.`, 16.284595489501953`, 0.971194326877594`, 0.22476936876773834`}, {9.486100196838379`, 0.126979, 40.51389694213867`, 0.9867909550666809`, 0.0632406696677208`}})
tt is the number of samples fulfilling the rule and belonging to the desired goal class.
tf is the number of samples fulfilling the rule, but not belonging to the desired goal class. 
rt is the number of samples in the goal class that are not covered by the rule.
conf is the confidence of the rule, that is the percentage of correctly classified samples.
supp is the support of the rule, that is the percentage of samples fulfilling the rule relative to the complete data set.

Reviewing the results

To check if the computed rule base is able to describe the data, we can now use it to forecast the classification of the flowers and compare the results with the original classification.

RecallFOIL[] applies a rule base to a new data set. Afterwards, the classifications obtained can be compared with the original ones by using the CompEvalMatrix[] command. The evaluation matrix contains the cross validation of input and recall in percent. With the PlotEvalChart3D[] command this matrix can also be shown in a graphical manner.

recalledFOIL = RecallFOIL[irisData, foilRules];

evalMatrix = 
        CompEvalMatrix[GetData[irisData, All, 5], recalledFOIL];
evalMatrix // MatrixForm
PlotEvalChart3D@evalMatrix;

We can see that 86.41 percent of the first class have been classified correctly. However, 7.54 percent have been predicted to be from the third class. The remaining 6.04 percent of the samples of the first class have not been classified.

To extract those samples that have not been classified correctly, the command ExtractNonMatchingRecs[] is used.

failedS = 
        Extract[GetData@irisData, 
          ExtractNonMatchingRecs[irisData, 5, 
            recalledFOIL]];
failedS // TableForm

headers

Show@
        Graphics[
          Join[
PlotAttributes[failedS, Dims -> {3, 4}, 
              PointSize -> 0.02], 
PlotAttributes[
              irisData, Dims -> {3, 4}, Goal -> 5, 
              PointSize -> 0.01]
], Axes -> True, 
          PlotRange -> All];

Descriptions

Compute a set of descriptions

To create a set of rules that can be used to forecast the classification of a new flower, an algorithm called FS-MINER is implemented in mlf.

To generate a rule base we have to specify a desired goal predicate. In this example we can use the class attribute, which is the fifth column in our input file.

Now we can generate the rule base using the CreateMINER[] command. The algorithm uses the data stored previously and tries to find rules to forecast the three classes.

descriptions = 
        CreateMINER[irisData, testVars, goalVars, 
          MinConf -> 0.9];

The command PrintMINERRules[] can be used to print out a set of rules computed by FS-MINER.

PrintMINERRules[descriptions, Info -> True]

We can see that all rules have very high confidence and support. In contrast to the rules created by FS-FOIL,these rules are not disjoint.

Reviewing the results

To evaluate the quality of the obtained descriptions, we can use the RecallMINER[] method and CompEvalMatrix[] to compute an evaluation matrix.

recalledMINER = RecallMINER[irisData, descriptions];

evalMatrix = 
        CompEvalMatrix[GetData[irisData, All, 5], 
          recalledMINER];
evalMatrix // MatrixForm
PlotEvalChart3D@evalMatrix;

Similar to the rule based approach, we can also extract the incorrectly classified examples using ExtractNonMatchingRecs[] and plot the results afterwards.

failedS = 
        Extract[GetData@irisData, 
          ExtractNonMatchingRecs[irisData, 5, 
            recalledMINER]];
failedS // TableForm

Show@
        Graphics[
          Join[
PlotAttributes[failedS, Dims -> {3, 4}, 
              PointSize -> 0.02], 
PlotAttributes[
              irisData, Dims -> {3, 4}, Goal -> 5, 
              PointSize -> 0.01]
], Axes -> True, 
          PlotRange -> All];

Simple example—wine data set

This data set is taken from the UCI machine learning repository and contains the results of an analysis of 178 wines grown in the same region in Italy, but the wines are from three different vineyards. The analysis determined the quantities of constituents found in each of the three types of wines and some optical properties. All together, there are 13 input attributes, all of which are numerical. The goal predicate is Boolean categorical with 3 different classes/labels corresponding to the three vineyards. Accordingly, we use fuzzy predicates induced by appropriate fuzzy sets on the domains of the numerical input attributes. Although the goal predicate is Boolean categorical, the use of fuzzy predicates for the input predicates is not meaningless. The reason is that using fuzzy sets allows us to more easily model regions of overlapping goal classes and in a more natural way than by splitting the numerical attributes into Boolean classes.

Setup data

Load data and create predicates

\(\(dataRaw = 
        Import[$TopDirectory <> "/AddOns/Applications/mlf/data/wine.data", "Table"];\)\)
headers = dataRaw[[1]];
data = 
        dataRaw[[1 + 
              Range[Length@dataRaw - 1]]];
{trainData, testData} = 
        SplitDataRandomly[data, 0.8];
wineData = 
        Def["WineTrainData", 
          DataSet[trainData, headers]];
wineTestData = 
        Def["WineTestData", 
          DataSet[testData, headers]];
Clear[dataRaw, data];

headers

Now we can compute appropriate predicates for all data dimensions and define the classes of the goal dimension.

testParts = 
        CreatePartition[wineData, 1 + Range[Length[headers] - 1], 
          5];
testPreds = testParts[[All, 2]];
goalPreds = CreatePredicatesIs[wineData, 1];

This lists contain pairs of predicate names and definitions.

To assign variables with the names given to the corresponding definitions, we use the following macro.

testVars = DefPredicateVars[testPreds];
goalVars = DefPredicateVars[goalPreds];

We now have a list of variables referring to our (now internal) predicates.

goalVars // TableForm

To get the underlying predicates we need to recall them.

GetObject[testVars[[1, 1]]]

Plot predicates

Do[
curSets = 
          testParts[[i - 1, 1]]; 
PlotFuzzySetHistogram[
          curSets, GetData[wineData, All, i], HistogramScale -> 1, 
          HistogramCategories -> 20, 
          PlotLabel -> headers[[i]]]
, {i, 2, 
          Length@headers}];

Decision tree

Compute a decision tree

Now we create the decision tree, where we refer to the data set already stored.

Options@CreateID3

id3tree = 
        CreateID3[wineData, testVars, goalVars, MinConf -> 0.9, 
          MaxLevel -> 3];

Now lets show the graphical tree.

Show[
        Graphics[{PlotClassLegend[goalVars], PlotID3[id3tree]}, 
          AspectRatio -> 0.7]];

Reviewing the results

recalledID3 = RecallID3[wineTestData, id3tree];

evalMatrix = 
        CompEvalMatrix[GetData[wineTestData, All, 1], 
          recalledID3];
evalMatrix // MatrixForm
PlotEvalChart3D@evalMatrix;

failedS = 
        Extract[GetData@wineTestData, 
          ExtractNonMatchingRecs[wineTestData, 1, recalledID3]];
failedS // TableForm

N@Length@failedS/Length@GetData@wineData

Rule base

Compute a set of rules

foilRules = CreateFOIL[wineData, testVars, goalVars];

PrintFOILRules[foilRules, Info -> True]

Reviewing the results

recalledFOIL = RecallFOIL[wineData, foilRules];

evalMatrix = 
        CompEvalMatrix[GetData[wineData, All, 1], recalledFOIL];
evalMatrix // MatrixForm
PlotEvalChart3D@evalMatrix;

failedS = 
        Extract[GetData@wineData, 
          ExtractNonMatchingRecs[wineData, 1, 
            recalledFOIL]];
failedS // TableForm

headers

Descriptions

Compute a set of descriptions

Options@CreateMINER

descriptions = 
        CreateMINER[wineData, testVars, goalVars, MinSup -> 0.05, 
          MinConf -> 0.9, Logic -> LogicL];

PrintMINERRules[descriptions, Info -> False]

Reviewing the results

recalledMINER = 
        RecallMINER[wineData, descriptions, 
          Logic -> LogicL];

evalMatrix = 
        CompEvalMatrix[GetData[wineData, All, 1], recalledMINER];
evalMatrix // MatrixForm
PlotEvalChart3D@evalMatrix;

Application example—credit grants

In this example we will analyze data on 404 customers of a bank to discover to which customers we can grant a credit. The data set contains information about profession, income, and properties owned by the customers.

Setup data

User-defined settings

Select data set

\(\(dataSetName = $TopDirectory <> "/AddOns/Applications/mlf/Data/credits.txt";\)\)
dataRaw = Import[dataSetName, "Table"];

Choose ratio of test and training samples, use 0 for no split

testPerc = 0.66;

The data is now imported and split into headers and data.

headers = 
        dataRaw[[1, 
            1 + Range[
                Length@dataRaw[[1]] - 
                  1]]];
data = 
        dataRaw[[1 + Range[Length@dataRaw - 1], 
            1 + Range@Length@headers]];
dataIndex = Length@headers - 1;

Transpose@{Range@Length@headers, headers}

Load data

completeDS = DataSet[data, headers];
completeData = Def["AllData", completeDS];

Split predicates into input and goal predicates

dataIndexPoint = Length@headers;
inputParams = Range[dataIndex];
goalParams = 
        Range[Length@headers - dataIndex] + 
          dataIndex;
dataHeaders = headers[[inputParams]];
goalHeaders = headers[[goalParams]];

Compute predicates

allPartitions = 
        CreatePartition[completeData, 
          Range@Length@headers];
goalVars = 
        DefPredicateVars@
          allPartitions[[goalParams, 2]];
testVars = 
        DefPredicateVars@allPartitions[[inputParams, 2]];

goalVars

Split data into training and test data

dataTmp = 
        SplitDataRandomly[data, testPerc];
trainData = 
        Def["TrainData", 
          DataSet[dataTmp[[1]], headers]];
testData = 
        Def["TestData", 
          DataSet[dataTmp[[2]], headers]];
Clear[dataTmp];

Show partitions defined

Do[
curSets = 
          allPartitions[[i, 1]]; 
PlotFuzzySetHistogram[
          curSets, GetData[completeDS, All, i], HistogramScale -> 10000, 
          PlotLabel -> headers[[i]]]
, {i, 
          Length@allPartitions}];

Decision tree

Compute decision tree

id3tree = 
        CreateID3[trainData, testVars, goalVars, MinEnt -> 0.075, 
          MinConf -> 0.85];

Show[
        Graphics[
{PlotClassLegend[goalVars], 
            PlotID3[id3tree]}, 
AspectRatio -> 
            0.5]];

Evaluate results

Print[Length@GetData@testData, "/", 
        Length@GetData@
            trainData, " test cases are used."];

recalledID3 = 
        RecallID3[testData, id3tree, Type -> "PROB"];

domIndex = 
        ComputeDomIndex[testData, goalVars];
evalMatrix = CompEvalMatrix[domIndex, recalledID3];

evalMatrix // MatrixForm
PlotEvalChart3D [evalMatrix];

Rule base

Compute rule set

foilRules = CreateFOIL[trainData, testVars, goalVars];

PrintFOILRules[foilRules, Info -> False]

Evaluate results

Print[Length@GetData@testData, "/", 
        Length@GetData@
            trainData, " test cases are used."];

recalledFOIL = RecallFOIL[testData, foilRules];

domIndex = 
        ComputeDomIndex[testData, goalVars];
evalMatrix = CompEvalMatrix[domIndex, recalledFOIL];

evalMatrix // MatrixForm
PlotEvalChart3D@evalMatrix;

Descriptions

Compute rule set

minerRules = 
        CreateMINER[trainData, testVars, goalVars, 
          ExpLevel -> 3];

PrintMINERRules[minerRules, Info -> False]

Evaluate results

Print[Length@GetData@testData, "/", 
        Length@GetData@
            trainData, " test cases are used."];

recalledMINER = RecallFOIL[testData, minerRules];

domIndex = 
        ComputeDomIndex[testData, goalVars];
evalMatrix = CompEvalMatrix[domIndex, recalledMINER];

evalMatrix // MatrixForm
PlotEvalChart3D@evalMatrix;