Neural Networks Examples

Classification of Paper Quality

Introduction

In this example different neural classifiers are compared using data from a hybrid gas array sensor, an electronic nose. The odors from five different cardboard papers from commercial manufacturers are recorded with the electronic nose. Different kinds of classifiers are trained to determine the origin of an unknown sample.

The data was kindly contributed by the Swedish Sensor Centre, S-SENCE, www.ifm.liu.se/Applphys/S-SENCE. More information on the data set can be found in “Identification of Paper Quality Using a Hybrid Electronic Nose” by Martin Holmberg, Fredrik Winquist, Ingemar Lundström, Julian W. Gardner, and Evor L. Hines, Sensors and Actuators B 26-27 (1995) 246-249.

Load the Neural Networks package and the data.

<< NeuralNetworks`

<< paper.dat;

There is one data set for estimation, xs and ys, and one for validation of the classifiers, xv and yv.

Check the dimensions of the data.

Dimensions[xs]
Dimensions[ys]

There are 48 data samples available for estimation. Each paper sample is characterized by 15 x values from the 15 sensors in the electronic nose sensor array.

Five different types of cardboard paper and plain air are measured, making six possible output classes. The correct class of each data sample is stored in y, with a 1 in the appropriate column indicating the class of the sample.

Check the class of the 27th sample of validation data.

yv[[27]]

The 27th sample belongs to class 3.

It is always a good idea to check how many data samples there are from each class.

Look at the distribution of estimation data over the classes.

NetClassificationPlot[xs, ys]

Look at the distribution of validation data over the classes.

NetClassificationPlot[xv, yv]

Vector Quantization Network

First, try a vector quantization (VQ) network on the paper data. You need at least one codebook vector for each class, a minimum of six codebook vectors. The result will vary each time the commands are evaluated due to the random nature of the initialization and training processes.

Load the Mathematica application package Neural Networks and the data.

<< NeuralNetworks`
<< LinearAlgebra`MatrixManipulation`

<< paper.dat;

Initialize a VQ network with six codebook vectors, and train it for 10 iterations with the Competitive algorithm.

{vq, fitrecord} = 
        VQFit[xs, ys, 6, 10, Method -> Competitive];

Obtain some information about the trained network.

VQInformation[vq]

The trained VQ network can now be used to classify paper samples by simply applying it to new data vectors.

Use the trained VQ network to classify data sample 15 of the validation data.

vq[xs[[15]]]

The classification result can also be illustrated using NetPlot. Using NetPlot on a data set with more that two dimensions produces a bar chart with the correctly classified samples on the diagonal. It is interesting to compare the classification results of the estimation and validation data.

Present the classification evaluated using estimation data.

NetPlot[vq, xs, ys]

Present the classification evaluated using validation data.

NetPlot[vq, xv, yv]

Although no perfect classifications were obtained on the validation data, it is clear from the plots that most samples were correctly classified by the VQ network. The off-diagonal bars correspond to incorrectly classified data, and the x and y axes show from which classes they come. Another way to illustrate this is to use the option Table.

Illustrate the classification on validation data with a table.

NetPlot[vq, xv, yv, DataFormat -> Table]

Each box illustrates the data assigned to a class. For example, the second box from the left shows that six data samples from class 2 were assigned to the second class. Note that this may turn out differently if you repeat the example.

You can also look at how the classification improves for each class as a function of the number of training iterations. In this way you can see if there is a problem with any specific class.

Plot the progress of the classifier on the validation data.

NetPlot[fitrecord, xv, yv]

The dashed lines indicate incorrectly classified data.

Radial Basis Function Network

A radial basis function (RBF) network will be used on the cardboard paper data. RBF networks are often not suitable for use in high-dimensional problems like this one, which has 15 input dimensions. It is, however, possible to choose to use only a subset of the inputs. Choosing a subset of inputs can be seen as a simple form of feature extraction. To simplify the problem, only three inputs will be used and only two of the paper types will be classified. The result will vary each time the commands are evaluated due to the random nature of the initialization and training processes.

Load the Mathematica application package Neural Networks and the data.

<< NeuralNetworks`
<< LinearAlgebra`MatrixManipulation`

<< paper.dat;

Select input data from sensors 6, 7, and 8 and class data from classes 4 through 6.

x2s = TakeColumns[xs, {6, 8}];
y2s = TakeColumns[ys, {4, 6}];
x2v = TakeColumns[xv, {6, 8}];
y2v = TakeColumns[yv, {4, 6}];

For classification, it is advantageous to add a sigmoidal nonlinearity at the output of the RBF network. This constrains the output to the range 0 to 1. Also, a better-conditioned training problem is often obtained if the linear submodel is excluded.

Initialize the RBF network with six neurons, no linear submodel, and a sigmoid output nonlinearity.

rbf = 
      InitializeRBFNet[x2s, y2s, 6, OutputNonlinearity -> Sigmoid, 
        LinearPart -> False]

Train the RBF network for 20 iterations.

{rbf2, fitrecord} = NeuralFit[rbf, x2s, y2s, 20];

The trained RBF network can now be used to classify input vectors by applying the network to them.

Classify the 24th paper data sample from the validation data.

rbf2[x2v[[24]]]

The output of this RBF network is a real-valued function that takes values between 0 and 1. A crisp classification can be obtained in various ways. A simple way is to set values larger than 0.5 to 1 and smaller than 0.5 to 0 in the following manner. (Here, True rather than 1 indicates the sample’s class. )

Map[# > 0.5 &, rbf2[x2v[[24]]]]

The classification can also be displayed with a bar chart in which the correctly classified data is on the diagonal and the incorrectly classified samples are off the diagonal. A crisp classification can be obtained by changing the output nonlinearity from the smooth step sigmoid to the discrete step by entering the option OutputNonlinearity -> UnitStep.

Plot the classification result on the validation data.

NetPlot[rbf2,x2v,y2v,DataFormat -> BarChart,OutputNonlinearity -> UnitStep]

The classification result is not particularly impressive, but due to the randomness in the initialization, you may reevaluate the example, and the result might become better.

Use NetPlot to look at the classification performance improvement during training for each class.

Plot the progress of the classifier on the validation data.

NetPlot[fitrecord,  x2v, y2v, DataFormat -> ClassPerformance, 
      OutputNonlinearity -> UnitStep]

You can repeat the example, selecting different components of the input values and different classes for the output. Because the radial basis functions have local support where they are nonzero, the training often gets trapped in poor local minima. Quite frequently, one of the classes will be totally misclassified. You will see this if you reevaluate the example with a different initialization of the RBF network. Correct classification is obtained when there is one basis function at each class center. If any of the class centers has not managed to attract a basis function during the training, then this class will not be correctly classified.

Feedforward Network

Now try a feedforward neural network. As for RBF networks, a sigmoidal nonlinearity is added to the output so that the outputs are constrained to the interval 0 to 1. The result will vary each time the commands are evaluated due to the random nature of the initialization and training processes.

Load the Mathematica application package Neural Networks and the data.

<< NeuralNetworks`
<< LinearAlgebra`MatrixManipulation`

<< paper.dat;

Initialize a feedforward network without any hidden neurons.

fdfrwrd = 
      InitializeFeedForwardNet[xs, ys, {}, 
        OutputNonlinearity -> Sigmoid]

Train the initialized network for 10 iterations.

{fdfrwrd2, fitrecord} = NeuralFit[fdfrwrd, xs, ys, 10];

The trained network can now be used to classify input vectors by applying the network to them.

Classify sample 27 of the validation data.

fdfrwrd2[xv[[27]]]

A crisp classification is obtained by setting all output values greater than 0.5 to True.

Map[# > 0.5 &, fdfrwrd2[xv[[27]]]]

The 27th sample is correctly classified in class 3.

As with VQ and RBF networks, classification with feedforward networks can also be illustrated by a bar chart with correctly classified data on the diagonal and incorrectly classified data on off-diagonal bars. By choosing OutputNonlinearity -> UnitStep the sigmoids at the outputs are changed to a discrete step. This gives crisp classification.

Plot the classification result on the validation data.

NetPlot[fdfrwrd2,xv,yv,DataFormat -> BarChart,OutputNonlinearity->UnitStep]

Next, plot the classification performance improvement during training for each class.

Plot the progress of the classifier on the validation data.

NetPlot[fitrecord, xv, yv, DataFormat -> ClassPerformance, 
      OutputNonlinearity -> UnitStep]

You can repeat the example using different options for the neural network structure. For example, you can introduce a layer of hidden neurons.

Three types of neural networks have been used to classify the paper quality data: VQ, RBF, and feedforward. What conclusion can you draw from the comparison?
As mentioned before, RBF networks often have problems with local minima, especially if the dimension of the input space is high. To reduce this problem, only three of the available 15 dimensions were used. Of course, when 12 dimensions are neglected, there is a danger that the remaining three dimensions do not contain enough information to separate the classes. Hence, RBF networks are not very good for this problem.
A feedforward network also may have problems with local minima, especially if you change the network and include a hidden layer, but these problems are not as severe as they are for the RBF network. You can test for problems by reevaluating the example a few times. Though the VQ network does not have any problems with local minima in this example, it may with other data. It is hard to say which classifier is the best, but the VQ network was the easiest one to train.