The Neural Network package

How to integrate package commands with normal Mathematica computations

Jonas Sjöberg

Chalmers University, Sweden

Neural Network AddOn package

Neural Networks, http://www.wolfram.com/products/applications/neuralnetworks/ is a tools to train, visualize, and validate neural network models. It supports a comprehensive set of neural network structures in the Mathematica environment. This talk first gives a very short overview of the package and then illustrates how the user can combine pure Mathematica commands with neural network algorithms so that package does not become a stand-alone neural network tool but a strong tool for modelling and data manipulation. There are possibilities to modify the package commands using options but one can also have pure Mathematica manipulations of intermediate neural networks computations.

Package

An overview of supported neural network structures and algorithms is available from  the Main Neural Network Palette and the sub-palettes for each network type.

<< NeuralNetworks`

Structure of the Documentation

({{Neural Networks}, {Load Neural Networks }, {Perceptron}, {Feedforward network}, {Radial basis function network}, {Dynamic neural network}, {Unsupervised network}, {Vector quantization network}, {Hopfield network}, {NetClassificationPlot}})

Typical problems

Function approximation

<< NeuralNetworks`
<<  LinearAlgebra`MatrixManipulation`

Data given

Ndata = 20;
x = Table[10 N[{i/Ndata}], {i, 0, Ndata - 1}];
y =  Sin[x];
ListPlot[AppendRows[x, y]]

Neural networks is a bag of tools for estimating relations in the data.

Noisy data

<<Statistics`ContinuousDistributions`

Ndata = 200;
x1 = Table[10 N[{i/Ndata}], {i, 0, Ndata - 1}];
y1 =  
        Sin[x1] + 
          RandomArray[
            NormalDistribution[0, 1], {Ndata, 
              1}];
ListPlot[AppendRows[x1, y1]]

Classification

The output data indicate class belonging of the data samples. In this case the network is a classifier based on the training data.

The approximation

Feedforward neural network

Illustrated with a figure

sum of functions

Plot[Sigmoid[x], {x, -7, 7}]

or as equation

\(y\&^\)(θ)=g(θ,x)=(∑_(i = 1)^nh)(Subscript[w, i]^2)σ((∑_(j = 1)^n)(Subscript[w, i, j]^1)(Subscript[x, j])+(Subscript[b, j, i]^1))+(b^2)

or the function plot

fdfrwrd = 
        InitializeFeedForwardNet[{Range[2]}, {Range[1]}, {1}, 
          RandomInitialization -> True] ;

fdfrwrd[[1]] = {{{{-1, -1}, {1, -1}, {0, 
                0}}, {{1}, {1}, {0}}}};
Clear[x1, x2];

Plot3D[
        Evaluate[fdfrwrd[{x1, 
                x2}][[1]]], {x1, -8, 8}, {x2, -8, 8}, ViewPoint -> {3.240, -0.038, 0.975}, 
        PlotPoints -> 20];

The approximation

Radial basis function neural network

Illustrated with a figure

sum of functions

Plot[Exp[-(x^2)], {x, -3, 3}, PlotRange -> All]

or as equation

\(\(\(\ \ \ \ \ \ \ \)\(\(y\&^\) \((θ)\) = \(g \((θ, 
            x)\) = \(∑\+\(i = 1\)\%nb\( w\_i\%2\) 
                  e\^\(\(-λ\_i\%2\) \((x - w\_i\%1)\)\^2\) + 
              w\_\(nb + 1\)\%2 + \(χ\_1\) x\_1 +  ... \) + \(χ\_n\) 
            x\_n\)\)\)\)

or the function plot

Clear[x1, x2]

rbf = 
        InitializeRBFNet[{Range[2]}, {Range[1]}, 2, 
          RandomInitialization -> True, 
          LinearPart -> False] ;
nb = 2;
rbf[[1]] = {{Table[Random[Real, {-2, 2}], {2}, {nb}], 
            Table[1.5, {nb}], Table[{1. }, {nb + 1}]}};

Plot3D[
        Evaluate[rbf[{x1, x2}][[1]]], {x1, -3, 3}, {x2, -3, 
          3}, PlotRange -> All, PlotPoints -> 30];

A neural network example: 1

Function Approximation in One Dimension

Consider a function with one input and one output.

<< NeuralNetworks`

Generate for the example and look at the it

Ndata = 20;
x = Table[10 N[{i/Ndata}], {i, 0, Ndata - 1}];
y =  Sin[x];
ListPlot[AppendRows[x, y]]

The training data consist of the input data x and the output y.

Train a network

fdfrwrd=InitializeFeedForwardNet[x, 
        y, {4}]

Fit the network to the data.

{fdfrwrd2, fitrecord} = NeuralFit[fdfrwrd, x, y, 10];

Use the network

The trained FF network can now be used in any way you would like. For example, you can apply it to new input data.

fdfrwrd2[{1.5}]

Evaluate the FF model at several new input values.

fdfrwrd2[{{1.5}, {0.3}, {2.5}}]

You can also obtain a Mathematica expression of the FF network by applying the network to a list of symbols. The list should have one component for each input of the network.

Clear[xx]
fdfrwrd2[{xx}]

Plot the network function

Clear[xx]
fdfrwrd2[{xx}]

You can plot the function of the FF network on any interval of your choice.

Plot[%, {xx, -2, 4}]

If you use NetPlot then you automatically get the plot in the range of your training data. The command relies on the Mathematica command Plot and any supported option may be used.

NetPlot[fdfrwrd2, x, y]

Relation to Mathematica

The network object

fdfrwrd2

The parameters are in the first position

fdfrwrd2[[1]]

The parameter values can be extracted, manipulated with Mathematica commands and inserted again.

The meaning of the parameter format is explained in13.1 Change the Parameter Values of an Existing Network

Relation to Mathematica, cont.

Map[MatrixForm, fdfrwrd2[[1, 1]]]

The dimensions of the matrices relates to the number of inputs, the number of outputs and the number of hidden neurons.

Some normal text information of the network is easily obtained

NetInformation[fdfrwrd2]

The network object can easily be converted to a normal Mathematica expression.

Clear[xx]

fdfrwrd2[{xx}]

A neural network example: 2

Spectral Data for Soft Sensoring

Quite often you have the situation where a physical value is impossible or expensive to measure directly but there exist the possibility to measure other physical values and then compute the important value. This is referred to as soft sensoring.

Load package and the data

<< NeuralNetworks`
<< Graphics`MultipleListPlot`
<< (FileNames["spectral.dat", Map[ToString, $Path], 
              3][[1]]);

The variable x contains absorption spectra in the infrared region of a number of samples, and y contains values of the water contents of the samples. The water contents has been measured with a time consuming and expensive laboratory analysis. The goal is to use the cheap spectra and to compute the water contents.

Check the dimensionality of the data and the number of data. There are 50 frequencies where the absorption is measure. This is a huge input space.

Dimensions[x]

Dimensions[y]

Soft Sensoring, cont.,

Plot the output data. Apparently the data items are sorted depending on the output.

ListPlot[Flatten[y]];

Look at some of the spectra

MultipleListPlot[x[[Range[1, 1000, 100]]], 
        PlotJoined -> True, 
        PlotStyle -> (Map[Hue[#] &, Range[10]/10]), 
        SymbolShape -> None];

Soft Sensoring, cont.,

Divide the data into estimation and validation data. A random order is used since the data are ordered.

randorder = Range[
            1231][[Ordering[Table[Random[], {1231}]]]];

xe = x[[randorder[[Range[600]]]]];
ye = y[[randorder[[Range[600]]]]];
xv = x[[randorder[[Range[601, 1231]]]]];
yv = y[[randorder[[Range[601, 1231]]]]];

Estimate an affine model

ffnet = InitializeFeedForwardNet[xe, ye, {}];

Evaluate the affine model on validation data

NetPlot[ffnet, xv, yv, DataFormat -> NetOutput]

Soft Sensoring, cont.,

NetPlot[ffnet, xv, yv, DataFormat -> ErrorDistribution]

Root-mean-square error

Sqrt[(Transpose[#]  .  #)[[1, 1]]/Length[#]] &[
      yv - ffnet[xv]]

Soft Sensoring, cont.,

Try a feedforward network with 4 neurons

ffnet = InitializeFeedForwardNet[xe, ye, {3}]

{ffnet1, fitrecord} = 
        NeuralFit[ffnet, xe, ye, xv, yv, 15];

NetPlot[ffnet1, xv, yv, DataFormat -> NetOutput];

Soft Sensoring, cont.,

NetPlot[ffnet1, xv, yv, 
      DataFormat -> ErrorDistribution]

Root-mean-square error

Sqrt[(Transpose[#]  .  #)[[1, 1]]/Length[#]] &[
      yv - ffnet1[xv]]

Relation to Mathematica

ffnet1

NetInformation[ffnet1]

Clear[xx]

Maybe it is nicer to keep the neural network object ....

ffnet1[Table[xx[k], {k, 50}]]

Using options to modify the neural network structures

Browser help

The Basis Function in an RBF Network

The Gaussian function is the most commonly used basis function in RBF networks. There is, however, no a priori reason for this choice, and you may use the Neuron option in the initialization to define a different basis function.

RBF network with nb basis functions:

\(\(y\&^\) \((θ)\) = ∑\+\(i = 1\)\%nb\( w\_i\%2\) 
            G \((\(-λ\_i\%2\) \((x - w\_i\%1)\)\^2)\) + 
        b\)

The default value of G(x) is Exp so that the Gaussian function is obtained.

Plot[Exp[-(x^2)], {x, -3, 3}, 
        PlotRange -> All];

Other choices

Another possible choice for the basis function is one that decays linearly with the distance from the basis center.

Plot[-SaturatedLinear[Sqrt[(x^2)]] + 1, {x, -3, 3}, 
        PlotRange -> All];

Now use this basis function in a small example.

Ndata = 10;
x1 = 
        Table[N[{i/Ndata, j/Ndata}], {i, 0, Ndata - 1}, {j, 0, 
            Ndata - 1}];
y1 = Map[Sin[10.  #[[1]] #[[2]]]  &, x1, {2}];
ListPlot3D[y1];
x = Flatten[x1, 1];
y = Transpose[{Flatten[y1]}];

Initialize and train

rbf = 
      InitializeRBFNet[x, y, 3, 
        Neuron -> Function[x, SaturatedLinear[Sqrt[-x]]]]

Train the network

{rbf2, fitrecord} = NeuralFit[rbf, x, y, 20];

Plot the result

NetPlot[rbf2, x, y, DataFormat -> FunctionPlot]

Modify the neural network structures by fixing some parameters

Browser help

The option FixedParameters is used to exclude parameters from the fit.

There are reasons to believe that most of the nonlinear behavior of the hydraulic actuator is near the physical input of the system, where the oil streams into the cylinder. Hence, it could be interesting to include only the past input values of the regressor in the nonlinear part of the model, while keeping the model linear in the past output values.

\(\(y\&^\) \((t)\) = \(a\_1\) y \((t - 1)\) + \(a\_2\) 
          y \((t - 2)\) + \(a\_3\) y \((t - 3)\) + 
        g \((θ, u \((t - 1)\), u \((t - 2)\))\)\)

where g(θ,u(t-1),u(t-2)) is a neural network.

Data

<< NeuralNetworks`
<< actuator.dat;

Divide the data into training and validation data.

ue = u[[Range[512]]];
ye = y[[Range[512]]];
uv = u[[Range[513, 1024]]];
yv = y[[Range[513, 1024]]];

ListPlot[Flatten[u], PlotJoined -> True];

ListPlot[Flatten[y], PlotJoined -> True];

Initialize a network

Before fixing the parameters, an initial model is obtained by invoking training with zero iterations.

na = 3; nb = 2; nk = 1; nh = {3, 3};
{model1, fitrecord} = 
        NeuralARXFit[ue, ye, {na, nb, nk}, FeedForwardNet, nh, 0, 
          LinearPart -> True, 
          CriterionPlot -> False];

model1

It is the first (Subscript[n, a]) rows of w1, corresponding to the past y values, which have to be set to zero, see Section 2.6, Dynamic Neural Networks.

Set parameters to zero

w1 = model1[[1, 1, 1, 1]] ;
MatrixForm[w1]

Set the first (Subscript[n, a]) rows to zero.

model1[[1, 1, 1, 1, {1, 2, 3}]] = 
        Table[0, {na}, {nh[[1]]}];

Check that the manipulation is correct.

w1 = model1[[1, 1, 1, 1]] ;
MatrixForm[w1]

Fix zeroed parameters and train

The zeroed parameters have to be held fixed in the training.

fixparameters = 
      Flatten[Position[Flatten[model1[[1, 1]]], 0]]

{model2, fitrecord} = 
        NeuralARXFit[ue, ye, model1, 15, 
          FixedParameters -> fixparameters];

Test the partial linear model

The trained model can now be tested on the validation data.

NetComparePlot[uv, yv, model2, 
      PredictHorizon -> Infinity]

Writing Your Own Training Algorithms

Browser help

The different options of NeuralFit offer you several training algorithms for feedforward and RBF networks. Its options give you further possibilities to modify the algorithms. Nevertheless, on occasion you may want to develop your own training algorithm.

Many algorithms are based on the derivative of the network with respect to its parameters, and SetNeuralD and NeuralD help you compute it in a numerically efficient way.

((d RBF output)/(d parameters))

1. Given a neural network structure (feedforward or RBF)

2. Call SetNeuralD to generate an optimized function to compute the derivative with respect to the parameters.

3. Call NeuralD to obtain the derivative.

4. Update the parameters and insert in the network.

5. Stop training or go to 3.

Demonstration data

<< NeuralNetworks`

<< NeuralNetworks`
<< Graphics`MultipleListPlot`
<< (FileNames["spectral.dat", Map[ToString, $Path], 
              3][[1]]);

x = x[[Range[100], Range[20]]];
y = y[[Range[100]]];

Check the dimensions of the data.

Dimensions[x]
Dimensions[y]

Compute the derivative

net = InitializeRBFNet[x, y, 2, LinearPart -> False]

Generate the code to calculate the derivative - this line should be called in the beginning of your own training program.

SetNeuralD[net];

Compute the derivative  for the submitted parameter values - this line should be placed inside the loop in the training program.

der = NeuralD[net, x];

Timing[NeuralD[net, x]][[1]]

Dimensions[der]

Compare a pure Mathematica implementation

Replace numerical parameters with symbols

net2 = net;

net2[[1]] = net[[1]] /. (_Real :> Unique[]);

parameters = Flatten[net[[1]]];
parameterSymbols = Flatten[net2[[1]]];

inputs = Table[Unique[], {Dimensions[x][[2]]}];

Express the RBF net as a symbolic expression in the parameters and the inputs

expr = net2[inputs]

derivativeExpr = Map[D[expr, #] &, parameterSymbols];

Put in the numerical values of the weights

derivativeExpr = 
        derivativeExpr /. 
          Inner[Rule, parameterSymbols, parameters, List];

Compute the derivative on the input data

Timing[
        der = Map[derivativeExpr /. Inner[Rule, inputs, #, List] &, 
            x]][[1]]

Dimensions[der]