Neural Networks Package

A neural network consists of

inputs, (Subscript[x, 1]),... (Subscript[x, n])

neurons (or units, basis functions)  Σ σ

Plot[Tanh[x], {x, -4, 4}]

one or several outputs

weights (or parameters) which are adapted in the neural network training

This can however be described by an equation

\(y\&^\)=(b^2)+(∑_(i = 1)^2)\(\(w\_i\%2\) σ \((b\_i\%1 + ∑\+\(j = 1\)\%3\( w\_\(i, j\)\%1\) x\_j)\)\)
=(Subscript[w, 1]^2) σ ((Subscript[w, 1, 1]^1) 
                  (Subscript[x, 1]) + (Subscript[w, 1, 2]^1) (Subscript[x, 2]) + (Subscript[w, 1, 3]^1) (Subscript[x, 3]) + 
                (Subscript[b, 1]^1)) +  (Subscript[w, 2]^2) σ ((Subscript[w, 2, 1]^1) 
                  (Subscript[x, 1]) + (Subscript[w, 2, 2]^1) (Subscript[x, 2]) + (Subscript[w, 2, 3]^1) (Subscript[x, 3]) + 
                (Subscript[b, 2]^1)) + (b^2)

People in the Neural Network Field ...

come from all traditional scientific- and engineering disciplines. Among others

Computer science

Statistics

Engineering

Biology

Medicine

Psychology

Hence, the common interest of neural networks has become a meeting place for different traditions. Due to this there  new terminology in this field which are traditionally known under other names. For example, training or learning, often corresponds to minimization.

Typical problems

Function estimation

<<NeuralNetworks`

<<onedimfunc.dat;
<< LinearAlgebra`MatrixManipulation`

ListPlot[AppendRows[x, y], 
      PlotStyle -> PointSize[0.02]]

One- and two dimensional problems are good for illustration since the result can be plotted. Higher dimensional problems is the real challenge.

Time Series and Dynamic Systems

<< NeuralNetworks`
<< LinearAlgebra`MatrixManipulation`

<< currency.dat;

The daily exchange rate between US Dollar and German Mark. The "system" has memory - dynamics.

ListPlot[Flatten[TakeColumns[ydeu, {2}]]]

The package contains special function to enhance estimation of linear and nonlinear time-series.

Classification and Clustering

<< NeuralNetworks`

<< vqthreeclasses.dat;

NetClassificationPlot[x, y, 
      SymbolStyle -> {Hue[ .0], Hue[ .6], Hue[ .8]}]

Tools

There are many different algorithms and tools to work with these problems. Neural Network is only one among many possibilities. Best tool depends on the problem at hand.

Objectives with the Neural Networks package:

Teaching

The package:

contains a tutorial chapter where a brief theoretical foundation is given (does not replace a book).

different algorithms are illustrated on small examples so that advantages and disadvantages are illuminated.

contains special commands which makes it easier to evaluate a trained network.

contains a rather small set of commands so that the beginner does not get lost. When a user gets more experienced they can use options to modify the algorithms.

supports all common used neural network types, both useful ones and some less useful ....

palettes gives a good overview of available networks and options with links to the documentation. (show them)

Useful Tool for Professional Users

The package:

contains state-of-the-art algorithms.

offers "open ends" so that an advanced user can write their own algorithms and integrate them with the package.

has more general algorithms than other comparable products. In this way the user is given more choices and Mathematica symbolic computation is utilized to make the algorithms efficient.

integrates with all commands available in Mathematica. You can construct complex solutions where the neural net is nothing but one out of many calculations.

Ask a beta tester for an independent point of view about the package

Some beta testers are here at the conference, I am very grateful for your point of views and your feedback.

Examples

Function approximation

Preliminaries

This notebook contains some introductory exercises on function approximation using neural networks.

Load Neural Network package and some additional packages

<< NeuralNetworks`
<<  LinearAlgebra`MatrixManipulation`
<< Statistics`ContinuousDistributions`

The package LinearAlgebra`MatrixManipulation` contains many functions to handle matrices.

Define a true function

trueFunction[xx_] := If[xx < 0, 0.3 xx, Sin[xx]]

Look at the true function

trueFP = Plot[trueFunction[x], {x, -5, 5}]

Define a function to generate a data set

generateData1[Ndata_Integer, noiseVariance_?NumericQ] := 
      Module[{x, y}, 
x = 
          Table[{N[i]}, {i, -5, 5, 
              10/(Ndata - 1)}]; 
y = 
          Map[trueFunction, x, {2}] + 
            RandomArray[
              NormalDistribution[0, noiseVariance], {Ndata, 
                1}]; 
Return[{x, 
            y}];
]

The function generateData1 is called with the number of data and the noise variance.

Generate some data with the true function

{x, y} = generateData1[16, 0.5];

{x2, y2} = generateData1[25, 0.2];

Approximation

A good start is always to look at the data

Look at the data

dataP = 
      ListPlot[AppendRows[x, y], 
        PlotStyle -> PointSize[0.02]]

In this example where the data has been computer generated you can plot the data together with the true function

Look at the data together with the true function

Show[trueFP, dataP];

In this first exercise you will compare a polynomial model with a neural network model

The function polynomialFit is called with the input and output data and it gives a polynomial function.

Initialize a feedforward net

ffnet = 
        InitializeFeedForwardNet[x, y, {5}, 
          RandomInitialization -> True];

Train the network

{ffnet2, fitrecord} = NeuralFit[ffnet, x, y, 5];

Look at the neural network approximation

ffnetFP = 
      NetPlot[ffnet2, x, y, PlotStyle -> PointSize[0.02]]

Plot[ffnet2[{x}], {x, -6, 6}]

Plot the true function, the data, and the neural net together

Show[trueFP, dataP, ffnetFP]

The network can be applied on new data using the evaluation rule

ffnet2[{4}]

or several data at the same time

ffnet2[{{4}, {0.4}, {-3}}]

A mathematical expression corresponding the trained network is obtained by apply it on a symbol

ffnet2[{xx}]

Go back and change the number of neurons.

{ffnet2, fitrecord} = NeuralFit[ffnet, x, y, 30];

<< Graphics`Animation`

Apply[ShowAnimation, 
      NetPlot[fitrecord, x, y, Intervals -> 1,  
        DataFormat -> FunctionPlot, 
        DisplayFunction -> Identity]]

This is overfitting.

A too flexible function is adapted to noisy data.

Not only the function is fitted but also the individual noisy disturbances.

The problem with overfitting increases in higher dimensional problems.

Remedies

Regularization - penalize the size of the parameters which limits the

Stopped search - the minimization of the criterion is stopped before the minimum is reached.

We need a second, independent, data set - validation data.

Stopped search

{ffnet2, fitrecord} = 
        NeuralFit[ffnet, x, y, x2, y2, 30];

ffnetFP = 
      NetPlot[ffnet2, x, y, PlotStyle -> PointSize[0.02]]

Regularization

{ffnet2, fitrecord} = 
        NeuralFit[ffnet, x, y, 30, Regularization -> 0.01];

ffnetFP = 
      NetPlot[ffnet2, x, y, PlotStyle -> PointSize[0.02]]

5.2.3 Function Approximation in Two Dimensions

An FF network can have any number of inputs and outputs. In the previous two examples there was only one input. Here is an example with two inputs.

Read in the Neural Networks package

<< NeuralNetworks`

Generate data and look at the function

Ndata = 10;
x = 
        Table[N[{i/Ndata, j/Ndata}], {i, 0, Ndata - 1}, {j, 0, 
            Ndata - 1}];
y = Map[Sin[10.  #[[1]] #[[2]]]  &, x, {2}];
ListPlot3D[y, MeshRange -> {{0, 0.9}, {0, 0.9}}];
x = Flatten[x, 1];
y = Transpose[{Flatten[y]}];

The training data is placed in x and y, where x is the input data and y is the output data.

You can modify the example by changing the function generating the data and by changing the number of neuron basis functions in the following initialization. Notice that you will obtain slightly different results even if you repeat the example without any changes at all. This is due to the randomness in the initialization algorithm of the FF network.

Check the dimensions of the data

Dimensions[x]
Dimensions[y]

There are 100 input-output data pairs with two-dimensional inputs and one-dimensional outputs.

Initialize an FF network with four neurons

fdfrwrd = InitializeFeedForwardNet[x, y, {4}]

You can apply the initialized network to the input data and plot the network output. Compare the result with the true function in the previous plot.

Look at the initialized FF network

y2 = fdfrwrd[x];
ListPlot3D[Partition[Flatten[y2], Ndata], 
      MeshRange -> {{0, 0.9}, {0, 0.9}}]

So far the network has only been initialized. Now it is time to train it.

Fit the FF network to the data

{fdfrwrd2, fitrecord} = NeuralFit[fdfrwrd, x, y, 20];

You can look at the result by evaluating the trained network at the input data points as follows.

Look at the result with the fitted FF network

y2 = fdfrwrd2[x];
ListPlot3D[Partition[Flatten[y2], Ndata], 
      MeshRange -> {{0, 0.9}, {0, 0.9}}]

Notice that there are usually several local minima. If you repeat the initialization and the training, you obtain different results. 
The plot does not, however, show how the function looks in between the data points. By using NetPlot, which gives a plot based on the Mathematica command, Plot3D, you obtain a plot with any number of evaluation points. If you apply NetPlot to the training record you obtain a graphics array showing the evolution of the function approximation during the training.

NetPlot[fitrecord, x, y, DataFormat -> FunctionPlot]

Apply[ShowAnimation, 
        NetPlot[fitrecord, x, y, Intervals -> 1,  
          DataFormat -> FunctionPlot, 
          DisplayFunction -> Identity]]

If you prefer an animation of the training progress, you can load <<Graphics`Animation` and then change the command to Apply[ShowAnimation,NetPlot[fitrecord,x,y,Intervals->1, DataFormat->FunctionPlot,DisplayFunction->Identity]].

You can obtain a histogram of the errors between the network output and the true output with NetPlot by choosing DataFormat->ErrorDistribution. This might help you to find outliers in the data and to explain if something goes wrong.

NetPlot[fdfrwrd2, x, y, 
      DataFormat -> ErrorDistribution]

Each bar shows the number of samples given an estimation error within the borders of the bars.

Classification and Clustering

4.2.1 Two Classes in Two Dimensions

<<NeuralNetworks`

<< twoclasses.dat;

y2 = Transpose[{1 - y, y}];

The input data is stored in x and the output in y.

The starting point should always be to look at the data.

Look at the data

NetClassificationPlot[x,y2,SymbolStyle->{Hue[.5],Hue[.9]}]

Initialize and train a perceptron using five iterations

{per, fitrecord} = PerceptronFit[x, y];

Classify two new input vectors

per[{{1. , 1.2}, {5.1, 5.3}}]

Display the training progress of the classification

NetPlot[fitrecord, x, y, Intervals -> 3, 
      SymbolStyle -> {Hue[ .5], Hue[ .9]}]

NetPlot[per, x, y, Intervals -> 3, 
      SymbolStyle -> {Hue[ .5], Hue[ .9]}]

Look at the number of correctly and incorrectly classified data objects to each class versus training iterations

NetPlot[fitrecord, x, y, 
      DataFormat -> ClassPerformance]

Intelligent Initialization

Initialize and train a perceptron using the smart initialization

{per, fitrecord} = 
        PerceptronFit[x, y, 5, CriterionLog -> False, 
          RandomInitialization -> False];

The message given tells you that the initialized perceptron correctly classifies all training data and, therefore, no iterative training is necessary. This can also be seen clearly if you plot the result.

Plot the perceptron classifier together with the data

NetPlot[per,x,y,SymbolStyle->{Hue[.5],Hue[.9]}]

Initialize and train a feedforwrd  neural network

ffneti = 
      InitializeFeedForwardNet[x, y2, {1}, 
        OutputNeuron -> Sigmoid]

{ffnet, fitrecord} = NeuralFit[ffneti, x, y2, 15];

NetPlot[ffnet, x, y2, DataFormat -> Classifier, 
      Intervals -> 3, 
      SymbolStyle -> {Hue[ .5], Hue[ .9]}]

NetPlot[ffnet, x, y2, DataFormat -> BarChart, Intervals -> 3, 
      SymbolStyle -> {Hue[ .5], Hue[ .9]}]

NetPlot[fitrecord, x, y2, DataFormat -> ClassPerformance, 
      Intervals -> 3]

Clustering and Classification, several classes

<< NeuralNetworks`

<< vqthreeclasses.dat;

Look at the data and indicate which class they belong to

NetClassificationPlot[x, y, 
      SymbolStyle -> {Hue[ .0], Hue[ .6], Hue[ .8]}]

Train a vector quantization network

{vq, fitrecord} = VQFit[x, y, 7, 8];

Illustrate the training

NetPlot[fitrecord, x, y, DataFormat -> DataMapArray, 
      SymbolStyle -> {Hue[ .0], Hue[ .6], Hue[ .8]}, 
      Voronoi -> False]

NetPlot[fitrecord, x, y, DataFormat -> DataMap, 
      SymbolStyle -> {Hue[ .0], Hue[ .6], Hue[ .8]}, 
      Voronoi -> False]

End result

NetPlot[vq, x, y, DataFormat -> DataMap, 
      SymbolStyle -> {Hue[ .0], Hue[ .6], Hue[ .8]}, 
      Voronoi -> False]

Display classification borders

NetPlot[vq, x, y, DataFormat -> DataMap, 
      SymbolStyle -> {Hue[ .0], Hue[ .6], Hue[ .8]}, 
      Voronoi -> True]

Assume that the true class is unknown. We can still do classification of the data:

{unsup, fitrecord} = UnsupervisedNetFit[x, 10];

NetPlot[unsup, x, y, DataFormat -> DataMap, 
      SymbolStyle -> {Hue[ .0], Hue[ .6], Hue[ .8]}]

Add a neighbor feature - Kohonen network

{unsup, fitrecord} = 
        UnsupervisedNetFit[x, 10, SOM -> {10, 1}];

NetPlot[unsup, x, y, DataFormat -> DataMap, 
      SymbolStyle -> {Hue[ .0], Hue[ .6], Hue[ .8]}]

You can find connected structures

{unsup, fitrecord} = 
        UnsupervisedNetFit[x, 10, SOM -> {10, 1}, 
          Connect -> True];

NetPlot[unsup, x, y, DataFormat -> DataMap, 
      SymbolStyle -> {Hue[ .0], Hue[ .6], Hue[ .8]}]

Dynamic problems - Prediction of Currency Exchange Rate

In this example, data consist of the daily exchange rates of the British pound and the German mark compared to the U.S. dollar from the beginning of 1987 to the end of 1997.

<< NeuralNetworks`
<< LinearAlgebra`MatrixManipulation`

<< currency.dat;

Dimensions[ybr]
Dimensions[ydeu]

Exchange rates from approximately 2850 days are available. There are four rates given for each day, representing the opening, high, low, and closing prices.

Concentrate now on the German mark.

Plot the high price of the German mark versus the day number

ListPlot[Flatten[TakeColumns[ydeu, {2}]]]

Suppose you want to predict the opening price using the exchange rates from the previous day. Therefore, the opening price is defined as the output of the process, which will be stored in y, and the three other rates are defined as inputs and will be stored in u.

Divide the columns into input and output

u = TakeColumns[ydeu, {2, 4}];
y = TakeColumns[ydeu, {1}];

The predictor model can be described by the following equation:

\(\(y\&^\) \((t)\) = g \((θ, x \((t)\))\)\)

Here \(TraditionalForm\`y\&^\)(t) is the prediction of the output y(t), the function g is the neural network model, θ represents the parameters of the model, and x(t) is the model’s regressor. The regressor is given by the following equation:

x (t) = [y (t - 1) (Subscript[u, 1]) (t - 1) (Subscript[u, 2]) (t - 
                1) (Subscript[u, 3]) (t - 1)]^T

Divide the data into training and validation data

ue = u[[Range[1000]]];
ye = y[[Range[1000]]];
uv = u[[Range[1001, 2000]]];
yv = y[[Range[1001, 2000]]];

To obtain the regressor in the form (12.), the regressor indices have to be chosen as follows: (Subscript[n, a])={1}, (Subscript[n, b])={1,1,1}, and (Subscript[n, k])={1,1,1}.

Estimate a linear model for the currency prediction

{model1, fitrecord} = 
        NeuralARXFit[ue, ye, {{1}, {1, 1, 1}, {1, 1, 1}}, FeedForwardNet, {}, 
          CriterionPlot -> False];

A linear model is just a linear combination of the regressor components and a DC-level parameter; that is, a linear model (12.) can be expressed as

\(\(y\&^\) \((t)\) = \(a\_1\) 
          y \((t - 1)\) + \(b\_1\) \(u\_1\) \((t - 
              1)\) + \(b\_2\) \(u\_2\) \((t - 1)\) + \(b\_3\) \(u\_3\) \((t - 
              1)\) + b\_4\)

Look at the parameters

model1[[1]][{yy[t - 1], u1[t - 1], u2[t - 1], 
        u3[t - 1]}]

Evaluate the one-step prediction on validation data

NetComparePlot[uv, yv, model1, PredictHorizon -> 1, 
      ShowRange -> {4, 100}]

It is now time to try nonlinear models.

Train a feedforward network with two neurons on the exchange rate data

{model2, fitrecord} = 
        NeuralARXFit[ue, ye, {{1}, {1, 1, 1}, {1, 1, 1}}, 
          FeedForwardNet, {2}, uv, yv, 5, LinearPart -> True, 
          Separable -> False];

The improvement of the neural network during the training is very small because the initialization of the network uses least-squares to fit the linear parameters.

Compute the one-step prediction on the same data interval used for the linear model

NetComparePlot[uv, yv, model2, PredictHorizon -> 1, 
      ShowRange -> {4, 100}]

The example illustrates that it is possible to predict the opening exchange rate using the rates from the previous day. The relationship is obviously nonlinear, since the nonlinear models based on feedforward and RBF networks performed slightly better than the linear model.