Neural Networks package, Tutorial

Time-series prediction and Classification

Jonas Sjöberg

Chalmers University, Sweden

Feedforward Neural Networks: Hidden Units

Artificial neurons are often called hidden units.

σ(∑(Subscript[x, i])(Subscript[w, i])+b)

({{{(Subscript[w, i]),b}=parameters or weights}, {{(Subscript[x, i])}=inputs to unit}})

σ = activation function or, equivalently, the sigmoid function. The most common choice for the sigmoid function is

σ(x)=(1/(1+(e^(-x))))

<<NeuralNetworks`
<<actuator.dat;
Plot[Sigmoid[x],{x,-8,8}]

For other possibilities see 13.3.0 NeuralNetworks.

Feedforward Neural Networks: Network Illustration

One-hidden-layer feedforward network

with mathematical definition

({{(OverHat[y])(θ,x)=g(θ,x)}, {=(∑_(i=1)^n)(Subscript[w, i]^2)σ((∑_(j=1)^d)(Subscript[w, i,j]^1)(Subscript[x, j])+(Subscript[b, i]^1))+(b^2)}, {θ=[(Subscript[w, 1]),(Subscript[b, 1]),(Subscript[w, 2]),(Subscript[b, 2])]}})

This network has two sigmoids (hidden units)

yt={Range[1]};
xt={Range[2]};
fdfrwrd=InitializeFeedForwardNet[xt,yt,{1},RandomInitialization->True] ;
fdfrwrd[[1]]={{{{-1,-1},{1,-1},{0,0}},{{1},{1},{0}}}};

Plot3D[Evaluate[fdfrwrd[{x1,x2}][[1]]],{x1,-8,8},{x2,-8,8},ViewPoint->{3.24,-0.038,0.975},PlotPoints->20]

Pure time series: NAR

(OverHat[y])(t)=g(θ,x(t))

NAR model based on a neural network

{model1,fitrecord}=NeuralARFit[y,{2},FeedForwardNet,{},0,CriterionPlot->False];

NetInformation[model1]

Nonlinear AutoRegressive with auXilliary input: NARX

(OverHat[y])(t)=g(θ,x(t))

NARX model based on a neural network

{model1,fitrecord}=NeuralARXFit[u,y,{2,2,2},FeedForwardNet,{},0,CriterionPlot->False];

NetInformation[model1]

User parameters

Number of lagged outputs, (Subscript[n, a]) and, possibly lagged inputs, (Subscript[n, b]), (order of the system).

The number of neurons, and layers of neurons.

Type of network, feedforward, or radial basis function, activation function.

Possibly exclude some parameters to reduce risk for overfitting.

Predict current exchange rate

<<NeuralNetworks`

<<currency.dat;

ListPlot[Flatten[ybr[[All,{2}]]]]

y=ybr[[All,{1}]];

ye=y[[;;1000]];
yv=y[[1001;;2000]];

Linear Model

{model1,fitrecord}=NeuralARFit[ye,{3},FeedForwardNet,{},CriterionPlot->False,CriterionLog->False];

model1

fitrecord

model1[[1]][{"y[t-1]","y[t-2]","y[t-3]"}]

NetComparePlot[yv,model1,PredictHorizon->1,ShowRange->{4,100}]

First Neural Network

{model2,fitrecord}=NeuralARFit[ye,{3},FeedForwardNet,{2},yv,30,LinearPart->True,Separable->False];

NetComparePlot[yv,model2,PredictHorizon->1,ShowRange->{4,100}]

Try different number of lags

ParallelNeeds["NeuralNetworks`"]

DistributeDefinitions[ye,yv]

ModelList=ParallelMap[NeuralARFit[ye,{#},FeedForwardNet,{2},yv,30,LinearPart->True,Separable->False,CriterionPlot->False,CriterionLog->False]&,Range[5]];

Performance on validation data for all models

ListPlot[Map[Min[(CriterionValidationValues/.#)]&,ModelList[[All,2,2]]],PlotStyle->PointSize[Large]]

Best one

{model3,fitrecord}=ModelList[[2]];

NetComparePlot[yv,model3,PredictHorizon->1,ShowRange->{4,100}]

Try different number of neurons

ModelList=ParallelMap[NeuralARFit[ye,{2},FeedForwardNet,{#},yv,30,LinearPart->True,Separable->False,CriterionPlot->False,CriterionLog->False]&,Range[5]];

Performance on validation data for all models

ListPlot[Map[Min[(CriterionValidationValues/.#)]&,ModelList[[All,2,2]]],PlotStyle->PointSize[Large]]

Best one

{model4,fitrecord}=ModelList[[1]];

NetComparePlot[yv,model4,PredictHorizon->1,ShowRange->{4,100}]

Try different type of neuron, and try various numbers of them

ModelList=ParallelMap[NeuralARFit[ye,{2},FeedForwardNet,{#},yv,30,LinearPart->True,Separable->False,CriterionLog->False,CriterionPlot->False,Neuron->Tanh]&,Range[5]];

Performance on validation data for all models

ListPlot[Map[Min[(CriterionValidationValues/.#)]&,ModelList[[All,2,2]]],PlotStyle->PointSize[Large]]

Best one

{model5,fitrecord}=ModelList[[2]];

NetComparePlot[yv,model5,PredictHorizon->1,ShowRange->{4,100}]

Load data finance, try some different model parameters, parallelize?, palette and help browser to change neuron
put some parameters to zero- documentation.

Possible extentions

Several variables (signals) at the same time, then the relationship between the signals is also estimated.

Include external signals (inputs) which (possibly) influence the modeled variable.

Relations to Built-in commands

The models in the examples could also have been defined and calculated with the following commands (new in 7)

LinearModelFit

NonlinearModelFit

That would, however, asked for some more programing effort from the user.

More Advanced Nonlinear Models .....

.... can be defined and fitted with the coming System Identification package, see earlier talk at this conference.

If there is time: classification

MouseOver