Selecting and Fitting Nonlinear Models to Time-Series Data from Dynamic Systems

Handling the Overfitting Problem with the Neural Network Package

Jonas Sjöberg

Chalmers university, Sweden

An estimated function might have poor performance on new data

x=Table[{k},{k,-5,5,1}];
y=1.5 x+5 Table[{RandomReal[]},{k,-5,5,1}];
y2=1.5 x+5 Table[{RandomReal[]},{k,-5,5,1}];
dataP=ListPlot[{Transpose[{Flatten[x],Flatten[y]}]},PlotMarkers->■]

Fit two models to the data.

polynomialFit[x_?MatrixQ,y_?MatrixQ,degree_Integer]:=With[{pol=Fit[Join[x,y,2],Table[((#1)^i),{i,0,degree}],#1]},pol&]
polyF=polynomialFit[x,y,8];
polyFP=Plot[polyF[x],{x,-5,5}];
polyF2=polynomialFit[x,y,1];
polyFP2=Plot[polyF2[x],{x,-5,5}];

Here is the estimation data shown with both a flexible model and a constrained model (a straight line). Which is better?

Show[dataP,polyFP,polyFP2]

The models can be tested on a second, independent data set

dataP2=ListPlot[{Transpose[{Flatten[x],Flatten[y2]}]},PlotMarkers->●];Show[dataP2,polyFP,polyFP2]

In general, we want to fit the unknown function and not the noisy data.

Feedforward Neural Networks: Hidden Units

Artificial neurons are often called hidden units.

σ(∑(Subscript[x, i])(Subscript[w, i])+b)

({{{(Subscript[w, i]),b}=parameters or weights}, {{(Subscript[x, i])}=inputs to unit}})

σ = activation function or, equivalently, the sigmoid function. The most common choice for the sigmoid function is

σ(x)=(1/(1+(e^(-x))))

<<NeuralNetworks`
Plot[Sigmoid[x],{x,-8,8}]

For other possibilities see 13.3.0 NeuralNetworks.

Feedforward Neural Networks: Network Illustration

This is a graphical description of a one-hidden-layer feedforward network.

Here is a mathematical definition of a feedforward network.

({{(OverHat[y])(θ,x)=g(θ,x)}, {=(∑_(i=1)^n)(Subscript[w, i]^2)σ((∑_(j=1)^d)(Subscript[w, i,j]^1)(Subscript[x, j])+(Subscript[b, i]^1))+(b^2)}, {θ=[(Subscript[w, 1]),(Subscript[b, 1]),(Subscript[w, 2]),(Subscript[b, 2])]}})

Feedforward Neural Networks: A Look at the Network Function

This network has one sigmoid (hidden unit) and two input signals.

y={Range[1]};
x={Range[2]};
fdfrwrd=InitializeFeedForwardNet[x,y,{1},RandomInitialization->True] ;
fdfrwrd[[1]]={{{{-1},{1},{0}},{{1},{0}}}};

x1=.;x2=.;

Plot3D[Evaluate[fdfrwrd[{x1,x2}][[1]]],{x1,-8,8},{x2,-8,8},ViewPoint->{3.24,-0.038,0.975}]

This network has two sigmoids (hidden units)

fdfrwrd[[1]]={{{{-1,-1},{1,-1},{0,0}},{{1},{1},{0}}}};

Plot3D[Evaluate[fdfrwrd[{x1,x2}][[1]]],{x1,-8,8},{x2,-8,8},ViewPoint->{3.24,-0.038,0.975},PlotPoints->20]

Feedforward Neural Networks: Training / Learning

Consider the parameter criterion

(Subscript[V, N])(θ)=(1/(2N))(∑_(t=1)^N)|y(t)-g(x(t),θ)(|^2)

and the minimization algorithm

((OverHat[θ])^((i+1)))=((OverHat[θ])^((i)))-(Subscript[μ, i])(Subscript[R, i]^(-1))(Subscript[V, N]^′)((θ^((i))))

((OverHat[θ])^((0))): initial parameter guess
(Subscript[μ, i]): step length to ensure descent updates
(Subscript[R, i]): matrix changing the search direction

Feedforward Neural Networks: Overtraining

During training, the model improves, then after a certain number of iterations, it starts to worsen.

Needs["PlotLegends`"];ListPlot[Table[{x,f[x]},{f,{2/(1+#)&,2/(1+#)+0.04#&}},{x,0,20}],PlotRange->{{0,20},{0,2}},AxesOrigin->{0,0},AxesLabel->{Style["# iterations",Large],Style["RMS error",Large]},PlotMarkers->Automatic,Joined->True, PlotLegend->{Style["Training data",16],Style[ "Validation data",16]},LegendPosition->{0.2,-0.2}]

This is seen by computing the criterion on validation data. Section 8.2.3, Bias-Variance Tradeoff: Avoiding Overfitting,.

Model Selection – Overfitting: Statistical Properties

There are two (and only two) problems associated with the identification problem:

These give rise to two types of errors:

Model Selection – Overfitting: The Curse of Dimensionality

The interpolation/extrapolation problem becomes worse in higher dimensions:

f:(ℝ^d)⟶ℝ

It may be difficult to approximate f if d is large (d≥3).

This is called the curse of dimensionality.

Plot3D[Evaluate[fdfrwrd[{x1,x2}][[1]]],{x1,-8,8},{x2,-8,8},ViewPoint->{3.24,-0.038,0.975},PlotPoints->20]

Model Selection – Overfitting: Some Standard Theory

Assume that the data is produced according to

y(t)=f(x(t))+e(t)

where f is the unknown function and e is a disturbance.

A common assumption is the e(t) can be described as (white) noise.

The best available model within the model structure is

({{(SuperStar[g])(x(t))=g((SuperStar[θ]),x(t))}, {(SuperStar[θ])=arg (min_θ) E∥g(θ,x(t))-f(x)(∥^2)}})

Note that (SuperStar[θ]) depends on the properties of x.

Model Selection – Overfitting: Three Error Sources

The noise e(t) is given by:

(Subscript[λ, 0])=E(e^2)(t)

The bias error is

B=E∥f(x(t))-(SuperStar[g])(x(t))(∥^2)

The variance error is

V=E∥g((Subscript[(OverHat[θ]), N]),x(t))-(SuperStar[g])(x(t))(∥^2)

The important relation

V=(Subscript[λ, 0])((dim θ)/N)

holds, provided the error is evaluated for the same properties of x as those of the estimation data (training set), and that the bias is “small”.

Note that this relation is independent of x and the particular model structure.

Model Selection – Overfitting: The Total Error

The total error is the sum of the random error and the bias error

E∥y(t)-g((Subscript[(OverHat[θ]), N]),x(t))(∥^2)=(Subscript[λ, 0])(1+(m/N))+E∥ f(x(t))-(SuperStar[g])(x(t))(∥^2)

where m is the number of estimated parameters.

In choosing a model structure that minimizes the total error, a good tradeoff is one where the bias error and the random error are of the same order of magnitude.

Hence, it is never advisable (or realistic) to neglect the bias error.

Model Selection – Overfitting: The Bias-Variance Tradeoff: Two Contributions to the Miss-Fit

The variance error is due to the noise in the data.

The bias error results from the fact that the model cannot describe the system exactly.

Model Selection – Overfitting: The Parsimonious Principle

A model should not be more flexible than necessary.

Needs["PlotLegends`"];ListPlot[Table[{x,f[x]},{f,{0.04#&,2/(1+#)&,2/(1+#)+0.04#&}},{x,0,20}],PlotRange->{{0,20},{0,2}},AxesOrigin->{0,0},AxesLabel->{Style["# parameters",Large],Style["RMS error",Large]},PlotMarkers->None,Joined->True,LegendSize->0.5, PlotLegend->{Style["Variance",16],Style[ "Bias",16],Style[ "Total error",16]},LegendPosition->{0.2,-0.4}]

⇒

Impose restrictions on g(·,·), for example, by

Overlearning: Regularization

Penalize the size of the parameters by introducing a second term in the criterion

W(θ)=(Subscript[V, N])(θ)+δ|θ-(Subscript[θ, 0])(|^2)
⇒
(Subscript[(OverBar[V]), N]^δ)=(Subscript[λ, 0])(1+((OverTilde[m])/N))
(OverTilde[m])=(∑_(i=1)^m)((Subscript[σ, i]^2)/((((Subscript[σ, i])+δ))^2))
 (Subscript[σ, i])=  eigenvalues of Q=(((d^2)(Subscript[V, N])(θ))/((dθ)^2))

Here is an interpretation of the variable (OverTilde[m]).

(OverTilde[m])≈number of eigenvalues of Q that are larger than δ

Needs["PlotLegends`"];ListPlot[Table[{x,f[x]},{f,{0.04#&,2/(1+#)&,2/(1+#)+0.04#&}},{x,0,20}],PlotRange->{{0,20},{0,2}},AxesOrigin->{0,0},AxesLabel->{Style["1/δ",Large],Style["RMS error",Large]},PlotMarkers->None,Joined->True,LegendSize->0.5, Ticks->None,PlotLegend->{Style["Variance",16],Style[ "Bias",16],Style[ "Total error",16]},LegendPosition->{0.2,-0.1}]

Overlearning: Stopped Search

The iterative minimization is described by

({{(Subscript[(OverHat[θ]), N]^((i+1)))=(Subscript[(OverHat[θ]), N]^((i)))-μH(Subscript[V, N]^′)((Subscript[(OverHat[θ]), N]^((i))))}, {(Subscript[(OverHat[θ]), N]^((0)))=(θ^(#))}})

The regularization can be implemented as a stopped search.

Needs["PlotLegends`"];ListPlot[Table[{x,f[x]},{f,{0.04#&,2/(1+#)&,2/(1+#)+0.04#&}},{x,0,20}],PlotRange->{{0,20},{0,2}},AxesOrigin->{0,0},AxesLabel->{Style["# iterations",Large],Style["RMS error",Large]},Joined->True,LegendSize->0.5, PlotMarkers->Automatic,PlotLegend->{Style["Variance",16],Style[ "Bias",16],Style[ "Total error",16]},LegendPosition->{0.2,-0.1}]

Example

In this small example (see 7.5.3 Example), you will see how to use stopped search and regularization to handle the bias-variance tradeoff.

<<NeuralNetworks`

Generate and look at some noisy data.

Ndata=30;
trueFunction[xx_]:=If[xx<0,0.3 xx,Sin[xx]]
x=Table[{N[i]},{i,-5,5,10/(Ndata-1)}];
y=Map[trueFunction,x,{2}]+RandomReal[NormalDistribution[0,0.4],{Ndata,1}];

Show[Plot[trueFunction[x],{x,-5,5}],ListPlot[Join[x,y,2]]]

Divide the data into training data and validation data.

xt=x[[Range[1,Ndata,2]]];
yt=y[[Range[1,Ndata,2]]];
xv=x[[Range[2,Ndata,2]]];
yv=y[[Range[2,Ndata,2]]];

Initialize a feedforward network.

fdfrwrd=InitializeFeedForwardNet[xt,yt,{4},RandomInitialization->LinearParameters]

Look at the initialized network, the true function, and the training data.

Show[Plot[{fdfrwrd[{a}],trueFunction[a]},{a,-5,5},PlotStyle->{{},{Dashing[{0.05,0.05}]}}],ListPlot[Join[xt,yt,2]]]

Train the network.

{fdfrwrd1,fitrecord}=NeuralFit[fdfrwrd,xt,yt,xv,yv,50,Separable->False];

Plot the obtained estimate.

Show[Plot[{fdfrwrd1[{a}],trueFunction[a]},{a,-5,5},PlotStyle->{{},{Dashing[{0.05,0.05}]}}],ListPlot[Join[xt,yt,2]]]

Introduce the parameters at the final iteration.

parameters=ParameterRecord/.fitrecord[[2]];
fdfrwrd1[[1]]=Last[parameters];

Look at the estimate without using stopped search.

Show[Plot[{fdfrwrd1[{a}],trueFunction[a]},{a,-5,5},PlotStyle->{{},{Dashing[{0.05,0.05}]}}],ListPlot[Join[xt,yt,2]]]

Train the network using regularization.

{fdfrwrd1,fitrecord}=NeuralFit[fdfrwrd,xt,yt,30,Regularization->0.001,Separable->False];

Look at the estimate obtained using regularization.

Show[Plot[{fdfrwrd1[{a}],trueFunction[a]},{a,-5,5},PlotStyle->{{},{Dashing[{0.05,0.05}]}}],ListPlot[Join[xt,yt,2]]]

Compare the result with those obtained using stopped search and normal learning.

Modify the design parameters and repeat the procedure. Try networks of different sizes and networks with several layers. Try an RBF network.

Modeling Principles and Approaches: Different Techniques for Handling the Bias-Variance Tradeoff

Traditionally, the tradeoff is handled by choosing the number of basis function depending on the number of estimattion data and noise level. Now we have a large number of alternative techniques.

Best number of basis functions depend on noise level, or equivalent, the inverse of the number of training data.

Needs["PlotLegends`"];
(* Pick out the minimum *)
Manipulate[data=Table[{x,f[x]},{f,{λ#&,2/(1+#)&,2/(1+#)+λ#&}},{x,0,20}];
Show[ListPlot[{data[[3,Position[data,Min[data[[3,All,2]]]][[1,2]]]]},PlotMarkers->],ListPlot[data,Joined->True, PlotMarkers->Automatic],PlotRange->{{0,20},{-0.01,2}},AxesOrigin->{0,0},AxesLabel->{Style["# basis functions",Large],Style["RMS error",Large]},ImageSize->600],{λ,0.01,0.1}]
(* data=.; *)

Modeling Principles and Approaches: The Best Basis Expansion Depends on the Amount of Data and Noise Level

The plot illustrates the bias error as a function of the number of basis functions for two different basis expansions.

Needs["PlotLegends`"];x=.;

Plot[{2/(1+x/5),0.2+2/(1+x)},{x,0,80},PlotRange->{{0,80},{0,2}},AxesOrigin->{0,0},AxesLabel->{Style["# parameters",Large],Style["RMS error",Large]},LegendSize->0.5, PlotLegend->{Style[ "Bias 2",16],Style[ "Bias 1",16]},AspectRatio->1/2,ImageSize->600,LegendPosition->{0.2,-0.2}]

Modeling Principles and Approaches: The Best Model Size Depends on the Number of Data

In general, there is no best model set. Instead, the number of available estimation data, or, equivalently, the noise level, points determines the level of detail you can achieve to estimate the unknown function.

The variance contribution to the misfit is (Subscript[λ, 0])(m/N).

Hence, a decreased noise level has the same effect as an increased number of estimation data points.

A small model with few basis functions is best for a small amount of data.

Needs["PlotLegends`"];x=.;λ=0.04;Manipulate[Plot[{λ x/5,2/(1+x/5),0.2+2/(1+x),2/(1+x/5)+λ x/5,0.2+2/(1+x)+λ x/5},{x,0,80},PlotRange->{{0,80},{0,2}},AxesOrigin->{0,0},AxesLabel->{Style["# parameters",Large],Style["RMS error",Large]},LegendSize->0.8, PlotLegend->{Style["Variance",16],Style[ "Bias 2",16],Style[ "Bias 1",16],Style[ "Total error 2",16],Style[ "Total error 1",16]},AspectRatio->1/2,ImageSize->600,LegendPosition->{0.5,-0.3}],{λ,0.01,0.1}]

A large model with many basis functions is best for when there is a large amount of data.

Conclusions / Questions??