Understanding the 
Back-Propagated Neural 
Network Using Mathematica Graphics

Visualizations illustrate learning in an artificial neural network.

by Steve Hunka and Mike Carbonaro

An excellent introduction to the simulation of neural networks in the context of Mathematica is given by Freeman [Freeman 1994] in the text Simulating Neural Networks with Mathematica.  Of the several algorithms described and detailed using Mathematica, the back-propagated neural network (BPNN) is a well behaved algorithm which can be used in a variety of applications by students and researchers in different disciplines.  The general purpose for using the BPNN is to classify as correctly as possible a set of exemplars based upon their varying attributes.  In some instances, interest is only in whether the network carries out a successful classification, but in other instances common in the behavioral sciences, it is considered important to understand what the network has "learned", e.g., what distinguishing attributes of the exemplars are identified by the network for a successful classification [Elman1993].  By expanding upon the  BPNN algorithm provided by Freeman, and developing further graphical representations of the dynamic and final state characteristics of the network using Mathematica, a better understanding of what a network has learned can be obtained.

THE BACK-PROPAGATED NEURAL NETWORK

A simple BPNN can be described as having a set of "input cells" which receive information about the attributes of the exemplars, a layer of "hidden cells" which receive information from each input cell, and one or more "output cells" which receive information from all the hidden cells.   A simple network is illustrated in Figure 1.

Figure 1. A simple neural network consisting of 3 input cells,
	2 hidden cells, and 1 output cell.

At each input cell, one numerically encoded attribute of an exemplar is passed on directly to each cell in the hidden layer.  Each cell of the hidden layer takes a linearly weighted sum (s) of all its inputs from the cells of the input layer, transforms the result by the logistic function y=1/(1+e-s), and passes the result (a response) to all the cells of the output layer.  Each output cell behaves in a manner similar to a hidden cell by transforming a linearly weighted sum of all its inputs using the logistic function to provide the network's response.  Random weights are used to start the process, and continuously adjusted (back-propagated to the output and hidden layer cells) according to a least squares criterion based on the difference between the desired network response and the obtained response each time an exemplar is presented. The adjustment is usually multiplied by a constant called the "learning rate parameter".  Each presentation of an exemplar is called an iteration, and the presentation of the complete set of exemplars an epoch.  The number of epochs required to reach an acceptable level of classification accuracy of the exemplars for some problems can be several thousands [Istvan et al 1995], and for some input data and configuration of cells a local minimum may prevent an adequate solution.

THE BPNN ALGORITHM AND FREEMAN' S T AND C PROBLEM

In order to provide the student with a more exploratory simulation, the basic algorithm given by Freeman is expanded to include the following user options:

1.  Generation of random starting weights within defined boundaries, or to use weights from a previous run of the network.
2.  At the end of a prescribed number of epochs to increase or decrease the span of the exemplars presented to the network.  This option allows exploration of network behavior when previously "unseen" exemplars are introduced together with those already presented. 
3. Increase the frequency of presentation of selected exemplars, i.e., provide more "practice" for the network in classifying specific exemplars.
4.  Setting the number of epochs, after which the results of the network are output together with the desired answers for the user to judge whether stability or a desired level of classification accuracy has been reached,  and whether additional epochs are required.
5. Change the learning rate parameter for the adjustment of the weights for cells at the output and hidden layers. 
6. Whether the presentation order of the exemplars is fixed or random. 
7. Writing to separate files the weights used and the output from the hidden and output cells.

Freeman's T-and-C problem requires the BPNN to distinguish between the characters "T" and "C" given four different orientations of each character.  Each character’s orientation  is encoded into a 3 by 3 matrix with each matrix (exemplar) being a rotation of 0, 90, 180, or 270 degrees. The eight exemplars are illustrated in Figure 2.

Figure 2.  The 3x3 character matrices forming the input patterns for the T-and-C problem.

Each shaded and unshaded element, i.e., an attribute, of the character matrix is encoded as 0.9 and 0.1 respectively, and the 9 elements of each matrix defines an exemplar of the character for input to the network.  The desired output from the network is encoded as 0.9 for T exemplars and 0.1 for C exemplars. The network Freeman uses for this problem contains 9 input cells, 3 hidden cells, and 1 output cell.  Replicating Freeman's network, and running it for 75 epochs (600 iterations) produces five files or data sets: (1) hidden cell weights of order (75,3,9); (2) hidden cell response of order (75,8,3); (3) output cell weights of order (75,3,1);  (4) output cell response of order (75,8,1); and an error output of order (600,1).  For purposes of illustration, an error file (errfile), a hidden response file (hidrfile), and a hidden weights file (hdwfile) are used.

Mathematica's GRAPHICAL REPRESENTATION OF 
THE DYNAMIC AND FINAL STATE CHARACTERISTICS

In order to provide the researcher and student with easy access to the plotting capabilities of Mathematica, five utility functions, four calling functions, and four plotting functions are provided.  The two utility functions kbin3 and kbinstrg  provide for the interactive input and checking of a numeric and string input from the keyboard, respectively. The functions getFdata and timeDate, interactively allow the user to select the data file to be used, and provide information about its date of creation.  These and other functions referred to in this article are listed in the accompanying notebook BPNNAppendix.nb.

 

A plot of the squared error is made first to illustrate the basic behavior of the network as provided by Freeman. To illustrate further behavior of the network, 3-dimensional plots of the hidden cell responses and weights are also made.  Finally, an array of 2-dimensional plots is made of the hidden cell weights arranged corresponding to their association with the cells of the 3x3 character matrix.

Plot of the Squared Error

Although a plot of the squared error at each output cell indicates the nature of a network's convergence, such a plot does not provide an indication as to what the network has learned.  The calling function errAnlz and plotnLists  used to make the plot of the 

squared error is given below.

errAnlz:= Module[{ttl, data, extnt, xunits},
  Print["Plot of Sqd-error at Output Cells"];
  {data, extnt} = getFdata;
  If[0 == extnt, Print["No data read"]; Return[]];
  ttl = " Sqd-error for Iterations=";
  ttl = StringInsert[ttl, ToString[extnt], -1]; 
  plotnLists[data, ttl, "iterations", 1, 0, 0];
];

The arguments for the function plotnLists are: m_ the matrix values to plot; tl_  a plot title; xu_ a label for the x-axis; ps_  set to 1 for single line plots, else 0; pm_ set to 1 for multiple line plots, else 0; and pn_ set to 1 for end of line numbering, else 0. 
Since some networks use many output cells, the function plotnLists provides up to 12 different colored lines for each output cell's error plot.  The different RGB colors are obtained by left shifting the vector colr 3 positions for each plot made by the RotateLeft operation.  Executing the function errAnlz, provides the following results.

errAnlz

Plot of Sqd - error at Output Cellserrfile was foundFile created:
Oct 23,1995 at 10 : 5Data read from file = allReturned data dimension = {600,  1}

Plotting the Hidden Cell Responses

The hidden cell responses are the outputs from each hidden cell for each exemplar taken at the end of each epoch and form a matrix of order (75,8,3).  Interest is in observing the discrimination made by hidden cells between the four T-and-C exemplars over the 75 epochs.  To provide this plot the function hdrAnlz1, included in the accompanying notebook BPNNAppendix.nb, obtains the data from the appropriate file as specified by the user and calls splt3d to carry out the plot.  The function splt3d[d_,ttl_] takes as arguments the data to be plotted (d_) and a title (ttl_), and allows the user to specify interactively which hidden cells are to be used, whether the start or end point should be marked for each exemplar-epoch sequence plotted, whether the plot points should be joined, the point size (if plot points are not joined), and whether single plots in addition to the combined plot of each exemplar-epoch sequence is to be displayed. The numbering of each exemplar-epoch sequence is made by a modification to the graphics object (pobj) created by ScatterPlot3D which is available in the Package Graphics`Graphics3D`.

Executing hdrAnlz1 using the file hidrfile produces the following result:

hdrAnlz1

Response of 3 hidden cells to exemplarsXaxis = hc[i]; Yaxis = hc[j]; Zaxis = hc[k]User specifies 3 hidden cells as vectorLine[j] = exemplar[j]hdrfile was foundFile created:
Oct 23,1995 at 10 : 5Data read from file = allReturned data dimension = {75,  8,  3}Number of exemplars = 8Number of hidden cells = 3Line id = endPoint size = 0.001Selected Cells = {1,  2,  3}

The plot entitled Hidden Cell Response shows the joint response of the 3 hidden cells (h1,h2,h3) from their initial random starting position.  Each dot represents an epoch, with the last dot replaced by the number of the exemplar to which the sequence belongs.  This 3D plot provides four items of information, i.e., three hidden weights and epoch. The four exemplars of T are identified at the end of each individual plot by the numbers 1 to 4 , and the exemplars of C by 5 to 8.  After about 20 epochs the paths of the T exemplars move in a direction opposite to those of C, while the C exemplars are approaching near convergence. Hidden cells 1 and 3 provide the most marked discrimination among the exemplars. With such a plot the migration of the hidden cell responses to each exemplar can be observed in terms of their paths and rate of convergence over the epoch sequence. By changing the network’s user options,  e.g., the learning rate parameter at the end of a set of epochs, one can observe the effect on the migration paths of each exemplar providing an effective method for evaluating modifications to the BPNN algorithm.

Plotting the Hidden Cell Weights

The data set of the hidden cell weights is of order (75,3,9) containing matrices of order (3,9) for each of the 75 epochs.  The (3,9) matrix holds the weights applied to each of the 9 attributes by each of the 3 hidden cells.  Using the calling function hdwAnlz to obtain the data file hdwfile and to rearrange the levels of the hidden cell weights matrix, the plotting function splt3d is used to plot the weights assigned by the network to each attribute.

hdwAnlz

Three Hidden Cell Weights for each Input cellXaxis = hc[i] wt;  Yaxis = hc[j] wt;  Zaxis = hc[k] wtUser specifies 3 hidden cells indices as vectorLine[n] = inputcell[n]hidwfile was foundFile created:
Oct 23,1995 at 10 : 5

The number appearing at the end of each individual plot identifies each of the nine attributes of the 3x3 character matrix.  Each coordinate point (h1,h2,h3) plotted represents the three weights assigned by the three hidden cells to an attribute at each epoch.  

The plot for input cell 5 "takes off" in its own direction ending with the largest of all the weights. Examination of the 3x3 character matrix reveals that attribute 5 is the center element, and that if the center element is shaded the character is a T, otherwise it is a C regardless of the other attributes.  Thus, the network has "learned" that the single most critical attribute required for classification is the center element.  Further examination of the plot indicates that weights for attributes 3, 9, 1, and 7 move in an opposite direction to the weights of the center element.  These elements are the corner elements of the 3x3 character matrix.  Thus, the network has identified the corner cells and assigned them considerably smaller weights than those for the center element.  Finally, attributes 2, 4, 6, and 8 representing elements between the corner cells are identified but change little from their random starting weights indicating their relative unimportance to the classification process. 

   If the number of hidden cells is not equal to 3, then the 3-dimensional plot can be used only by selecting subsets of three hidden cells.  A solution to this problem is to plot the weights of the hidden cells as a single plot for each attribute using MultipleListPlot (available in the Package Graphics`MultipleListPlot`), and then to display each plot in its corresponding position in the 3x3 character matrix using GraphicsArray.   If MultipleListPlot is used in its normal manner when there are a large number of data points as is characteristic with network data, then the plotting symbols overlap so much that the line is essentially made up of the plotting symbol itself.  Furthermore, when the number of hidden cells is large, it is difficult to distinguish the lines based on dashing options. 

   The function hdwAray sets up the data, calls mplotLists to do the plotting, and allows the user to specify the number of plots to be placed in each row of the array configuration as required by araydis.  In addition the function hdwAray, allows the user the option of setting the plot range for the y-axis so that plots show  consistency  in the scaling of the y-axis.

In the function mplotLists[m_,lim_], the data to be plotted is in m_ and the limit of the plot used in the PlotRange option of MultipleListPlot is in  lim_.  In mplotLists the vector colr provides up to 12 different RGB plotting colors by cycling through left shifts for each line plotted and a redefinition of $LineStyles. The call to MultipleListPlot is made following the examples outlined by Wickham-Jones [Wickham-Jones 1994] so that individual lists do not have to be specified.  The plotting symbols are removed from the plot object (pobj) by a Map[Delete[#,indx]&,{pobj}] operation in which  indx identifies the parts of the graphic structure to be deleted.

The function araydis[m_,in_ ] restructures the graphic objects in m_ into a new structure in which the number of plots per row of the displayed array is given by the vector in_.

hdwAray[d___]:=
 Module[{data,extnt,nhc,nic,p,prng,ans,lim,apat,ok},
 Print["Hidden Weights,Plots=inc[j]"];
 Print["Yaxis=wt; Xaxis=epoch; Line=hc[j]"];
 Print["Point=wt at each epoch"];
 If[Length[{d}]==0,
 {data,extnt}=getFdata;,data=d];
 nhc=Part[Dimensions[data],2];
 nic=Part[Dimensions[data],3];
 Print["Number Hidden cells=",nhc];
 Print["Number of input cells=",nic];
 Print["Minimum weight=",Min[data]];
 Print["Maximim weight=",Max[data]];
 p=List[];
 ans=kbinstrg["Set plot range? y/n",{{"y"},{"n"}}];
 If[ans=="y",
   prng=kbin3["Type {min,max} plot range",{2}];,
   prng=All];
 Do[ (* over number of input cells *)
 AppendTo[p,mplotLists[Transpose[
       Transpose[data,{3,2,1}][[i]]],prng]];           
 ,{i,1,nic}]; (*end Do *)
 
  ok=0;
  While[ok==0,
   apat=Input["Type vector of # plots in each row of array"];
   If[Apply[Plus,apat]>Length[p],
     Print["Error: too many plots in vector"];,
     ok=1]; (*end if *)
  ]; (*end while *)
 (*now display plots according to pattern in apat*)
 Show[GraphicsArray[
 araydis[Flatten[p],apat]],
      DisplayFunction->$DisplayFunction];                   
 ]; (*end Module *)

Executing hdwAray provides the following results.

hdwAray[]

Hidden Weights, Plots = inc[j]Yaxis = wt;  Xaxis = epoch;  Line = hc[j]Point = wt at each epochhidwfile was foundFile created:
Oct 23,1995 at 10 : 5Data read from file = allReturned data dimension = {75,  3,  9}Number Hidden cells = 3Number of input cells = 9Minimum weight = -2.366320029725546Maximim weight = 2.34162624108719

The array plot shows the network's identification of the importance of the center element, i.e., the plot in position (2,2),  and the similarity in size and form of the weights for the corner elements, i.e., the plots in positions  (1,1), (1,3), (3,1), and (3,3),  and between-corner elements as the simulation progresses.    Clearly,  the 3-dimensional and array plots provide much more insight into the behavior of the BPNN in case of Freeman's example than that provided by the plot of the squared error.

Discussion

Using Mathematica's rich collection of graphical procedures can enhance one's understanding of the dynamic and final state characteristics of the BPNN beyond that available from a simple plot of the squared error function. Because the data produced by the BPNN can be very extensive, graphic objects require some modification such as removal of the plotting symbol.  Because the network configuration used here had relatively few cells some of the problems encountered when a very large number of cells is used are not illustrated.  For example, in problems which require the network to identify characters through Morse code, or the diagnosis of cardiac conditions, 20 to 30 hidden cells might be used. In such problems the user would require the option to select the combination of three cells for a 3-dimensional plot.  Furthermore, because the attributes of the exemplars do not always lend themselves to a meaningful spatial arrangement such as the 3x3 character matrix illustrated here, the display of plots in a graphics array may not always be meaningful.  A set of five Notebooks and a utility Package containing the functions for the BPNN and plotting functions, which include density plots and plots resulting from singular value decomposition, is in preparation for MathSource.

References

Elman, J.L., "Learning and development in neural networks: the importance of starting small", Cognition, 48, (1993), 71-99. 

Freeman, J., A. Simulating Neural Networks with Mathematica, Addison-Wesley Publishing Co., Menlo Park (1994)

Wickham-Jones, T. Mathematica Graphics, TELOS/Springer-Verlag, Santa Clara (1994).

Istvan, S.N., Berkely, M.R.W., Dawson, D.A., Medler, D.P., Schopflocher, Hornsby, L. "Density Plots of Hidden Value Unit Activations Reveal Interpretable Bands", Connection Science, 7,2, (1995).

About the Authors

Steve Hunka is

Mike Carbonaro is

Electronic Subscriptions

Included in the distribution for each electronic subscription are the files BackProp.nb and BPNNAppendix.nb containing Mathematica code for the material described in this article.