readMe: Description of Neural Network Notebooks 
Notebook Version: 2.0
Steve Hunka, University of Alberta
steve.hunka@ualberta.ca
Mod. Date: Oct 31/97 V3.0 revision

There are 9 Notebooks in this collection. Two Notebooks provide documentation, and seven form a collection of functions for running a simple back propagated neural network and analyzing its dynamic and final state characteristics. These Notebooks are as follows:

1.  Neural.nb; holds the back propagated network functions
2.  netCls.nb; holds functions for classifying exemplars (usually not in the training set) and provides bar charts representing the output from the hidden layer cells and output layer cells.
3.  netAn1.nb; graphical representations of the error sqd at each output cell, and responses from the hidden and output layer cells. Uses data from each iteration for the error plot, and from each epoch for the plot of responses.
4.  netAn2.nb; graphical representations of weights at the hidden and output layer cells. Uses data from each epoch for plotting.
5.  netAn3.nb; graphical representations of hidden and output cell responses using bar plots (as in netCls.nb) and density plots showing input attributes together with hidden cell and output cell data at last epoch.
6.  netAn4.nb; provides functions for carrying out a singular value decomposition of hidden cell response data as well as 2D and 3D plots. Last epoch data only are used.
7.  netUtil.nb;  is a Package containing the most commonly used functions for the  6 Notebooks listed above. The user must do a save of this notebook to produce the Package netUtil.m required for running with V3.0.

The Notebook netUdoc.nb contains documentation for the functions in netUtil.nb.
                      
Each Notebook contains documentation for those functions in the Notebook.  Neural.nb also contains some test data referred to as the "TCL" problem.  The TCL problem is a modification of the example problem J.A. Freeman illustrates in his textbook Simulating Neural Networks with Mathematica , which requires the network to discriminate between the characters T and C in different orientations, and to which the character L has been added.

Conversion of Notebooks from V2.2 to V3.0

In converting from Version 2.2 to Version 3.0 of Mathematica the following changes occurred or were explicitly made:
1)  Comments were removed by the conversion process. Some, but not all comments were replaced.
2) FileNames[fname} now returns file names prepended with a colon. Functions rcvrExpr, rprtFdata and getFdata in netUtil were modifided to accommodate this change.
3)  The plotting function ScatterPlot3D in V3.0 returns a directive GrayLevel[0] not in V2.2 resulting in incorrect indexes in modification of the graphic object by function splt3d in Notebooks netAn1 and netAn2. Indexing was modified for V3.0.
4) If the result of $RandomState saved by option in running the simulation function bpnn in Notebook Neural is done using V2.2 it cannot be used in V3.0. Thus,  a simulation run in V2.2 cannot be recreated in V3.0, and for this reason the numerical values obtained by using V3.0 are different than those using V2.0 although acceptable convergence was obtained in both cases.(A preferred method may be to ask the user for a seed n, saving this value, and then use SeedRandom[n] to recreate a simulation.) 
5)  The function mplotLists in Notebook netAn2 in V2.2 removed the plotting symbol by manipulating the graphic object produced by MultipleListPlot. The option SymbolShape->None is now used.

Development of the Notebooks

All the functions were developed using Mathematica version 2.2 on a Macintosh 6100/60 and 7200/120, and then revised for Version 3.0.1.1x on a Macintosh 7200/120.  The larger sized runs of the Neural.nb functions have used 1144 input cells, 12 hidden cells, and 18 output cells. 

Using the Notebooks

The Package netUtil.nb when used with Version 3.0 must be first saved by the user to produce netUtil.m so that its functions can be loaded in the Notebooks.  Some Notebooks require additional Packages supplied by Mathematica, and in this case these are documented in each Notebook. Normally, one would use Neural.nb first, and then depending upon the type of analysis wanted of either the dynamic or final state characteristics of the network, use the other Notebooks. 

The Notebook netUtil.nb, although written as a Mathematica Package, is not in the programming style of Packages supplied by Mathematica since the user is not normally expected to call the functions directly, except in the case of the functions rcvrExpr, rprtFdata, and saveExpr. Since almost all functions are written for interactive user input of parameters, checking of arguments is made before the functions in netUtil are called. netUdoc.nb contains documentation of the functions in netUtil.nb. 

Displayed Results

Results of computations displayed to the user's screen may be more extensive than usually expected.  This has been done to provide the user with additional information confirming that, for example, calculations appear reasonable, and that the appropriate data file is being used. 

Files Written

User options are provided for writing a number of data sets to a file during the running of Neural.nb : (a) saving of the result of $RandomState, (b) saving the squared error at each iteration, (c) saving the responses and weights at the hidden and output layer at each epoch, and (d) saving the random seed and last epoch data. When data is extensive, it is much faster to obtain last epoch data from (d) than from (c).  Functions in Notebook netAn1 require error and epoch data, i.e., items (b and c). Functions in Notebook netAn2 requires item (c).  Functions in netAn3 and netAn4 require data of the last epoch. Writing to files as in (c) adds to the run time of the simulation. (Hidden and output cell weights for the last epoch are also returned by the function bpnn in Neural.nb). The user may wish to set a specific file path for files written.

Keeping Accurate Records

Experience suggests that users must keep accurate records of the files they create if data are to be recovered reliably at some future date.  Keeping a log of the runID, date time stamps, and associated file names is important together with information about the simulation such as the nature of the exemplars, number of input, hidden, and output cells. (runID is a user input in the function bpnn in Neural.nb, but the date time stamp is generated automatically and displayed.) If the user saves a number of Mathematica expressions, such as $RandomState for several runs of the simulator, to a file (using the function saveExpr), the function rprtFdata can be used to provide a summary of the file contents. 


Cautions

Checking Parameter Integrity: The interactive keyboard input functions check for the structure of the input, but there are many combinations of options that a user might attempt to use.  It is likely all possible combinations have not been tested.  Obvious illogical input errors are not checked, e.g., 0 input cells, 0 hidden cells, and 0 output cells.

Exemplar Presentation Order:  The graphical analysis Notebooks assume that a fixed order (not random order) of exemplars has been used in Neural.nb. 

Amount of Data Generated:  The amount of data generated and which could be saved in files by Neural.nb could be very extensive depending upon the number of iterations, epochs, and number of cells used. For example, having thousands of input cells, run for 200 or 300 epochs with just a few exemplars could generate a hidden cell weight file of 20MB or more.  Any graphical analysis of the dynamic characteristics of the network requiring plotting from such a large file, would require extensive amounts of RAM allocated to the Mathematica front end and kernel. Generally, the problem is not as severe when only the final state data of the network is used, however, being congnizant of the number of plots which may result is important.