Neural: Back Propagated Neural Network 
Notebook Version 2.0
University of Alberta, steve.hunka@ualberta.ca
Last Mod: Oct 22/97; changes to text

General Description

This Notebook contains the functions required to run a simple back propagated neural network and documentation for its use. No bias or momentum parameters are used. During execution queries are made of the user to provide parameters to tailor the network to the task, and for the selection of optional features such as plots and file storage of the results. The function bpnn, which does the simulation, is designed to run in cycles of epochs specified by the user. (An iteration is the presentation of one exemplar; an epoch one presentation of the set of exemplars; a cycle the number of epochs requested by the user.) At the end of each cycle, the user must examine the results to see whether a new cycle of epochs is required. Data saved to files can be analyzed further using Notebooks netAn1, netAn2, netAn3, and netAn4. Random number "seeds" can also be saved. The term "seed" refers to the result of $RandomState and not a user defined seed as would be used in SeedRandom[n].

Reference: Freeman, J.A., Simulating Neural Networks with Mathematica , Addison-Wesley Publishing Co., Menlo Park, 1994.

Required Packages

<< "netUtil`"

General Structure of a Neural Network

A general neural network structure with 3 input cells, 2 hidden cells and 1 output cell is illustrated in the diagram shown below:

In the back propagated neural network in this Notebook, the user may specify the number of input cells, hidden cells, and output cells, as well as "learning rate" parameters, exemplar order, and  exemplar span.

Summary of Interactive User Input

1) Run ID e.g., Ab102 to identify the current run
2) Number of input cells
3) Number of hidden cells
4) Number of output cells
5) Exemplar order of presentation:  0 natural order; 1 random order
6) Exemplar span (ordered set of exemplars to use, eg., span=3 uses
    exemplars 1,2,3; span=5 uses exemplars 1,2,3,4,5)
7) Learning rate parameter 1 (applied to output cells wts) and 2 (applied to
    hidden cell wts); can be scalar or vector. 
8) Random number seed: 0 to use new seed; 1 to use old seed if available as
    global variable rstX from last run in current session or retrieved from a file;
    option to save the random number seed to a file.     
9) Number of epochs
10) Maximum amount of output cell error data to save in RAM by epoch
11) Maximum amount of output cell response data to save in RAM by epoch
12) Option to save epoch data to files; if yes, then the user is asked in
      order whether a file is to be set up for sqd-error, hidden wts, hidden
      response, output weights, and output response, i.e., the user 
      can select specific data to be saved. For each set of data to be saved
      the user is asked for a file name; if the file already exists the user is 
      asked whether to continue (y) or not (n); if the user continues the 
      existing file is over written and if the response is not to continue, 
      the user is asked for a new file name.
13) Option to save random seed to a random seed file; the queries are
      as described in item (12) above, except that if the filename exists,
      continuing will append to the file rather than rewriting the file. This
      feature allows the user to save the seeds from many different simulations
      and to recreate exactly the starting point of any simulation.
14) vector of range for random number generation of weights if no
      external weights appear in the argument list of the main function
      (bpnn). User query to display starting weights. (See Note (j) below.)
15) Option to replicate exemplars (y/n) (increased presentation of specific
      exemplars.) Exemplar token frequencies, e.g., {1,1,1,3,1} indicates of 5
      exemplars the 4th exemplar should be presented 3 times each time it is
      selected. The number of elements in the vector must equal the current
      exemplar span.  

At this point the simulation starts. At the end of the current number of
epochs, a matrix having exemplars as rows, and columns as output cells
and associated target adjacent to each other, is displayed for the user 
to decide whether convergence has been reached. 

16) Option to plot rms error (y/n) one line for each output cell in a 
      single plot; lines are colored and numbered; leading color
      legend. X-axis is epoch
17) After the current number of epochs has been reached, the user is asked
      to enter the number of additional epochs (or 0 to end the simulation).
      If a new value for the number of epochs is entered, the user is
      queried for setting a new exemplar span (see 6 ) and learning parameters
      (see 7 ). 
      
When the simulation is ended by specifying 0 more epochs, data is output
and the user is queried for the following:

18)  Option to save the seed and results of last epoch only (responses and weights of the 
       hidden and output cells). This option is available because sometimes a user
       identifies an interesting run only after the run has been completed and did not anticipate
       saving the random seed and epoch data files. The
       queries here are the same as the file queries in item 12 above.     
19) Option to plot the response for each output cell for each exemplar over
      epochs. A single plot will contain as many colored lines as there are
      exemplars.
20) Option to plot for each output cell a 3D plot of exemplars by epochs
      by response. There is one plot for each output cell, so a large amount
      of front-end RAM could be used if there are many output cells. 
  

Notes to input parameters:

a) Some checks of input parameter integrity are made. User input is
    checked for an acceptable structure, e.g., if input is to be a vector the
    input is checked for vector form and number of elements. 
     
b) If the variable (exwts) optional external weights is not in the calling
    statement of the main function (bpnn), starting weights are generated
    randomly; if included it is checked to make sure the order is conformable to
    the number of input, hidden, and output cells.  If the order is not
    conformable, a message is given and computation stops.

c) When a vector is used to define the learning parameter, setting the jth
    element in learning parameter 1 to 0 prevents updating each output cell
    weight associated with the jth hidden cell because the adjustment is
    multiplied by 0. Similarly, setting the kth element in learning parameter 2 to
    0 prevents updating each hidden cell weight associated with the kth input
    cell.  If 0 is used for some elements of parameters 1 and 2 at the start of the
    run, and if the random wts are near zero, then corresponding hidden cell
    output and/or exemplar attributes are close to being ignored. 

	d) If random number reseed=0 (item 8 above) a new seed is always created; if
	    it is equal to 1 a previous seed is used if one exists. The current seed is
	    the Integer rstXwhich is the result of $#RandomState. rstX is GLOBAL. A printed
	     message "RNGV reset" or "RNGV created" is displayed depending on the condition
	      encountered. 
	    Once rstX has been created it can be saved to a file for future use. (See Tech Report
	    "Major New Features in Mathematica  v2.2" 1993 p16 for $RandomState
	     description.)
	
	e) This version of bpnn allows the option to save one or more random seeds
	    in a file.The file holding the random seeds holds the following information:
	     
    {{RunID,time-date stamp,{randomseed}},{RunID,time-date stamp,
         {randomseed}},......}
          
    and recovering a seed is done using the function rcvrExpr and
    requires knowing the RunID and time-date stamp which is output during a
    run of bpnn. RunID is a string. If you specify an existing file name for
    storing the random seed, a new seed is appended to those already in the
    file.

f) The maximum size of the data held during the calculation cycle of the
    network is determined by the query at (10) and (11). Once this maximum
    has been reached, with each additional epoch the oldest data item  is discarded.
    This is done to prevent exceedingly large numbers of epochs of system rms error 
    and network output data using up all of RAM. At each epoch rms error is saved  
    for each output cell, and network output for each output cell is saved for each
    exemplar. Thus, the number of output cells and exemplars determines how
    much RAM memory will be used during a run. Plotting of
    data by epoch at the end of the simulation only uses the maximum
    amount of data saved in RAM. Data saved to a file is for all epochs.
  
g) When the input parameter "exemplar order" is 0 (item 5 above), each
    exemplar is presented once in its natural order in one epoch. When this
     input parameter is 1, a random index is used to select an exemplar and
     although the long range expectation is that exemplars will be presented
     with equal frequency when token frequencies are set to 1, variations 
     will exist. A frequency vector of exemplar presentations is given at the
     end of each epoch cycle. If you use the plotting routines (e.g., Notebook
     netAn1) you must select fixed order of presentation since the
     functions assume a natural ordering of the exemplars.  When the
     exemplar span is less than the number of exemplars, data for the
     non-presented exemplars are padded with zero values to maintain a
     consistent data structure for purposes of plotting.
 
h) When the selected number of epochs have been reached the user is prompted
    by "How many more epochs?"; if the user enters an integer n, then n more
    epochs are carried out. The data saved in RAM at each epoch is still subject
    to the maximum specified in items 10 and 11.
   
i) An estimate of the simulation run time can be obtained from the vector
    printed following Start of simulation and End of simulation.  The 6 
    element vector is {yr,month, day,hour,min,sec}.
    
j)  If plots showing the dynamic characteristics of the network behavior are to be
    used, there is merit in assigning the random number range to a narrow band, e.g.,
    {-0.1,+0.1}.  The narrow band will allow the user to more easily observe how
    responses and weights "migrate" to their final values.

Data Input Structure

1) The exemplar matrix is of the form 
   {{{attributes of exemplar 1},{attributes of exemplar 2},...}},{{answer for
    output cell 1 for all exemplars},{answer for output cell 2 for all
    exemplars}...}}}
   Part 1 is of Dimensions[] (# of exemplars, # of input cells)
   Part 2 is of Dimensions[] (# of output cells, # of exemplars)
   
2) optional external weights of the form (j=#input cells, k=#hidden cells):
   {{{h11,h21,...hj1},{h12,h22,...hj2},...}},
   {{o11,o21,....ok1},{o12,o22,...ok2},...}}}
   Part 1 is of  Dimensions[] (# of hidden cells, # number of input cells)
   Part 2 is of Dimensions[] (# output cells, # of hidden cells)
   The results returned by the neural network function bpnn is in the form
   described here.

Executing the Simulation

The simulation is started by the statement:
  bpnn[data] when no external weights for the hidden and output cells are
  used, and
  bpnn[data,wts] if weights are used. The structure of data is defined in
  Data Input Structure given above. bpnn will return a data structure of
  weights which can be used to continue a run, eg., use
  runwts=bpnn[data]  (no external starting weights for initial run), then
  bpnn[data,runwts]   (use external weights called "runwts" for next run)
  
  The user should make a note of the Run ID and the date time stamp
  displayed at the beginning of the execution as these data are required
  in the case data saved to a file are to be recovered.

Results Displayed

1) Input parameters entered by the user are displayed, e.g., run ID, date-time stamp, number of input, hidden, and output cells, number of epochs, files set up, source of random seed.
2) At the end of a cycle of epochs, the output response to each exemplar adjacent to the target answer is displayed as well as the RMS error at each output cell. The frequency of exemplar presentation is also displayed.
3) Upon no further epochs being requested, the hidden cell weight matrix, the hidden cell response matrix, and ouput cell weights are displayed. 
4) The user may optionally ask for a plot of the RMS error, a plot of the output response for each cell over epochs, and a 3D plot of exemplar by epoch by response for each output cell.

Structure of Files Written

The user is queried at 3 points to save data once the bpnn function has been started: 
a) during the parameter input phase the user is queried as to which of the following data files are to be saved: squared-error, hidden weights, hidden response,output weights, and output response.
b) also during the input phase, the user is queried as to whether the random seed should be saved.
c) after the simulation has been terminated by specifying 0 more epochs, the user is given the option to save in one file, the random seed and for the last epoch only, the hidden weights, hidden response, output weights, and output response.  This option allows the user a "last chance" to save basic data of the simulation deemed of interest but not necessarily anticipated prior to starting the simulation.

File structure for squared-error, hidden weights, hidden responses, output weights, and output responses.
The following abbreviations are used:
# = number of
ic[[i]]     ith input cell               hc[[j]]     jth hidden cell
oc[[k]]    kth output cell            ex[[n]]     nth exemplar

Error File: 
            {oc[[1]], oc[[2]],.....oc[[k]]} repeated for each ITERATION. 
The order is (#iterations, #output cells)
            
Hidden Cell Responses File:
     {
       {hc[[1]], hc[[2]],...hc[[j]]},  for exemplar 1
       {hc[[1]], hc[[2]],...hc[[j]]},  for exemplar 2
         .................
       {hc[[1]], hc[[2]],...hc[[j]]},   for exemplar n
     } 
   repeated for each epoch. The order is (#epochs, #exemplars, #hidden cells).

Hidden Cell Weights File
    {
      {ic[[1]], ic[[2]],....ic[[i]]}, for hidden cell 1
      {ic[[1]], ic[[2]],....ic[[i]]}, for hidden cell 2
      ...............
      {ic[[1]], ic[[2]],....ic[[i]]}   for hidden cell j
    }   
    repeated for each epoch. The order is (#epochs, #hidden cells, #input cells).
      
Output Cell Weights File
  {
     {hc[[1]], hc[[2]],.....hc[[j]]}, output cell 1
     {hc[[1]], hc[[2]],.....hc[[j]]}, output cell 2
      ...........
       {hc[[1]], hc[[2]],.....hc[[j]]} output cell k
  }
   repeated for each epoch. The order is (#epochs,#output cells, #hidden cells).
    
Output Cell Response File
 {
   {ex[[1]], ex[[2]], ...,ex[[n]]},  output cell 1
   {ex[[1]], ex[[2]], ...,ex[[n]]},  output cell 2
   ..........
   {ex[[1]], ex[[2]], ...,ex[[n]]},  output cell k
 } 
 repeated for each epoch. The order is (#epochs, #output cells, #exemplars).
 
 Analysis of the dynamic characteristics of the simulation require data in epoch units. Notebook netAn1 uses hidden and output cell responses at each epochs,  and netAn2 uses hidden and output weights at each epoch for plotting. 

Random Seed File
The format of the expression(s) stored in the random seed file are as follows:
{runID, date-time stamp,{random seed}}. The date-time-stamp is made up of an integer formed by the digits of the year, month, day, hour, minute, and second. It is displayed at the start of the simulation. If the user stores many random seeds in the same file, the function rprtFdata can be used to obtain a list of the runID and date-time stamps in the file, and the function rcvrExpr can be used to retrieve them.

Note: If the last simulation has been run using external weights from a previous session, the global variable rstX containing a seed for generating random numbers may or may not exist. (If a run using a random seed was made during the current session, rstX will exist, but may not be the seed from which the external weights were generated as the latter may have been recovered from a file created at a different session. In this case saving rstX will not provide a regeneration of the simulation.)

Last Epoch Data File
The structure of this file is as follows:
{
{runIDsed, date-time stamp,{random seed}},
{runIDerr, date-time stamp,{squared error at each output cell}},
{runIDhwt, date-time stamp,{hidden cell weights}},
{runIDhdr, date-time stamp,{hidden cell responses}},
{runIDowt, date-time stamp,{output cell weights}},
{runIDrsp, date-time stamp,{output cell responses}}
}
where runID is given by the user during the parameter input phase. The function rcvrExpr[] may be used to recover any of these items. For example, rcvrExpr["ABCsed",19970318220020]  would recover the random seed for the runID "ABC" and date-time stamp 19970318220020. Note that 3 additional characters are appended to the runID and must be included when using rcvrExpr to get the appropriate expression from the file. The user named file is opened for appending so results of several simulations may be accumulated in the same file. If this is not desired, in function bpnn change the statement iofname=openFna[filelist,2] to iofname=openFna[filelist,1].  

The function rprtFdata can be used to obtain a list of the runID and date-time stamps of expressions in the file. It is important for the user to keep accurate records of file names, Run IDs, and date time stamps of the simulation runs in order to be able to recover the appropriate data.

Description of Variable Names in the function bpnn

ans        general variable holding current user response
c           response to each exemplar; structure is 
            {{{outcel1},{outcel2},....}},{{outcel1},{outcel2}}....} in which
            each {outcel} has as many elements as exemplars.
            {{outcel1},{outcel2}} appears once for  each iteration saved.
cflat      Flatten[c]; used for plotting
dtyp      vector holding 3 character expression identifier
err        error at output cell layer
excntr   exemplar presentation frequencies
filelist   descriptive name of files to hold data
hc         number of hidden cells
hdel      hidden cell layer delta; adjustment
hdout    hidden cell layer output
hdrsp    hidden cell layer response
hedr      holds runID and 3 characters from dtyp 
hwt       hidden cell weights
i            index counter
inc        number of input cells
inx        index to select exemplars
ioerfsw  switch to write error-sqd file; 0=no write; 1=write
iofname  name of files opened; returned by openFna
iofsw     file output switch; 0=none;1= one or more to be setup
iterinc    number of additional epochs
itr          epochs for current cycle
itrcopy   copy of itr argument
labrc      holds row/column titles for display of matrices
lpar1      "learning parameter" applied to adjustment of output weights
lpar2      "learning parameter" applied to adjustment of hidden weights
maxepoc maximum amount of output data to save in RAM by epoch
maxts     maximum amount of rms error data to save in RAM by epoch
namerf    holds file name for error-sqd data
odel        output layer adjustment
ord         exemplar order 0/1
otc         number of output cells
output    output at output layer
owt        weights used at output layer
padhdr    holds padding (0) to bring hdrsp to full exemplar size
padzero   holds padding (0) to bring xresp to full exemplar size
rmse       root mean sqd error at end of an epoch
rndt        run date-time stamp; 14 digit integer; yr,m,d,h,m,s
rsin2      holds runidc and 3 char description of data in file when seed and
              last epoch data is saved
runidc     run ID code input by user; string
sqderr     accumulates rmse error current cycle;{{outcel1,outcel2...},....}
totitr       total number of epochs
ttl           holds titles for plots
x            exemplar matrix
xfreq      values used in this list are used as the iterator to repeat the 
             presentation of an exemplar as given by xseq
xresp      holds the system response for each output cell at a given iteration; it
              is padded with 0 if the exemplar span is less than the number of
              exemplars
xseq        holds index order for presentation of exemplars
xsp         exemplar span; ordered number of exemplars to use
y            exemplar correct response or target response
ysp         y for current range of exemplars

Functions Called

The Main function (bpnn) calls the following functions directly or 
indirectly. Functions contained in this Notebook are as follows:
                 
matmerg:   to interlace cols of output and desired response (target) so they 
                can be seen side by side
getxfrc:    presents option to input a token frequency vector
logistic:    to carryout logistic transformation
getLpar:   to get learning parameters as scalar or vector
usrsed:     to establish a random number seed, and generate hidden and output
               weights randomly
wrsdFile:  to write a random number seed to a file
clr2:         a color function which determines the color of a 3D surface plotted
               using RGB; currently commented out in bpnn and Hue used instead.

The following functions are used by bpnn and contained in the Package netUtil with their documentation in the Notebook netUdoc:
kbin3:        to input numeric parameter values by user
kbinstrg:    to input alphanumeric answer by user
plotnLists: to plot sqd-error and output for each output cell
plotLegend: to plot a legend for colored lines used in plotnLists
pmat2:      displays a matrix m in column widths of pc units; 2nd parameter can
                be changed so that for small screens no matrix wrap-around
                appears.
genrch:     used to generate row and column headings for pmat2
openFna:  to open the files user wants for storage of results
timeDate:   used to get the year,month,day,hr,min, sec
rndoff:     roundoff values in m to d decimal digits

The following are useful functions not called by bpnn which are contained in the Package netUtil. Their documentation documentation is in the Notebook netUdoc.
rcvrExpr:  used to recover an expression from a random seed file, or last
                epoch data file written by an earlier run of bpnn in which
                by option, the expression was saved.
rprtFdata:  can be used to obtain a summary of data stored in the
                random seed file or last epoch data file.
saveExpr:  saves an expression to a file
                
Other Functions:
tcdspla:     used to generate density plot of character matrix of example
                problem

Example Problem

The data set tcL is a variation of Freeman's T and C problem (see Simulating Neural Networks with Mathematica , Freeman, J.A., Addison Wesley, p77) in which the characters T and C are rotated clockwise successively by 90 degrees to form 8 exemplars. The network is to distinguish the characters regardless of their orientation. In the example used here, the character L has been added to the exemplars to make the problem a little more difficult.  See the section "The T, C, and L pattern recognition problem" following the listing of the functions. The example is run using 9 input cells, 5 hidden cells, and 3 output cells. This example data is in the cell "Input Data (exemplars and answers (target)".

bpnn[xy,exwts] is the back propagated neural network function.
xy  holds the structure containing the encoded exemplar matrix and answer (target) matrix. (See Data Input Structure above).
exwts is optional; if used it contains a structure holding the hidden layer cell weights  (#hidden cells, #input cells) and the output layer cell weights  (#output cells, #hidden cells). When exwts is in the argument list, random number weights are not generated. Normally, exwts would be used when an earlier simulation is to be continued.
The function bpnn returns a structure containing the hidden cell weights of order (#hidden cells, #input cells), and the output cell weights of order (#output cells, #hidden cells) which may be used as the variable exwts in further runs, for purposes of further analysis, or for assessing the network's behavior with exemplars not used in the training set of exemplars. 

Calls: See Functions Called (above)
Returns: {hidden cell wts, output cell wts}
Required: netUtil Package

Note: In the 3D plot of output layer responses, the statement ColorFunction->(clr2[#]&),ColorOutput->RGBColor has been commented out and replaced with ColorOutput->Hue. If the user wishes to specify the color separation using RGB color and the function clr2, then comment markers can be removed, and ColorOutput->Hue commented out. 

Mod. Date: Apr 29/97

\(\(bpnn[xy_, exwts___] := 
      Module[{ans, hwt, owt, itrcpy, totitr, x, y, sqderr, c, err, cflat, 
          iterincr, inx, hdout, output, xseq, xsp, ysp, rnr, padzero, odel, 
          hdel, ttl, hdrsp, excntr, xfreq, lpar1, lpar2, itr, inc, hc, otc, 
          rs, maxts, maxepoc, ord, xresp, padhdr, iofsw, iofname, filelist, 
          ioerfsw, namerf, rmse, runidc, rndt, rsin, fileseed, fios, rsin2, 
          labrc, i, dtyp, hedr}, Print["BPN v1.1 Mod date: Apr 29/97"]; 

        		 (*check\ the\ structure\ of\ xy\ before\ going\ on*) 
		
        If[Length[xy] == 0, Print["Data matrix not evaluated"]; 
          Return[]]; runidc = InputString["Type a run ID code"]; 
        Print["Run ID code=", runidc]; 
        rndt = Plus@@\((Date[]\ Reverse[Table[10\^\(x\ 2\), {x, 0, 5}]])\); 
        Print["Run date-time stamp=", rndt]; ioerfsw = 0; 
        inc = kbin3["How many input cells?", 0]; 
        Print["Input cells=", inc]; 
        If[Length[Transpose[xy[[1]]]] 
            ≠ inc, 
          Print["Error:input cells NE data input cols"]; Return[]]; 
        hc = kbin3["How many hidden cells?", 0]; 
        Print["Hidden cells=", hc]; 
        otc = kbin3["How many output cells?", 0]; 
        Print["Output cells=", otc]; 
        If[Length[xy[[2]]] ≠ 
            otc, Print["Error:# response vectors NE # outcells"]; 
          Return[]]; 
        ord = kbin3["Exemplar order fixed/random? (0/1)", 0]; 
        If[ord == 0, Print["Exemplar order=fixed"], 
          Print["Exemplar order=random"]]; 
        xsp = kbin3["Exemplar span?", 0]; 
        Print["Starting exemplar span=", xsp]; {lpar1, lpar2} = 
          getLpar[otc, hc, inc]; itr = kbin3["How many epochs?", 0]; 
        Print["Starting epochs=", itr]; 
        maxts = kbin3["Maximum error data to save in RAM?", 0]; 
        Print["Max error data to save in RAM=", maxts]; 
        maxepoc = kbin3["Maximum response data to save in RAM?", 0]; 
        Print["Max response data to save in RAM=", maxepoc]; 
        ans = 
          kbinstrg["Epoch data to files? y/n", {{"y"}, {"n"}}]; 
        Print["Epoch data to files=", ans]; 

        		 (*ask\ user\ whether\ each\ of\ the\ following\ files\ is

              					to\ be\ set\ up\ if\ ans == y*) 
		
        If[ans == "y", iofsw = 1; 
          filelist = {{1, {"Sqd error"}}, {2, {"Hidden weights"}}, {
                3, {"Hidden response"}}, {4, {"Output weights"}}, {
                5, {"Output response"}}}; iofname = openFna[filelist, 1]; 
          \(If[\(iofname[[1
                  ]]\)[[1
                ]] == 1, ioerfsw = 1; 
            namerf = 
              \(iofname[[1
                  ]]\)[[2
                ]]]; \), iofsw = 0]; 

        		 (*check\ for\ external\ weights\ or\ generate\ random\ wts*) 

        		If[Length[{exwts}] ≠ 0, 
          \(If[Dimensions[
                  exwts[[1]]] == {hc, 
                  inc} && 
              Dimensions[exwts[[2]]] == {
                  otc, hc}, 
            hwt = exwts[[1]]; 
            owt = exwts[[2]]; 
            \(Print["External starting weights used"]; \), 
            Print["External wts matrix order error"]; Return[]]; 
          \), {hwt, owt} = usrsed[inc, hc, otc]; wrsdFile[runidc, rndt]]; 
        If["y" == 
            kbinstrg[
              "Display starting weights? y/n", {{"y"}, {"n"}}], 
          Print[""]; Print["Starting Hidden Cell Weights(i,h)"]; 
          pmat2[Transpose[hwt], 4, 
            genrch["I", "HidC", Reverse[Dimensions[hwt]]]]; 
          Print[""]; Print["Starting Output Cell Weights (h,o)"]; 
          pmat2[Transpose[owt], 4, 
            genrch["H", "OutC", Reverse[Dimensions[owt]]]], 
          Print["First hidden wt=", Take[Flatten[hwt], 1]]]; 
        itrcpy = itr; totitr = 0; 

        		 (*now\ partition\ xy\ to\ x\ and\ y*) 
		
        x = xy[[1]]; 
        y = Transpose[xy[[2]]]; 
        sqderr = {}; c = {}; iterincr = 1; 
        excntr = 
          Table[0, {Length[xy[[1]]]}]; 
        While[iterincr ≠ 0, 
          xseq = Range[xsp];  (*fixed\ exemplar\ order*) \ 
          padzero = Table[Table[0, {otc}], {Length[x] - Length[xseq]}]; 
          xfreq = getxfrc[xseq]; Print["Exemplar replications:", xfreq]; 
          Print["Starting Simulation at:", Date[]]; 
          Do[ (*over\ epochs; \ pass\ exemplars\ thru\ the\ cells*) 
				
            If[ord == 1, xseq = Table[Random[Integer, {1, xsp}], {xsp}]]; 
            Do[inx = xseq[[1]];  (*
                Do\ over\ exemplar\ range*) 
                Do[ (*over\ exemplar\ frequency*) 
                  excntr[[inx]] = 
                  excntr[[inx]] + 1;  (*
                  frequency\ counter*) hdout = 
                  logistic[
                    hwt . x[[inx]]]; 
                output = logistic[owt . hdout]; 
                err = y[[inx]] - output; 
                odel = err\ \((output\ \((1 - output)\))\); 
                hdel = \((hdout\ \((1 - hdout)\))\)\ Transpose[owt] . odel; 
                owt += lpar1\ Outer[Times, odel, hdout]; 
                hwt += lpar2\ 
                    Outer[Times, hdel, 
                      x[[inx]]]; 
                \(If[ioerfsw == 1, Write[namerf, err\ err]]; 
                \), {xfreq[[inx]]}];  (*
                end\ Do\ over\ exemplar\ frequency*) \(xseq = 
                RotateLeft[xseq, 1]; 
              \) (*next\ exemplar\ index*) , {Length[xseq]}];  (*
              end\ Do\ over\ exemplar\ range*) \ 

            				 (*\(calc\ hidden\ &\)\ output\ layer\ response*) 

            				hdrsp = 
              logistic[
                hwt . Transpose[
                    x[[Range[xsp]]]]]; 
            xresp = logistic[owt . hdrsp];  (*output\ layer\ response*) 
              rmse = Transpose[xresp] - 
                y[[Range[xsp]]]; 
            rmse = Plus@@\((rmse\ rmse)\)\/xsp; 
            sqderr = Append[sqderr, rmse]; 

            				 (*
              now\ check\ if\ data\ saved\ in\ RAM\ needs\ dropping*) 

            				If[Length[sqderr] > maxts, sqderr = Drop[sqderr, 1]]; 
            xresp = Transpose[Join[Transpose[xresp], padzero]]; 
            c = Append[c, xresp]; 
            padhdr = Table[Table[0, {hc}], {Length[x] - Length[xseq]}]; 
            hdrsp = Join[Transpose[hdrsp], padhdr]; 
            If[Length[c] > maxepoc, c = Drop[c, 1]]; 

            				 (*write\ output\ data\ by\ epoch, \ 
              except\ err\ which\ was\ done\ by\ iteration*) 
				
            \(If[iofsw == 1, 
              \(Do[ (*over\ io\ units*) \(Which[
                  \(iofname[[j
                        ]]\)[[1
                      ]] == 2, 
                  \(Write[
                    \(iofname[[j
                        ]]\)[[2
                      ]], hwt]; \), 
                  \(iofname[[j
                        ]]\)[[1
                      ]] == 3, 
                  \(Write[
                    \(iofname[[j
                        ]]\)[[2
                      ]], hdrsp]; \), 
                  \(iofname[[j
                        ]]\)[[1
                      ]] == 4, 
                  \(Write[
                    \(iofname[[j
                        ]]\)[[2
                      ]], owt]; \), 
                  \(iofname[[j
                        ]]\)[[1
                      ]] == 5, 
                  \(Write[
                    \(iofname[[j
                        ]]\)[[2
                      ]], xresp]; \)]; 
                \), {j, 1, Length[iofname]}]; \)]; \), {itrcpy}];  (*
            end\ epoch\ Do\ loop*) Print[
            "Exemplar Presentation Frequency="]; 

          			 (*print\ estimated\ response\ for\ all\ exemplars*) 
			
          Print[excntr]; Print[""]; 
          Print["Last Response to each Exemplar"]; 

          		 (*get\ appropriate\ y\ elements\ and\ pad\ if\ necessary*) 	

          			ysp = y[[Range[xsp]]]; 
          ysp = Join[ysp, 
              Table[Table[0, {otc}], {Length[x] - Length[xseq]}]]; 

          		 (*generate\ row/col\ headings\ for\ output/target\ 
              comparison*) 	
			
          labrc = genrch["Out", "Trgt", {otc, otc}]; 
          labrc = Flatten[
              matmerg[{labrc[[1]]}, {
                  labrc[[2]]}]]; 
          labrc = Join[{
                Flatten[
                  Drop[genrch["Ex", "a", {Length[x], 1}], \(-1\)]]}, {
                labrc}]; 
          pmat2[rndoff[matmerg[Transpose[Flatten[Take[c, \(-1\)], 1]], ysp], 
              3], 6, labrc]; totitr += itrcpy; 
          Print["Total epochs=", totitr]; 
          Print["Last epoch RMS error=", rmse]; 
          Print["Mean RMS error=", Plus@@rmse\/otc]; 
          If["y" == 
              kbinstrg[
                "Display Rms error plot? y/n", {{"y"}, {"n"}}], 
            plotLegend[otc]; 
            ttl = StringInsert["Rms error at epoch ", ToString[totitr], 
                \(-1\)]; plotnLists[sqderr, ttl, "epoch", 0, 1, 1]]; 
          iterincr = kbin3["How many more epochs?", 0]; 
          If[iterincr ≠ 0, Print[""]; 
            Print["Next cycle epochs=", iterincr]; 
            xsp = kbin3["Exemplar span?", 0]; 
            Print["Next cycle exemplar span=", xsp]; {lpar1, lpar2} = 
              getLpar[otc, hc, inc]; ]; 
          Which[iterincr ≠ 0, itrcpy = iterincr; Continue[], 
            iterincr == 0, Break[]]; ]; 

        		 (*Now\ output\ system\ results*) 
		
        Print["End of simulation at:", Date[]]; 
        Print["
Hidden Cell weights (i,h)"]; 
        pmat2[rndoff[Transpose[hwt], 3], 5, 
          genrch["I", "HidC", Reverse[Dimensions[hwt]]]]; 
        hdrsp = logistic[
            x[[Range[xsp]]] . 
              Transpose[hwt]]; Print["
Hidden Cell response (ex,h)"]; 
        pmat2[rndoff[hdrsp, 3], 5, 
          genrch["Ex", "HidC", Dimensions[hdrsp]]]; 
        Print["
Output Cell weights (h,o)"]; 
        pmat2[rndoff[Transpose[owt], 3], 5, 
          genrch["H", "OutC", Reverse[Dimensions[owt]]]]; 

        		 (*close\ all\ files\ opened\ earlier*) 
		
        If[iofsw == 1, 
          \(Do[Close[
              \(iofname[[j
                  ]]\)[[2
                ]]]; 
            \(Print["File closed: ", 
              \(iofname[[j
                  ]]\)[[2
                ]]]; \), {j, 1, Length[iofname]}]; \)]; 

        		 (*
          allow\ saving\ one\ file\ with\ seed\ and\ last\ epoch\ data\ 
            only*) 
		filelist = {{1, {"Seed & Last epoch data"}}}; 
        iofname = openFna[filelist, 2]; 
        If[Length[iofname] ≠ 0, Print["opened ", iofname]; 
          iofname = 
            \(iofname[[1
                ]]\)[[2
              ]]; 
          dtyp = {"sed", "err", "hwt", "hdr", "owt", 
              "rsp"}; 
          hedr = MapThread[
              Join[{StringInsert[runidc, #1, \(-1\)]}, {rndt}]&, {dtyp}]; 
          i = 0; MapThread[
            Write[iofname, 
                Append[hedr
                    [[i = 
                      i + 1]], #1]]&, {{{rstX}, err, hwt, 
                hdrsp, owt, xresp}}]; Close[iofname]; 
          Print["Closed file ", iofname]; ]; 

        		 (*end\ if\ iofname != 0\ *) 
		cflat = Flatten[c]; 
        If["y" == 
            kbinstrg[
              "Plot response at each output? y/n", {{"y"}, {
                  "n"}}], plotLegend[xsp]; 

          			 (*Do\ over\ the\ number\ of\ output\ cells*) 
			
          Do[ttl = "Response at Output cell "<>ToString[i]; 
            \(plotnLists[
              Partition[Drop[cflat, \((i - 1)\)\ Length[x]], Length[x], 
                otc\ Length[x]], ttl, "epochs", 0, 1, 1]; 
            \), {i, 1, otc}]; ]; 

        		 (*Plot\ 3  d\ output\ for\ each\ output\ cell; 
          exemplar\ by\ epoch\ by\ output\ value*) 
		
        If["y" == 
            kbinstrg[
              "3D plots for each output? y/n", {{"y"}, {"n"}}], 
          \(Do[ttl = "Response at Output cell "<>ToString[i]; 
            \(ListPlot3D[
              Partition[Drop[cflat, \((i - 1)\)\ Length[x]], Length[x], 
                otc\ Length[x]], 

              		 (*ColorFunction :> 
                  \((clr2[#]&, ColorOutput -> RGBColor, \)*) 			

              					ColorFunction -> Hue, Mesh -> False, 
              AxesLabel 
                -> {"exemplars", "epochs", "response"}, 
              PlotLabel -> ttl]; \), {i, 1, otc}]; \)];  (*
          end\ if\ y == *) 
		Return[{hwt, owt}]; ]; \)\)

usrsed[ic,hc,oc] generates the random hidden and output cell weights.
ic: number of input cells; hc: number of hidden cells; oc: number output cells
User Input: 
a) whether a new random seed is used (0) or a previously used seed (1) in rstX is to be used.
b) a 2 element vector of the range of random number wts 
Note: If rstX is not available but (1) in (a) is selected, a message is given and the user is asked for a 2 element vector of the range for generation of random number wts. rstX is a GLOBAL variable.
Return: {hidden weights, output weights)
Mod. Date: Mar 15/97

\(\(usrsed[ic_, hc_, oc_] := 
      Module[{rs, rnr, hw, ow}, 
        rs = kbin3["Random seed new/rstX? (0/1)", 0]; 
        Print["Random seed source=", rs]; 

        		 (*generate\ new\ random\ weights\ if\ rs == 0*) 
		
        If[rs == 0, rstX = $RandomState; Print["Random seed: new"], 

          	 (*else\ rs == 1, \ determine\ random\ state\ vector\ status; \ 
            if\ rstX\ is\ a\ list\ as\ it\ would\ be\ if\ recovered\ from\ a\ 
              file\ by\ rcvrExpr, \ 
            it\ is\ brought\ to\ the\ appropriate\ Integer\ form\ by\ Part\ 
              operation*) 		
			
          \(If[True == ValueQ[rstX], 
            If[List === Head[rstX], 
              rstX = rstX[[1
                  ]]]; $RandomState = rstX; 
            Print["Random seed: last reset"], 
            Print["No previous random seed found; new seed used"]; 
            rstX = $RandomState; Print["Random seed: new"]]; \)]; 
        rnr = kbin3["Type vector random number range", {2}]; 
        Print["Random number range=", rnr]; 
        hw = Table[Table[Random[Real, rnr], {ic}], {hc}]; 
        ow = Table[Table[Random[Real, rnr], {hc}], {oc}]; 
        Return[{hw, ow}]; ]; \)\)

wrsdFile[rid,rdt] writes a current random seed (the global variable rstX) to a file at user option. The file is opened for appending.
rid:  run ID
rdt:  run date-time stamp
User Input: See openFna in netUtil
Calls: openFna
Mod. Date: Mar 15/97

\(\(wrsdFile[rid_, rdt_] := 
      Module[{fileseed, fios, rsin}, 
        
 (*user\ option\ to\ write\ random\ seed\ to\ a\ file*) 		
		
        fileseed = {{1, {"Random Seed"}}}; fios = openFna[fileseed, 2]; 
        If[Length[fios] ≠ 0,  (*file\ opened*) \ 
          Print["Opened ", fios]; rsin = Join[{rid}, {rdt}]; 
          Write[\(fios[[1
                ]]\)[[2
              ]], Append[rsin, {rstX}]]; 
          Close[\(fios[[1
                ]]\)[[2
              ]]]]; ]; \)\)

getLpar[noc,nhc,nic]; gets learning parameters 1 and 2 as scalar or vector; 
noc: is number of output cells, 
nhc: number of hidden cells, 
nic: number of input cells.
If vector form selected, learning parameter 1 is replicated to order (noc,nhc) and learning parameter 2 replicated to order (nhc,nic). If a single value (scalar) is selected, then {lpar1,lpar2} is returned. A scalar can be used for one of the parameters and a vector for the other.
Calls: kbin3.
Returns: {lpar} in the form {lpar1,lpar2} when scalars are used and  {{lpar1},{lpar2}} when vectors are used. When a scalar is used for parameter 1 and vector for parameter 2 is used, {lpar1,{lpar2}} where lpar2 is a matrix is returned.
Mod. Date: Jan 31/97

getLpar[noc_, nhc_, nic_] := 
      Module[{p, q, d1, d2, d3, d4, ans, lpar}, p = {noc, nhc, nic}; 
        q = "Learning parameter  :scalar (0) or vector (1)?"; 
        d1 = "Type learning parameter   as scalar with decimal"; 
        d2 = "Type learning parameter   as   element vector"; 
        d4 = "Learning Parameter  = "; lpar = {}; 
        Do[q1 = StringInsert[q, ToString[j], 20]; 
          If[0 == kbin3[q1, 0], 
            ans = kbin3[StringInsert[d1, ToString[j], -24], 0.1]; 
            Print[StringInsert[d4, ToString[j], 20], ans]; , 
            d3 = StringInsert[d2, ToString[j], 25]; 
            d3 = StringInsert[d3, 
                ToString[p[[j + 1]]], 
                -16]; 
            ans = kbin3[
                d3, {p[[j + 1]]}]; 
            Print[StringInsert[d4, ToString[j], -4], ans]; 
            ans = Table[
                ans, {p[[j]]}]; ]; 
          AppendTo[lpar, ans]; , {j, 1, 2}]; Return[lpar]; ];

getxfrc[xs] gets exemplar presentation frequency vector for the sequence of exemplars given by the vector xs.
Calls: kbinstrg, kbin3
Returns: frequency vector
Mod. Date: Mar 21/96

getxfrc[xs_] := 
      Module[{ans, pv}, pv = Table[1, {Length[xs]}]; ans = "y"; 
        While[ans == "y", 
          ans = kbinstrg[
              "Replicate exemplars? y/n", {{"y"}, {"n"}}]; 
          If[ans == "y", 
            pv = kbin3["Type exemplar frequency vector", {Length[xs]}]]; 
          Return[pv]; ]; ];

logistic[p] returns p transformed by logistic function
Mod. Date: Mar 21/96

logistic[p_] := (1/(1 + (E^(-p))));

clr2[y] is the color function to color the 3D plot of exemplars by epochs by output; y is the network output. red if y<=.39; blue if y>=.61; else green. Note these cut points were set because y was either .1 or .9. More cut points would be required when y takes on more than 2 values. This function is called by ListPlot3D[m_] in bpnn but is currently commented out in place of using Hue. 
Mod. Date: Mar 21/96

clr2[y_] := 
      Which[y ≤  .39, RGBColor[1, 0, 0], y ≥  .61, 
        RGBColor[0, 0, 1], y < 1.0, RGBColor[0, 1, 0]];

matmerg[m1,m2] merges columns of matrices m1 and m2 placing m1[j] and m2[j] in adjacent columns. Assumes m1 and m2 of the same order. If m1 and m2 are vectors, then matmerg[{m1},{m2}] will merge their elements. 
Mod. Date: Mar 21/96

matmerg[m1_, m2_] := 
      Module[{m3, indx, c}, 
        If[Dimensions[m1] ≠ Dimensions[m2], 
          Print["matmerg:matrix dimensions not same"]; Return[]]; 
        m3 = Join[Transpose[m1], Transpose[m2]]; 
        c = Dimensions[m3][[1]]; 
        indx = Flatten[Transpose[Partition[Range[c], {(c/2)}]]]; 
        m3 = Transpose[m3[[indx]]]; 
        Return[m3]; ];

The T, C, and L pattern recognition problem

tcdspla[d] creates 12 density graphics representing character matrices of order 3 by 3 as used in the TCL problem. d is the input data (see example below).
Date: Feb 3/97

tcdspla[d_] := 
      Module[{pobj}, 
        pobj = {MapThread[
              ListDensityPlot[#1, FrameTicks -> None, 
                  Frame -> False, 
                  ColorFunction -> 
                    (If[#1 ≤ 0.1, GrayLevel[0.4], 
                          GrayLevel[1.0]]&), 
                  DisplayFunction -> Identity]&, {
                MapThread[Partition[##1, 3]&, {d}]}]}; 
        Show[GraphicsArray[Partition[Flatten[pobj], 4]], 
          DisplayFunction -> $DisplayFunction]];

Distinguishing T,C, and L; Character matrix (not data input matrix)

tcr2 = {{ .9,  .1,  .9,  .9,  .1,  .9,  .1,  .1,  .1}, { .9,  .9,  .1, 
           .1,  .1,  .1,  .9,  .9,  .1}, { .1,  .1,  .1,  .9,  .1,  .9,  .9, 
           .1,  .9}, { .1,  .9,  .9,  .1,  .1,  .1,  .1,  .9,  .9}, { .1, 
           .1,  .1,  .1,  .9,  .9,  .1,  .1,  .1}, { .1,  .9,  .1,  .1,  .9, 
           .1,  .1,  .1,  .1}, { .1,  .1,  .1,  .9,  .9,  .1,  .1,  .1, 
           .1}, { .1,  .1,  .1,  .1,  .9,  .1,  .1,  .9,  .1}, { .1,  .1, 
           .1,  .1,  .9,  .9,  .1,  .9,  .9}, { .1,  .9,  .9,  .1,  .9,  .9, 
           .1,  .1,  .1}, { .9,  .9,  .1,  .9,  .9,  .1,  .1,  .1, 
           .1}, { .1,  .1,  .1,  .9,  .9,  .1,  .9,  .9,  .1}}

The problem for the network is to correctly identify the characters T, C, and L, in 4 different orientations, as illustrated below.

tcdspla[tcr2]

Example User Input for TCL problem

The network is to distinguish T, C, and L regardless of the orientation. Each 3x3 matrix is considered an exemplar to be input to the network. Shaded areas are encoded in the input matrix  as .9 and unshaded areas as .1.  The target or answer vector representing the correct answer for each output cell is encoded .9 and .1. For example, for output cell 1, .9 represents T and .1 "not T". (See Input Data cell below.)

Execution of the T, C, L problem shown below was defined using the following user input options:
1) A runID is given as TCL953
2) Number of input cells: 9
3) Number of hidden cells: 5
4) Number of output cells:  3
5) Exemplar order: 0 (fixed)
6) Exemplar span: 12
7) Learning parameter 1: scalar (0); 1.0
8) Learning parameter 2: scalar (0); 1.0
9) Random seed: 0 (generated)
10) Starting epochs: 300

11) Max. error data to save in RAM: 300
12) Max. response data to save in RAM: 300
13) Epoch data to file: y (yes)
      At this point the user is asked whether each file (sqd error, hidden wts,hidden responses,
      output wts, output respons) are to be set up. If the reply is y (yes) a file name is requested
      for each file. In the example run which follows the following files were set up:
      TCLe2 for sqd. error, TCLhw for hidden weights, TCLhr for hidden responses, TCLow for
      output weights, TCLor for output cell responses.

13) Random number range: {-0.1,0.1}
14) write random seed to file: y  (yes); file name requested of user
15) Exemplar replication: n (no)
16) Display starting wts: n (no)

At this point the simulation is running for 300 epochs. At the end of 300 epochs a display is made of the output cell responses together with the target desired, and the RMS error vector.  
17) Display RMS error plot: y  (yes)
18) How many more epochs: 0
19) Save seed and last epoch data: y (yes); file name requested of user was given as TCLast
20) Plot response at each output: y (yes)
21) 3D plots: y (yes)

Note: The user should watch for error messages which may be displayed during the input of parameters. Functions kbin3 will give an error message if the numeric input is not of the appropriate structure and kbinstrg will give an error message when the string input is not an acceptable string.  In both cases, the user is prompted again for input.

Input Data (exemplar and answer (target)).

tcL = {{{ .9,  .9,  .9,  .1,  .9,  .1,  .1,  .9,  .1}, { .1,  .1, 
             .9,  .9,  .9,  .9,  .1,  .1,  .9}, { .1,  .9,  .1,  .1,  .9, 
             .1,  .9,  .9,  .9}, { .9,  .1,  .1,  .9,  .9,  .9,  .9,  .1, 
             .1}, { .9,  .9,  .9,  .9,  .1,  .1,  .9,  .9,  .9}, { .9,  .9, 
             .9,  .9,  .1,  .9,  .9,  .1,  .9}, { .9,  .9,  .9,  .1,  .1, 
             .9,  .9,  .9,  .9}, { .9,  .1,  .9,  .9,  .1,  .9,  .9,  .9, 
             .9}, { .9,  .1,  .1,  .9,  .1,  .1,  .9,  .9,  .9}, { .9,  .9, 
             .9,  .9,  .1,  .1,  .9,  .1,  .1}, { .9,  .9,  .9,  .1,  .1, 
             .9,  .1,  .1,  .9}, { .1,  .1,  .9,  .1,  .1,  .9,  .9,  .9, 
             .9}}, {{ .9,  .9,  .9,  .9,  .1,  .1,  .1,  .1,  .1,  .1,  .1, 
             .1}, { .1,  .1,  .1,  .1,  .9,  .9,  .9,  .9,  .1,  .1,  .1, 
             .1}, { .1,  .1,  .1,  .1,  .1,  .1,  .1,  .1,  .9,  .9,  .9, 
             .9}}};

bpnn[tcL]

(*The rms error plot shows that error was quickly reduced for the character T (black line), while the reduction of error for L (green line) took considerably more epochs. Reduction to the same error level was reached for the 3 output cells representing T,C, and L in 300 epochs (3600 iterations). The 2D line plot for output cell 3 indicates that the network had difficulty separating C and L in about the first 175 epochs.
In the 3D plots above the first shows the system's discrimination for each of the  4 orientations of the character T, the second for C, and the third for L over the 300 epochs*)

Examples using rprtFdata, rcvrExpr, and saveExpr

These functions are contained and more fully documented in the Package netUdoc.
rprtFdata:  no arguments; reports on contents of file in which each entry
                is of the form {stringID,date-time-stamp,{expression}}. User
                is asked to enter a file name.
rcvrExpr[stringID,date-time-stamp]: recovers the expression with id "stringID" 
                and date-time-stamp. User is asked to enter a file name. The form
                rcvrExpr[] will recover the last expression in the file.
saveExpr[x]: saves the expression x; user is asked for a string ID for the
                expression, and name of file; the date-time-stamp is generated
                automatically. Uses write append. The format of the expression stored
                is {stringID,date-time-stamp,{expression}}.

In this example the run ID was TCL953, and data of the last epoch only was saved in the file named TCLast. The function rprtFdata reports the Mathematica expressions in the file. The use of the function rcvrExpr is illustrated by recovery of the result of $RandomState which is assigned to the Global variable rstX. If the random seed option in running bpnn is given as 1, the seed variable rstX will be used to generate random weights rather than a new seed.  In this manner a run can be replicated.

<< "netUtil`"

rprtFdata

\(\(rcvrExpr["TCL953sed", 19971022214210]
 (*
      recover\ result\ of\ $RandomState\ used\ in\ simulation*) \)\)

rstX

Length[IntegerDigits[rstX]]

\(\(saveExpr[{rstX}]
 (*
      the\ result\ of\ $RandomState\ is\ the\ global\ variable\ rstX\ which\ 
        is\ a\ long\ integer\ having\ 558\ digits . \ It\ is\ saved\ in\ file
        \ TCLrsi*) \)\)