(*********************************************************************** Mathematica-Compatible Notebook This notebook can be used on any computer system with Mathematica 3.0, MathReader 3.0, or any compatible application. The data for the notebook starts with the line of stars above. To get the notebook into a Mathematica-compatible application, do one of the following: * Save the data starting with the line of stars above into a file with a name ending in .nb, then open the file inside the application; * Copy the data starting with the line of stars above to the clipboard, then use the Paste menu command inside the application. Data for notebooks contains only printable 7-bit ASCII and can be sent directly in email or through ftp in text mode. Newlines can be CR, LF or CRLF (Unix, Macintosh or MS-DOS style). NOTE: If you modify the data for this notebook not in a Mathematica- compatible application, you must delete the line below containing the word CacheID, otherwise Mathematica-compatible applications may try to use invalid cache data. For more information on notebooks and Mathematica-compatible applications, contact Wolfram Research: web: http://www.wolfram.com email: info@wolfram.com phone: +1-217-398-0700 (U.S.) Notebook reader applications are available free of charge from Wolfram Research. ***********************************************************************) (*CacheID: 232*) (*NotebookFileLineBreakTest NotebookFileLineBreakTest*) (*NotebookOptionsPosition[ 1313446, 42178]*) (*NotebookOutlinePosition[ 1314983, 42226]*) (* CellTagsIndexPosition[ 1314939, 42222]*) (*WindowFrame->Normal*) Notebook[{ Cell[TextData[{ StyleBox["Neural: Back Propagated Neural Network ", Evaluatable->False, AspectRatioFixed->True, FontWeight->"Bold"], StyleBox[ "\nNotebook Version 2.0\nUniversity of Alberta, steve.hunka@ualberta.ca\n\ Last Mod: Oct 22/97; changes to text", Evaluatable->False, AspectRatioFixed->True] }], "Text", Evaluatable->False, AspectRatioFixed->True], Cell[CellGroupData[{ Cell[TextData[{ StyleBox["General Description", Evaluatable->False, AspectRatioFixed->True, FontWeight->"Bold"], StyleBox["", Evaluatable->False, AspectRatioFixed->True] }], "Text", Evaluatable->False, AspectRatioFixed->True], Cell[TextData[{ StyleBox[ "This Notebook contains the functions required to run a simple back \ propagated neural network and documentation for its use. No bias or momentum \ parameters are used. During execution queries are made of the user to provide \ parameters to tailor the network to the task, and for the selection of \ optional features such as plots and file storage of the results. The function \ bpnn, which does the simulation, is designed to run in cycles of epochs \ specified by the user. (An iteration is the presentation of one exemplar; an \ epoch one presentation of the set of exemplars; a cycle the number of epochs \ requested by the user.) At the end of each cycle, the user must examine the \ results to see whether a new cycle of epochs is required. Data saved to files \ can be analyzed further using Notebooks netAn1, netAn2, netAn3, and netAn4. \ Random number \"seeds\" can also be saved. The term \"seed\" refers to the \ result of $RandomState and not a user defined seed as would be used in \ SeedRandom[n].\n\nReference: Freeman, J.A., Simulating Neural Networks with ", Evaluatable->False, AspectRatioFixed->True], StyleBox["Mathematica", Evaluatable->False, AspectRatioFixed->True, FontSlant->"Italic"], StyleBox[" , Addison-Wesley Publishing Co., Menlo Park, 1994.", Evaluatable->False, AspectRatioFixed->True] }], "Text", Evaluatable->False, AspectRatioFixed->True] }, Open ]], Cell[CellGroupData[{ Cell[TextData["Required Packages"], "Subsubsection", Evaluatable->False, AspectRatioFixed->True], Cell[BoxData[ \(<< "\"\)], "Input", AspectRatioFixed->True] }, Closed]], Cell[CellGroupData[{ Cell[TextData["General Structure of a Neural Network"], "Subsubsection", Evaluatable->False, AspectRatioFixed->True], Cell[TextData[{ StyleBox[ "A general neural network structure with 3 input cells, 2 hidden cells and \ 1 output cell is illustrated in the diagram shown below:", Evaluatable->False, AspectRatioFixed->True, FontWeight->"Plain"], StyleBox["\n", Evaluatable->False, AspectRatioFixed->True] }], "Subsubsection", Evaluatable->False, AspectRatioFixed->True], Cell[GraphicsData["PostScript", "\<\ %! %%Creator: Mathematica %%AspectRatio: 1 MathPictureStart %% Graphics /Courier findfont 10 scalefont setfont % Scaling calculations 0.0238095 0.952381 0.0238095 0.952381 [ [ 0 0 0 0 ] [ 1 1 0 0 ] ] MathScale % Start of Graphics 1 setlinecap 1 setlinejoin newpath [ ] 0 setdash 0 g p P 0 0 m 1 0 L 1 1 L 0 1 L closepath clip newpath % Start of user PostScript /mathtops { gsave MBeginOrig moveto MEndOrig currentpoint grestore } bind def /MAtocoords { mathtops 4 2 roll mathtops 4 copy pop pop 3 -1 roll sub /arry exch def exch sub /arrx exch def arrx dup mul arry dup mul add sqrt /arrl exch def translate } bind def /MAarrowhead1 { gsave MAtocoords [ arrx arrl div arry arrl div -1 arry mul arrl div arrx arrl div 0 0 ] concat -0.05 0.0125 moveto 0 0 lineto -0.05 -0.0125 lineto fill -0.05 0.0125 moveto 0 0 lineto -0.05 -0.0125 lineto -0.05 0.0125 lineto stroke grestore } def % End of user PostScript p p p 1 0 0 r .004 w .1 .2 m .1 .2 .09524 0 365.73 arc F P p 1 0 0 r .004 w .5 .2 m .5 .2 .09524 0 365.73 arc F P p 1 0 0 r .004 w .9 .2 m .9 .2 .09524 0 365.73 arc F P P p p 0 1 0 r .004 w .25 .5 m .25 .5 .09524 0 365.73 arc F P p 0 1 0 r .004 w .75 .5 m .75 .5 .09524 0 365.73 arc F P P p 0 0 1 r .004 w .5 .75 m .5 .75 .09524 0 365.73 arc F P P p p .004 w .1 .17619 m .2619 .5 L s % Start of user PostScript 0.08 0.16 0.25 0.5 MAarrowhead1 % End of user PostScript P P p p .004 w .1 .17619 m .7381 .5 L s % Start of user PostScript 0.08 0.16 0.75 0.5 MAarrowhead1 % End of user PostScript P P p p .004 w .5 .17619 m .2619 .5 L s % Start of user PostScript 0.5 0.16 0.25 0.5 MAarrowhead1 % End of user PostScript P P p p .004 w .5 .17619 m .7381 .5 L s % Start of user PostScript 0.5 0.16 0.75 0.5 MAarrowhead1 % End of user PostScript P P p p .004 w .88095 .17619 m .7381 .5 L s % Start of user PostScript 0.9 0.16 0.75 0.5 MAarrowhead1 % End of user PostScript P P p p .004 w .88095 .17619 m .2619 .5 L s % Start of user PostScript 0.9 0.16 0.25 0.5 MAarrowhead1 % End of user PostScript P P p p .004 w .2619 .5 m .5 .7381 L s % Start of user PostScript 0.25 0.5 0.5 0.75 MAarrowhead1 % End of user PostScript P P p p .004 w .7381 .5 m .5 .7381 L s % Start of user PostScript 0.75 0.5 0.5 0.75 MAarrowhead1 % End of user PostScript P P p p .004 w .1 .03333 m .1 .11905 L s % Start of user PostScript 0.08 0.01 0.08 0.1 MAarrowhead1 % End of user PostScript P P p p .004 w .5 .03333 m .5 .11905 L s % Start of user PostScript 0.5 0.01 0.5 0.1 MAarrowhead1 % End of user PostScript P P p p .004 w .88095 .03333 m .88095 .11905 L s % Start of user PostScript 0.9 0.01 0.9 0.1 MAarrowhead1 % End of user PostScript P P p p .004 w .5 .7381 m .5 .92857 L s % Start of user PostScript 0.5 0.75 0.5 0.95 MAarrowhead1 % End of user PostScript P P p [(Input)] .30952 .17619 0 0 Mshowa P p [(Cells)] .69048 .17619 0 0 Mshowa P p [(Hidden Cells)] .5 .5 0 0 Mshowa P p [(Output Cell)] .78571 .78571 0 0 Mshowa P % End of Graphics MathPictureEnd \ \>"], "Graphics", Evaluatable->False, AspectRatioFixed->True, ImageSize->{282, 282}, ImageMargins->{{35, Inherited}, {Inherited, Inherited}}, ImageCache->GraphicsData["Bitmap", "\<\ CF5dJ6E]HGAYHf4PAg9QL6QYHgOolSO01>OolRO00OolIO0000`00O000000H O00=Ool01@00Oomoogoo00000goo0`0017oo00<007ooOol00goo00@007ooOomoo`@0019ooa1l0080 009l0003001l07`001il00=oo`0000=oog`0O0006W`000<007`0O0000W`00P004W`04Woo1@0057oo 00<007ooOol03Woo67`000D007`0O01l000001Ql00eoo`@000Yoo`03001oogoo00=oo`03001oogoo 01Moo`il008000Al0003001l07`001il00=oo`0000=oog`0O0006g`000<007`0O0000g`00P0047`0 :goo00<007ooOol03Woo5g`000<007`0O0000g`000<007`0O0005G`06Goo0`000goo0`006Goo37`0 0P001G`000<007`0O0007g`00goo00000gooO01l000KO0000`00O01l0005O002000>O00lOolGO000 0`00O01l0003O0000`00O01l000EO00kOol:O0020007O0000`00O01l000OO003Ool00003Ooml07`0 01al0003001l07`000Il008000al03]ooaMl0003001l07`000El0003001l07`001Al03]oo`Ul0003 001l07`000Ml0003001l07`001ml00=oo`0000=oog`0O00077`000<007`0O00027`00P002W`0>goo 5W`000<007`0O0001g`000<007`0O0004g`0>goo1g`00P002G`000<007`0O00087`00goo00000goo O01l000MO0000`00O01l0009O0020008O00kOolEO0000`00O01l0009O0000`00O01l000BO00kOol5 O002000;O0000`00O01l000PO003Ool00003Ooml07`001el0003001l07`000]l008000Il03]ooaEl 0003001l07`000Ul0003001l07`0019l03]oo`=l008000al0003001l07`0025l00=oo`0000=oog`0 O0007W`000<007`0O00037`00P0017`0?7oo4g`000<007`0O0002g`000<007`0O0004G`0>goo00=l 000000003W`000<007`0O0008G`00goo00000gooO01l000NO0000`00O01l000>O0020002O00lOolB O0000`00O01l000=O0000`00O01l000@O00jOol2000?O0000`00O01l000RO003Ool00003Ooml07`0 01ml0003001l07`000ml008003aooa9l0003001l07`000el0003001l07`000ml03Uoo`800003Ooml 07`000il0003001l07`0029l00=oo`000Woo87`000<007`0O0004G`00P00>Woo4G`000<007`0O000 3g`000<007`0O0003W`0=goo0P000goo47`000<007`0O0008W`00goo0002OolQO0000`00O01l000@ O002Ool2000hOol@O0000`00O01l000AO0000`00O01l000=O00eOol20005Ool?O0000`00O01l000S O003Ool0009oob5l0003001l07`000ml00Eoo`03001oogoo03Ioo`il0003001l07`001=l0003001l 07`000al03=oo`8000Qoo`il0003001l07`0029l00Aoo`000Woo8W`000<007`0O0003W`01Woo0P00 =Woo3W`000<007`0O0004g`000<007`0O00037`0O008Ool2000dOol=O0000`00O01l000EO0000`00O01l000;O00`Ool00`00 Oomoo`0:Ool=O0000`00O01l000SO004Ool000=oob9l0003001l07`000al00]oo`8003=oo`]l0003 001l07`001Ml0003001l07`000Ul02moo`8000ioo`]l0003001l07`002=l00Eoo`000goo8W`000<0 07`0O00037`03Goo0P00Goo00<007ooOol0Goo00@007oo000003Eoo`8000=oo`03001oogoo03Moo`03001o ogoo039oo`00Goo00@007ooOomoo`80035oo`8000Eoo`03001oogoo03Ioo`03 001oogoo03=oo`00Ool2000NOol2000COol00`00Oomoo`0`Ool00`00Oomoo`0eOol003Aoo`03001oogoo035oo`03 001oogoo015oo`8001Yoo`8001Ioo`03001oogoo02moo`03001oogoo03Eoo`00=7oo00<007ooOol0 7oo000hOol00`00Oomoo`0WOol00`00Oomoo`0TOol200000goo0000 000XOol00`00Oomoo`0UOol00`00Oomoo`0iOol003Qoo`03001oogoo02Moo`03001oogoo029oo`80 00Eoo`8002Ioo`03001oogoo02Eoo`03001oogoo03Uoo`00>Goo00<007ooOol09Goo00<007ooOol0 8Goo0P002Goo0P009Goo00<007ooOol08goo00<007ooOol0>Woo000iOol00`00Oomoo`0TOol00`00 Oomoo`0POol2000=Ool2000TOol00`00Oomoo`0ROol00`00Oomoo`0jOol003Yoo`03001oogoo029o o`03001oogoo01moo`80015oo`8002=oo`03001oogoo021oo`03001oogoo03]oo`00>Woo00<007oo Ool08Woo00<007ooOol07Goo0P005Goo0P008Goo00<007ooOol087oo00<007ooOol0>goo000kOol0 0`00Oomoo`0POol00`00Oomoo`0MOol00`00Oomoo`0GOol2000POol00`00Oomoo`0OOol00`00Oomo o`0kOol003]oo`03001oogoo01moo`03001oogoo01aoo`8001aoo`8001moo`03001oogoo01eoo`03 001oogoo03aoo`00?7oo00<007ooOol00Woo2@?P4goo00<007ooOol06Woo0P0087oo0P007Goo00<0 07ooOol047oo2@?P17oo00<007ooOol0?7oo000lOol00`00Oomoo`0A0n0Goo000fOol80n000`000n03h00E0n03Ool00`00Oomoo`0DOol2 000dOol2000GOol01000Oomoogoo5@?P00<000?P0n0020?P=goo000eOol:0n000`000n03h00E0n00 0goo001oo`0DOol2000hOol2000FOol00`00Ool3h00E0n000`000n03h0090n0fOol003Aoo`/3h003 0003h0?P01D3h003001oogoo019oo`8003aoo`8001Eoo`030003h0?P01@3h0030003h0?P00X3h3Eo o`000n000`000n03h00C0n0010000n03h0?P3goo00<007ooOol0@Woo0P00 3goo0`?P00<000?P0n004P?P00<000?P0n003@?P0n0cOol0031ooa43h003 0003h0?P0103h0030003h0?P00D3h0Uoo`8004]oo`03001oogoo00Qoo`L3h0030003h0?P00l3h003 0003h0?P00l3h39oo`00<7oo4P?P00<000?P0n000P003@?P00<000?P0n001@?P1goo0P00CWoo0P00 1goo20?P00<000?P0n002P?P0P000P?P00<000?P0n004@?P7oo000hOolL0n0Goo00<007ooOol0KGoo001_ Ool00`00Oomoo`0gOol00`00Oomoo`1^Ool0071oo`03001oogoo03Eoo`03001oogoo06moo`00LGoo 00<007ooOol001l7000701l7000>01meOol0 07Aoo`l07`H000L07`H000l07gEoo`00Lgoo4P0O1@001@0O1@004P0OM7oo001cOolC01l4000501l4 000C01mdOol0079ooaH07`<000<07`<001D07gAoo`00LWoo5`0O0`0000@07`00000001H07gAoo`00 LWoo6@0O00<0000O0000600OM7oo001bOolJ01l00`0001l07`0H01mcOol0079ooaX07`0300007`0O 01P07g=oo`00LGoo6`0O00<0000O01l0600OLgoo001aOolK01l00`0001l07`0H01mcOol0075ooa/0 7`0300007`0O01P07g=oo`00LGoo6`0O00<0000O01l0600OLgoo001aOolK01l00`0001l07`0H01mc Ool0079ooaX07`0300007`0O01P07g=oo`00LWoo6P0O00<0000O01l0600OLgoo001bOolJ01l00`00 01l07`0G01lWOol40019Ool0079ooaX07`0300007`0O01L07bQoo`03001oogoo04Uoo`00LWoo6P0O 00<0000O01l05`0O5Woo0`000goo1@000Woo0`000Woo10000Woo1@000Woo0`0027oo10000goo1000 00=oo`0000000`0000=oo`0000000`007Goo001cOolI01l00`0001l07`0G01lEOol01@00Oomoogoo 00000Woo00@007ooOol000=oo`06001oogoo001oo`000goo00<007oo00000Woo00D007ooOomoo`00 009oo`03001oogoo00Aoo`03001oogoo00Aoo`03001oogoo00Eoo`03001oogoo00=oo`03001oogoo 01eoo`00Lgoo6@0O00<0000O01l05`0O5Goo00D007ooOomoo`00009oo`04001oogoo0003Ool00`00 Oomoo`02Ool01`00Oomoogoo001oo`00009oo`05001oogooOol00009Ool00`00Oomoo`04Ool50003 Ool00`00Oomoo`03Ool00`00Oomoo`0MOol007AooaP07`0300007`0O01H07aIoo`05001oogooOol0 0002Ool01000Oomoo`000goo00<007ooOol00Woo00L007ooOomoo`00Ool00002Ool01@00Oomoogoo 00002Goo00<007ooOol017oo00D007ooOomoo`0000=oo`03001oogoo00=oo`03001oogoo01eoo`00 M7oo600O00<0000O01l05P0O5Woo00H007ooOomoo`00Ool200000goo00000002Ool400000goo0000 0003000017oo000007oo0P000Woo10001goo00D007ooOomoo`0000=oo`<000Aoo`03001oogoo00=o o`03001oogoo01eoo`00M7oo600O00<0000O01l05P0O5goo0`002Woo00<007ooOol03goo00<007oo Ool027oo10002Woo00<007ooOol00goo00<007ooOol07Goo001eOolG01l00`0001l07`0E01lUOol0 0`00Oomoo`0?Ool00`00Oomoo`0DOol30003Ool3000OOol007IooaH07`0300007`0O01@07gMoo`00 Mgoo5@0O00<0000O01l04`0ON7oo001gOolE01l00`0001l07`0C01mhOol007Qooa@07`0300007`0O 01807gUoo`00NGoo4`0O00<0000O01l04@0ONWoo001jOolB01l00`0001l07`0@01mkOol007]ooa40 7`0300007`0O00l07gaoo`00O7oo400O00<0000O01l03P0OOGoo001oOol=01l00`0001l07`0=01mn Ool0081oo``07`0300007`0O00/07h1oo`00PGoo2`0O00<0000O01l0200OPgoo0023Ool901l00`00 01l07`0601n5Ool008Uoo`<07`0300007`0O00<07hQoo`00S7oo00<007ooOol0Rgoo002"], ImageRangeCache->{{{0, 281}, {281, 0}} -> {-0.0250052, -0.0250052, 0.00373669, 0.00373669}}], Cell[TextData[{ StyleBox[ "In the back propagated neural network in this Notebook, the user may \ specify the number of input cells, hidden cells, and output cells, as well as \ \"learning rate\" parameters, exemplar order, and exemplar span.", Evaluatable->False, AspectRatioFixed->True, FontWeight->"Plain"], StyleBox[" ", Evaluatable->False, AspectRatioFixed->True] }], "Subsubsection", Evaluatable->False, AspectRatioFixed->True] }, Closed]], Cell[CellGroupData[{ Cell[TextData[{ StyleBox["Summary of Interactive User Input", Evaluatable->False, AspectRatioFixed->True, FontWeight->"Bold"], StyleBox["", Evaluatable->False, AspectRatioFixed->True] }], "Text", Evaluatable->False, AspectRatioFixed->True], Cell[TextData[{ StyleBox[ "\n1) Run ID e.g., Ab102 to identify the current run\n2) Number of input \ cells\n3) Number of hidden cells\n4) Number of output cells\n5) Exemplar \ order of presentation: 0 natural order; 1 random order\n6) Exemplar span \ (ordered set of exemplars to use, eg., span=3 uses\n exemplars 1,2,3; \ span=5 uses exemplars 1,2,3,4,5)\n7) Learning rate parameter 1 (applied to \ output cells wts) and 2 (applied to\n hidden cell wts); can be scalar or \ vector. \n8) Random number seed: 0 to use new seed; 1 to use old seed if \ available as\n global variable rstX from last run in current session or \ retrieved from a file;\n option to save the random number seed to a file. \ \n9) Number of epochs\n10) Maximum amount of output cell error data to \ save in RAM by epoch\n11) Maximum amount of output cell response data to save \ in RAM by epoch\n12) Option to save epoch data to files; if yes, then the \ user is asked in\n order whether a file is to be set up for sqd-error, \ hidden wts, hidden\n response, output weights, and output response, \ i.e., the user \n can select specific data to be saved. For each set of \ data to be saved\n the user is asked for a file name; if the file \ already exists the user is \n asked whether to continue (y) or not (n); \ if the user continues the \n existing file is over written and if the \ response is not to continue, \n the user is asked for a new file name.\n\ 13) Option to save random seed to a random seed file; the queries are\n \ as described in item (12) above, except that if the filename exists,\n \ continuing will append to the file rather than rewriting the file. This\n \ feature allows the user to save the seeds from many different simulations\n \ and to recreate exactly the starting point of any simulation.\n14) vector \ of range for random number generation of weights if no\n external \ weights appear in the argument list of the main function\n (bpnn). User \ query to display starting weights. (See Note (j) below.)\n15) Option to \ replicate exemplars (y/n) (increased presentation of specific\n \ exemplars.) Exemplar token frequencies, e.g., {1,1,1,3,1} indicates of 5\n \ exemplars the 4th exemplar should be presented 3 times each time it is\n \ selected. The number of elements in the vector must equal the current\n \ exemplar span. \n\nAt this point the simulation starts. At the end of the \ current number of\nepochs, a matrix having exemplars as rows, and columns as \ output cells\nand associated target adjacent to each other, is displayed for \ the user \nto decide whether convergence has been reached. \n\n16) Option to \ plot rms error (y/n) one line for each output cell in a \n single plot; \ lines are colored and numbered; leading color\n legend. X-axis is epoch\n\ 17) After the current number of epochs has been reached, the user is asked\n \ to enter the number of additional epochs (or 0 to end the simulation).\n \ If a new value for the number of epochs is entered, the user is\n \ queried for setting a new exemplar span (see 6 ) and learning parameters\n \ (see 7 ). \n \nWhen the simulation is ended by specifying 0 more \ epochs, data is output\nand the user is queried for the following:\n\n18) \ Option to save the seed and results of last epoch only (responses and weights \ of the \n hidden and output cells). This option is available because \ sometimes a user\n identifies an interesting run only after the run has \ been completed and did not anticipate\n saving the random seed and \ epoch data files. The\n queries here are the same as the file queries \ in item 12 above. \n19) Option to plot the response for each output cell \ for each exemplar over\n epochs. A single plot will contain as many \ colored lines as there are\n exemplars.\n20) Option to plot for each \ output cell a 3D plot of exemplars by epochs\n by response. There is one \ plot for each output cell, so a large amount\n of front-end RAM could be \ used if there are many output cells. \n \n\n", Evaluatable->False, AspectRatioFixed->True], StyleBox["Notes to input parameters:", Evaluatable->False, AspectRatioFixed->True, FontWeight->"Bold"], StyleBox[ "\n\na) Some checks of input parameter integrity are made. User input is\n \ checked for an acceptable structure, e.g., if input is to be a vector the\n \ input is checked for vector form and number of elements. \n \nb) If \ the variable (exwts) optional external weights is not in the calling\n \ statement of the main function (bpnn), starting weights are generated\n \ randomly; if included it is checked to make sure the order is conformable to\n\ the number of input, hidden, and output cells. If the order is not\n \ conformable, a message is given and computation stops.\n\nc) When a vector is \ used to define the learning parameter, setting the jth\n element in \ learning parameter 1 to 0 prevents updating each output cell\n weight \ associated with the jth hidden cell because the adjustment is\n multiplied \ by 0. Similarly, setting the kth element in learning parameter 2 to\n 0 \ prevents updating each hidden cell weight associated with the kth input\n \ cell. If 0 is used for some elements of parameters 1 and 2 at the start of \ the\n run, and if the random wts are near zero, then corresponding hidden \ cell\n output and/or exemplar attributes are close to being ignored. \n\n\t\ d) If random number reseed=0 (item 8 above) a new seed is always created; if\n\ \t it is equal to 1 a previous seed is used if one exists. The current \ seed is\n\t the Integer rstXwhich is the result of $#RandomState. rstX is \ GLOBAL. A printed\n\t message \"RNGV reset\" or \"RNGV created\" is \ displayed depending on the condition\n\t encountered. \n\t Once rstX \ has been created it can be saved to a file for future use. (See Tech Report\n\ \t \"Major New Features in ", Evaluatable->False, AspectRatioFixed->True], StyleBox["Mathematica", Evaluatable->False, AspectRatioFixed->True, FontSlant->"Italic"], StyleBox[ " v2.2\" 1993 p16 for $RandomState\n\t description.)\n\t\n\te) This \ version of bpnn allows the option to save one or more random seeds\n\t in \ a file.The file holding the random seeds holds the following information:\n\t \ \n {{RunID,time-date stamp,{randomseed}},{RunID,time-date stamp,\n \ {randomseed}},......}\n \n and recovering a seed is done \ using the function rcvrExpr and\n requires knowing the RunID and time-date \ stamp which is output during a\n run of bpnn. RunID is a string. If you \ specify an existing file name for\n storing the random seed, a new seed is \ appended to those already in the\n file.\n\nf) The maximum size of the \ data held during the calculation cycle of the\n network is determined by \ the query at (10) and (11). Once this maximum\n has been reached, with \ each additional epoch the oldest data item is discarded.\n This is done \ to prevent exceedingly large numbers of epochs of system rms error \n and \ network output data using up all of RAM. At each epoch rms error is saved \n \ for each output cell, and network output for each output cell is saved for \ each\n exemplar. Thus, the number of output cells and exemplars determines \ how\n much RAM memory will be used during a run. Plotting of\n data by \ epoch at the end of the simulation only uses the maximum\n amount of data \ saved in RAM. Data saved to a file is for all epochs.\n \ng) When the input \ parameter \"exemplar order\" is 0 (item 5 above), each\n exemplar is \ presented once in its natural order in one epoch. When this\n input \ parameter is 1, a random index is used to select an exemplar and\n \ although the long range expectation is that exemplars will be presented\n \ with equal frequency when token frequencies are set to 1, variations \n \ will exist. A frequency vector of exemplar presentations is given at the\n \ end of each epoch cycle. If you use the plotting routines (e.g., Notebook\n \ netAn1) you must select fixed order of presentation since the\n \ functions assume a natural ordering of the exemplars. When the\n \ exemplar span is less than the number of exemplars, data for the\n \ non-presented exemplars are padded with zero values to maintain a\n \ consistent data structure for purposes of plotting.\n \nh) When the selected \ number of epochs have been reached the user is prompted\n by \"How many \ more epochs?\"; if the user enters an integer n, then n more\n epochs are \ carried out. The data saved in RAM at each epoch is still subject\n to the \ maximum specified in items 10 and 11.\n \ni) An estimate of the simulation \ run time can be obtained from the vector\n printed following Start of \ simulation and End of simulation. The 6 \n element vector is {yr,month, \ day,hour,min,sec}.\n \nj) If plots showing the dynamic characteristics of \ the network behavior are to be\n used, there is merit in assigning the \ random number range to a narrow band, e.g.,\n {-0.1,+0.1}. The narrow \ band will allow the user to more easily observe how\n responses and \ weights \"migrate\" to their final values.", Evaluatable->False, AspectRatioFixed->True] }], "Text", Evaluatable->False, AspectRatioFixed->True] }, Closed]], Cell[CellGroupData[{ Cell[TextData[{ StyleBox["Data Input Structure", Evaluatable->False, AspectRatioFixed->True, FontWeight->"Bold"], StyleBox["", Evaluatable->False, AspectRatioFixed->True] }], "Text", Evaluatable->False, AspectRatioFixed->True], Cell[TextData[ "\n1) The exemplar matrix is of the form \n {{{attributes of exemplar \ 1},{attributes of exemplar 2},...}},{{answer for\n output cell 1 for all \ exemplars},{answer for output cell 2 for all\n exemplars}...}}}\n Part 1 \ is of Dimensions[] (# of exemplars, # of input cells)\n Part 2 is of \ Dimensions[] (# of output cells, # of exemplars)\n \n2) optional external \ weights of the form (j=#input cells, k=#hidden cells):\n \ {{{h11,h21,...hj1},{h12,h22,...hj2},...}},\n \ {{o11,o21,....ok1},{o12,o22,...ok2},...}}}\n Part 1 is of Dimensions[] (# \ of hidden cells, # number of input cells)\n Part 2 is of Dimensions[] (# \ output cells, # of hidden cells)\n The results returned by the neural \ network function bpnn is in the form\n described here."], "Text", Evaluatable->False, AspectRatioFixed->True] }, Closed]], Cell[CellGroupData[{ Cell[TextData[{ StyleBox["Executing the Simulation", Evaluatable->False, AspectRatioFixed->True, FontWeight->"Bold"], StyleBox[" ", Evaluatable->False, AspectRatioFixed->True] }], "Text", Evaluatable->False, AspectRatioFixed->True], Cell[TextData[ "The simulation is started by the statement:\n bpnn[data] when no external \ weights for the hidden and output cells are\n used, and\n bpnn[data,wts] if \ weights are used. The structure of data is defined in\n Data Input Structure \ given above. bpnn will return a data structure of\n weights which can be \ used to continue a run, eg., use\n runwts=bpnn[data] (no external starting \ weights for initial run), then\n bpnn[data,runwts] (use external weights \ called \"runwts\" for next run)\n \n The user should make a note of the Run \ ID and the date time stamp\n displayed at the beginning of the execution as \ these data are required\n in the case data saved to a file are to be \ recovered."], "Text", Evaluatable->False, AspectRatioFixed->True] }, Closed]], Cell[CellGroupData[{ Cell[TextData[{ StyleBox["Results Displayed", Evaluatable->False, AspectRatioFixed->True, FontWeight->"Bold"], StyleBox["", Evaluatable->False, AspectRatioFixed->True] }], "Text", Evaluatable->False, AspectRatioFixed->True], Cell[TextData[ "\n1) Input parameters entered by the user are displayed, e.g., run ID, \ date-time stamp, number of input, hidden, and output cells, number of epochs, \ files set up, source of random seed.\n2) At the end of a cycle of epochs, the \ output response to each exemplar adjacent to the target answer is displayed \ as well as the RMS error at each output cell. The frequency of exemplar \ presentation is also displayed.\n3) Upon no further epochs being requested, \ the hidden cell weight matrix, the hidden cell response matrix, and ouput \ cell weights are displayed. \n4) The user may optionally ask for a plot of \ the RMS error, a plot of the output response for each cell over epochs, and a \ 3D plot of exemplar by epoch by response for each output cell."], "Text", Evaluatable->False, AspectRatioFixed->True] }, Closed]], Cell[CellGroupData[{ Cell[TextData[{ StyleBox["Structure of Files Written", Evaluatable->False, AspectRatioFixed->True, FontWeight->"Bold"], StyleBox["", Evaluatable->False, AspectRatioFixed->True] }], "Text", Evaluatable->False, AspectRatioFixed->True], Cell[TextData[ "The user is queried at 3 points to save data once the bpnn function has been \ started: \na) during the parameter input phase the user is queried as to \ which of the following data files are to be saved: squared-error, hidden \ weights, hidden response,output weights, and output response.\nb) also during \ the input phase, the user is queried as to whether the random seed should be \ saved.\nc) after the simulation has been terminated by specifying 0 more \ epochs, the user is given the option to save in one file, the random seed and \ for the last epoch only, the hidden weights, hidden response, output weights, \ and output response. This option allows the user a \"last chance\" to save \ basic data of the simulation deemed of interest but not necessarily \ anticipated prior to starting the simulation. "], "Text", Evaluatable->False, AspectRatioFixed->True], Cell[TextData[{ StyleBox[ "File structure for squared-error, hidden weights, hidden responses, output \ weights, and output responses.\nThe following abbreviations are used:\n# = \ number of\nic[[i]] ith input cell hc[[j]] jth hidden \ cell\noc[[k]] kth output cell ex[[n]] nth exemplar\n\nError \ File: \n {oc[[1]], oc[[2]],.....oc[[k]]} repeated for each \ ITERATION. \nThe order is (#iterations, #output cells)\n \nHidden \ Cell Responses File:\n {\n {hc[[1]], hc[[2]],...hc[[j]]}, for \ exemplar 1\n {hc[[1]], hc[[2]],...hc[[j]]}, for exemplar 2\n \ .................\n {hc[[1]], hc[[2]],...hc[[j]]}, for exemplar n\n \ } \n repeated for each epoch. The order is (#epochs, #exemplars, #hidden \ cells).\n\nHidden Cell Weights File\n {\n {ic[[1]], \ ic[[2]],....ic[[i]]}, for hidden cell 1\n {ic[[1]], \ ic[[2]],....ic[[i]]}, for hidden cell 2\n ...............\n \ {ic[[1]], ic[[2]],....ic[[i]]} for hidden cell j\n } \n repeated \ for each epoch. The order is (#epochs, #hidden cells, #input cells).\n \n\ Output Cell Weights File\n {\n {hc[[1]], hc[[2]],.....hc[[j]]}, output \ cell 1\n {hc[[1]], hc[[2]],.....hc[[j]]}, output cell 2\n \ ...........\n {hc[[1]], hc[[2]],.....hc[[j]]} output cell k\n }\n \ repeated for each epoch. The order is (#epochs,#output cells, #hidden cells).\ \n \nOutput Cell Response File\n {\n {ex[[1]], ex[[2]], ...,ex[[n]]}, \ output cell 1\n {ex[[1]], ex[[2]], ...,ex[[n]]}, output cell 2\n \ ..........\n {ex[[1]], ex[[2]], ...,ex[[n]]}, output cell k\n } \n \ repeated for each epoch. The order is (#epochs, #output cells, #exemplars).\n \ \n Analysis of the dynamic characteristics of the simulation require data in \ epoch units. Notebook netAn1 uses hidden and output cell responses at each \ epochs, and netAn2 uses hidden and output weights at each epoch for \ plotting. \n\n", Evaluatable->False, AspectRatioFixed->True], StyleBox["Random Seed File", Evaluatable->False, AspectRatioFixed->True, FontWeight->"Bold"], StyleBox[ "\nThe format of the expression(s) stored in the random seed file are as \ follows:\n{runID, date-time stamp,{random seed}}. The date-time-stamp is made \ up of an integer formed by the digits of the year, month, day, hour, minute, \ and second. It is displayed at the start of the simulation. If the user \ stores many random seeds in the same file, the function rprtFdata can be used \ to obtain a list of the runID and date-time stamps in the file, and the \ function rcvrExpr can be used to retrieve them.\n\nNote: If the last \ simulation has been run using external weights from a previous session, the \ global variable rstX containing a seed for generating random numbers may or \ may not exist. (If a run using a random seed was made during the current \ session, rstX will exist, but may not be the seed from which the external \ weights were generated as the latter may have been recovered from a file \ created at a different session. In this case saving rstX will not provide a \ regeneration of the simulation.)\n\n", Evaluatable->False, AspectRatioFixed->True], StyleBox["Last Epoch Data File", Evaluatable->False, AspectRatioFixed->True, FontWeight->"Bold"], StyleBox[ "\nThe structure of this file is as follows:\n{\n{runIDsed, date-time \ stamp,{random seed}},\n{runIDerr, date-time stamp,{squared error at each \ output cell}},\n{runIDhwt, date-time stamp,{hidden cell weights}},\n\ {runIDhdr, date-time stamp,{hidden cell responses}},\n{runIDowt, date-time \ stamp,{output cell weights}},\n{runIDrsp, date-time stamp,{output cell \ responses}}\n}\nwhere runID is given by the user during the parameter input \ phase. The function rcvrExpr[] may be used to recover any of these items. For \ example, rcvrExpr[\"ABCsed\",19970318220020] would recover the random seed \ for the runID \"ABC\" and date-time stamp 19970318220020. Note that 3 \ additional characters are appended to the runID and must be included when \ using rcvrExpr to get the appropriate expression from the file. The user \ named file is opened for appending so results of several simulations may be \ accumulated in the same file. If this is not desired, in function bpnn change \ the statement iofname=openFna[filelist,2] to iofname=openFna[filelist,1]. \n\ \nThe function rprtFdata can be used to obtain a list of the runID and \ date-time stamps of expressions in the file. It is important for the user to \ keep accurate records of file names, Run IDs, and date time stamps of the \ simulation runs in order to be able to recover the appropriate data. \n\n", Evaluatable->False, AspectRatioFixed->True] }], "Text", Evaluatable->False, AspectRatioFixed->True] }, Closed]], Cell[CellGroupData[{ Cell[TextData[{ StyleBox["Description of Variable Names in the function bpnn", Evaluatable->False, AspectRatioFixed->True, FontWeight->"Bold"], StyleBox[" ", Evaluatable->False, AspectRatioFixed->True] }], "Text", Evaluatable->False, AspectRatioFixed->True], Cell[TextData[ "ans general variable holding current user response\nc \ response to each exemplar; structure is \n \ {{{outcel1},{outcel2},....}},{{outcel1},{outcel2}}....} in which\n \ each {outcel} has as many elements as exemplars.\n \ {{outcel1},{outcel2}} appears once for each iteration saved.\ncflat \ Flatten[c]; used for plotting\ndtyp vector holding 3 character \ expression identifier\nerr error at output cell layer\nexcntr \ exemplar presentation frequencies\nfilelist descriptive name of files to \ hold data\nhc number of hidden cells\nhdel hidden cell layer \ delta; adjustment\nhdout hidden cell layer output\nhdrsp hidden cell \ layer response\nhedr holds runID and 3 characters from dtyp \nhwt \ hidden cell weights\ni index counter\ninc number of input \ cells\ninx index to select exemplars\nioerfsw switch to write \ error-sqd file; 0=no write; 1=write\niofname name of files opened; returned \ by openFna\niofsw file output switch; 0=none;1= one or more to be setup\n\ iterinc number of additional epochs\nitr epochs for current cycle\ \nitrcopy copy of itr argument\nlabrc holds row/column titles for \ display of matrices\nlpar1 \"learning parameter\" applied to adjustment \ of output weights\nlpar2 \"learning parameter\" applied to adjustment of \ hidden weights\nmaxepoc maximum amount of output data to save in RAM by epoch\ \nmaxts maximum amount of rms error data to save in RAM by epoch\nnamerf \ holds file name for error-sqd data\nodel output layer adjustment\n\ ord exemplar order 0/1\notc number of output cells\noutput \ output at output layer\nowt weights used at output layer\npadhdr \ holds padding (0) to bring hdrsp to full exemplar size\npadzero holds \ padding (0) to bring xresp to full exemplar size\nrmse root mean sqd \ error at end of an epoch\nrndt run date-time stamp; 14 digit integer; \ yr,m,d,h,m,s\nrsin2 holds runidc and 3 char description of data in file \ when seed and\n last epoch data is saved\nrunidc run ID code \ input by user; string\nsqderr accumulates rmse error current \ cycle;{{outcel1,outcel2...},....}\ntotitr total number of epochs\nttl \ holds titles for plots\nx exemplar matrix\nxfreq \ values used in this list are used as the iterator to repeat the \n \ presentation of an exemplar as given by xseq\nxresp holds the system \ response for each output cell at a given iteration; it\n is \ padded with 0 if the exemplar span is less than the number of\n \ exemplars\nxseq holds index order for presentation of exemplars\nxsp \ exemplar span; ordered number of exemplars to use\ny \ exemplar correct response or target response\nysp y for current range \ of exemplars "], "Text", Evaluatable->False, AspectRatioFixed->True] }, Closed]], Cell[CellGroupData[{ Cell[TextData[{ StyleBox["Functions Called", Evaluatable->False, AspectRatioFixed->True, FontWeight->"Bold"], StyleBox["", Evaluatable->False, AspectRatioFixed->True] }], "Text", Evaluatable->False, AspectRatioFixed->True], Cell["\<\ The Main function (bpnn) calls the following functions directly or indirectly. Functions contained in this Notebook are as follows: matmerg: to interlace cols of output and desired response (target) so they can be seen side by side getxfrc: presents option to input a token frequency vector logistic: to carryout logistic transformation getLpar: to get learning parameters as scalar or vector usrsed: to establish a random number seed, and generate hidden and output weights randomly wrsdFile: to write a random number seed to a file clr2: a color function which determines the color of a 3D surface \ plotted using RGB; currently commented out in bpnn and Hue used \ instead. The following functions are used by bpnn and contained in the Package netUtil \ with their documentation in the Notebook netUdoc: kbin3: to input numeric parameter values by user kbinstrg: to input alphanumeric answer by user plotnLists: to plot sqd-error and output for each output cell plotLegend: to plot a legend for colored lines used in plotnLists pmat2: displays a matrix m in column widths of pc units; 2nd parameter \ can be changed so that for small screens no matrix wrap-around appears. genrch: used to generate row and column headings for pmat2 openFna: to open the files user wants for storage of results timeDate: used to get the year,month,day,hr,min, sec rndoff: roundoff values in m to d decimal digits The following are useful functions not called by bpnn which are contained in \ the Package netUtil. Their documentation documentation is in the Notebook \ netUdoc. rcvrExpr: used to recover an expression from a random seed file, or last epoch data file written by an earlier run of bpnn in which by option, the expression was saved. rprtFdata: can be used to obtain a summary of data stored in the random seed file or last epoch data file. saveExpr: saves an expression to a file Other Functions: tcdspla: used to generate density plot of character matrix of example problem \ \>", "Text", Evaluatable->False, AspectRatioFixed->True] }, Closed]], Cell[CellGroupData[{ Cell[TextData[{ StyleBox["Example Problem", Evaluatable->False, AspectRatioFixed->True, FontWeight->"Bold"], StyleBox["", Evaluatable->False, AspectRatioFixed->True] }], "Text", Evaluatable->False, AspectRatioFixed->True], Cell[TextData[{ StyleBox[ "\nThe data set tcL is a variation of Freeman's T and C problem (see \ Simulating Neural Networks with ", Evaluatable->False, AspectRatioFixed->True], StyleBox["Mathematica", Evaluatable->False, AspectRatioFixed->True, FontSlant->"Italic"], StyleBox[ " , Freeman, J.A., Addison Wesley, p77) in which the characters T and C are \ rotated clockwise successively by 90 degrees to form 8 exemplars. The network \ is to distinguish the characters regardless of their orientation. In the \ example used here, the character L has been added to the exemplars to make \ the problem a little more difficult. See the section \"The T, C, and L \ pattern recognition problem\" following the listing of the functions. The \ example is run using 9 input cells, 5 hidden cells, and 3 output cells. This \ example data is in the cell \"Input Data (exemplars and answers (target)\".\n\ \n", Evaluatable->False, AspectRatioFixed->True] }], "Text", Evaluatable->False, AspectRatioFixed->True] }, Closed]], Cell[CellGroupData[{ Cell[TextData[{ StyleBox["bpnn[xy,exwts]", Evaluatable->False, AspectRatioFixed->True, FontWeight->"Bold"], StyleBox[ " is the back propagated neural network function.\nxy holds the structure \ containing the encoded exemplar matrix and answer (target) matrix. (See Data \ Input Structure above).\nexwts is optional; if used it contains a structure \ holding the hidden layer cell weights (#hidden cells, #input cells) and the \ output layer cell weights (#output cells, #hidden cells). When exwts is in \ the argument list, random number weights are not generated. Normally, exwts \ would be used when an earlier simulation is to be continued.\nThe function \ bpnn returns a structure containing the hidden cell weights of order (#hidden \ cells, #input cells), and the output cell weights of order (#output cells, \ #hidden cells) which may be used as the variable exwts in further runs, for \ purposes of further analysis, or for assessing the network's behavior with \ exemplars not used in the training set of exemplars. \n\nCalls: See Functions \ Called (above)\nReturns: {hidden cell wts, output cell wts}\nRequired: \ netUtil Package\n\nNote: In the 3D plot of output layer responses, the \ statement ColorFunction->(clr2[#]&),ColorOutput->RGBColor has been commented \ out and replaced with ColorOutput->Hue. If the user wishes to specify the \ color separation using RGB color and the function clr2, then comment markers \ can be removed, and ColorOutput->Hue commented out. \n\nMod. Date: Apr 29/97\n\ ", Evaluatable->False, AspectRatioFixed->True] }], "Text", Evaluatable->False, AspectRatioFixed->True], Cell[CellGroupData[{ Cell[BoxData[ \(\(bpnn[xy_, exwts___] := Module[{ans, hwt, owt, itrcpy, totitr, x, y, sqderr, c, err, cflat, iterincr, inx, hdout, output, xseq, xsp, ysp, rnr, padzero, odel, hdel, ttl, hdrsp, excntr, xfreq, lpar1, lpar2, itr, inc, hc, otc, rs, maxts, maxepoc, ord, xresp, padhdr, iofsw, iofname, filelist, ioerfsw, namerf, rmse, runidc, rndt, rsin, fileseed, fios, rsin2, labrc, i, dtyp, hedr}, Print["\"]; \n \t\t (*check\ the\ structure\ of\ xy\ before\ going\ on*) \n\t\t If[Length[xy] == 0, Print["\"]; Return[]]; runidc = InputString["\"]; Print["\", runidc]; rndt = Plus@@\((Date[]\ Reverse[Table[10\^\(x\ 2\), {x, 0, 5}]])\); Print["\", rndt]; ioerfsw = 0; inc = kbin3["\", 0]; Print["\", inc]; If[Length[Transpose[xy\[LeftDoubleBracket]1\[RightDoubleBracket]]] \[NotEqual] inc, Print["\"]; Return[]]; hc = kbin3["\