netCls:Classify a Set of Exemplars
Notebook Version: 2.0
Steve Hunka, University of Alberta
steve.hunka@ualberta.ca
Mod Date: Oct 25/97; minor text revisions

General Purpose of Notebook

Frequently a researcher uses a "training" set of exemplars to train the network, and then once the hidden and output weights are available these are applied to a new set of exemplars which were not used in the "training" set. The new set of exemplars may have, for example, more "noise" in their attributes. The behavior of the network to these exemplars compared with those in the training set indicates the network's robustness under the conditions of noise. This Notebook has functions for calculating the hidden cell and output cell responses for a set of exemplars and bar plotting functions for their visual representation. Data for the last epoch are used from files TCLhw and TCLow. The data can also be obtained from TCLast.

Required Packages

<< "Graphics`Graphics`"<< "netUtil`"

Functions used by this Notebook

This Notebook contains the following functions:

otrbarplot  bar plotting function for output responses
hdrbarplot bar plotting function for hidden responses
rspbarplot  to do the bar plotting
araydis  called by rspbarplot to display array of bar plots
logistic  to calculate logistic function to find responses
bpnresp  to calculate hidden and output cell response for a set of exemplars

The following functions are in the Package netUtil and documented in netUdoc.
kbin3      to input numeric data from keyboard
kbinstrg   to input string data from keyboard
inpvec     to check range of elements in a vector
pmat2      to display a matrix
genrch     to provide row & column headings for pmat2
rndoff      to round off numbers

Input Data Structure

Data Input Structure to bpnresp

1) xwts: weights of the form (j=#input cells,k=#hid cells:
   {{{h11,h21,...hj1},{h12,h22,...hj2},...}},
   {{o11,o12,....ok1},{o12,o22,...ok2}...}}}
   The hidden cell weights is a matrix of order (# hid cells,# in cells), and the
   output cell weight is a matrix of order (# out cells, #hid cells) returned by
   the neural network program. Alternatively, the last hidden and output cell
   weights can be obtained from the files loaded by the neural net program.
   
2) ex: exemplar matrix; of the form 
   {{attributes of exemplar 1},{attributes of exemplar 2},...}}

Functions

bpnresp[xwts,ex] : calculates hidden cell and output cell response to a set of 
exemplar inputs and provides bar plots as a user option. The user's input of
number of input, hidden, and output cells is used to check the order of the
the weights in xwts.
xwts:  are the last epoch hidden weights and output weights having the structure
         {{hidden wts},{output wts}}
ex:      are the exemplars to be classified
Calls: kbin3, logistic,rndoff,pmat2,genrch
Mod. Date: Mar 16/97

bpnresp[xwts_, ex_] := 
      Module[{inc, hc, oc, hwt, owt, hdresp, otresp, hdrlist, otrlist, rc}, 
        inc = kbin3["Number of input cells?", 1]; 
        hc = kbin3["Number of hidden cells?", 1]; 
        oc = kbin3["Number of output cells?", 1]; 
        Print["Input parameters=", {inc}, {hc}, {oc}]; 
        Print["Dimensions Hidden Wts=", 
          Dimensions[xwts[[1]]]]; 
        Print["Dimensions Output Wts=", 
          Dimensions[xwts[[2]]]]; 
        If[Dimensions[xwts[[1]]] == {hc, 
                inc} && 
            Dimensions[xwts[[2]]] == {
                oc, hc}, 
          hwt = xwts[[1]]; 
          owt = xwts[[2]]; 
          Print["Weight matrices OK"];, 
          Print["External wts dimension error"]; Return[]]; 
        Print["Number of exemplars=", 
          Length[xwts[[1]]]]; 
        hdrlist = {}; otrlist = {}; 
        Do[hdresp = 
            logistic[hwt . ex[[j]]]; 
          AppendTo[hdrlist, hdresp]; otresp = logistic[owt . hdresp]; 
          AppendTo[otrlist, otresp];, {j, 1, Length[ex]}]; 
        Print["Calculated Hidden Cell Responses"]; 
        rc = genrch["Ex", "Hid", Dimensions[hdrlist]]; 
        pmat2[rndoff[hdrlist, 3], 5, rc]; 
        Print["\nCalculated Output Cell Responses"]; 
        rc = genrch["Ex", "Out", Dimensions[otrlist]]; 
        pmat2[rndoff[otrlist, 3], 5, rc]; 
        If["y" == 
            kbinstrg[
              "Bar plot of hidden response? y/n", {{"y"}, {
                  "n"}}], hdrbarplot[hdrlist]]; 
        If["y" == 
            kbinstrg[
              "Bar plot of output response? y/n", {{"y"}, {
                  "n"}}], otrbarplot[Transpose[otrlist]]]; ];

Logistic Function

logistic[p_] := N[(1/(1 + (E^(-p))))];

hdrbarplot[d] creates a bar plot of the hidden cell response values for each exemplar. X-axis is hidden cell number; Y-axis is response value
d:  is the matrix of hidden cell response values of order (#exemplars,#hidden cells)
User Input: See rspbarplot
Calls: rspbarplot, kbinstrg, kbin3, araydis
Mod. Date: Mar 11/97

hdrbarplot[d_] := 
      Module[{nex}, 
        Print["Bar Plot of Hidden cell responses for each exemplar"]; 
        Print["Input data dimensions=", Dimensions[d]]; 
        Print["Number of hidden cells=", 
          Dimensions[d][[2]]]; 
        nex = Part[Length[d]]; Print["Number of exemplars=", nex]; 
        rspbarplot[Transpose[d], nex, {"HidCell", "Resp"}]; ];

otrbarplot[d] creates a bar plot of the output cell response values for each exemplar. X-axis is output cell number; Y-axis is response value
d:  is the matrix of output cell response values of order (#outcells,#exemplars)
User Input: See rspbarplot
Calls: rspbarplot, kbinstrg, kbin3, araydis
Mod. Date: Mar 11/97

otrbarplot[d_] := 
      Module[{nex}, 
        Print["Bar Plot of Output cell responses for each exemplar"]; 
        Print["Input data dimensions=", Dimensions[d]]; 
        Print["Number of output cells=", Length[d]]; 
        nex = Dimensions[d][[2]]; 
        Print["Number of exemplars=", nex]; 
        rspbarplot[d, nex, {"OutCell", "Resp"}]; ];

rspbarplot[d,nx,xat] plots cell response to each exemplar as a bar chart. 
d: is the matrix of output responses of order (#output cells or #hidden cells,#exemplars).
nx: is the number of exemplars
xat: contains x and y axis labels of the form {"xaxis","yaxis"}
X-axis is 1 to the number of cells; Y-axis is the value of the response.  Individual plots of all or selected exemplars can be made as well as an array plot.
Called by: hdrbarplot, otrbarplot
User Input:
a) plot all exemplars (y/n); if n plot selected exemplars
b) single plots for exemplars (y/n)
c) group single plots into an array (y/n)
d) if array plot selected, a vector of the number of plots per row
e) if selected exemplars plotted, a vector of the exemplar indices
Notes: The adequacy of the numbers on the X-axis will depend upon the number of cells. The font size of the X-axis digits is set in DefaultFont as 8. The font size of the PlotLabel is set to 12. These font values may need to be changed by the user to suite a particular plotting case. 
Calls: kbinstrg, kbin3, araydis,inpvec
Mod. Date: Mar 25/97

rspbarplot[d_, nx_, xat_] := 
      Module[{dsl, neplt, nex, exsl, pobj, sp, ap, i, err, npr}, 
        If["y" == 
            kbinstrg["Plot all exemplars? y/n", {{"y"}, {"n"}}], 
          Print["Plot all exemplars=y"]; dsl = Transpose[d]; neplt = nx; 
          exsl = Range[nx];, Print["Plot all exemplars=n"]; 
          neplt = kbin3["How many exemplars to plot?", 1]; 
          exsl = inpvec[{1, nx}, neplt, "exemplars"]; 
          Print["Exemplars to be plotted=", exsl]; 
          dsl = Transpose[d][[exsl
              ]]; ]; 
        If["y" == 
            kbinstrg[
              "Single plots for each exemplar? y/n", {{"y"}, {
                  "n"}}], sp = $DisplayFunction; 
          Print["Single plots=y"], sp = Identity; 
          Print["Single plots=n"]]; 
        If["y" == 
            kbinstrg[
              "Group single plots into array? y/n", {{"y"}, {
                  "n"}}], ap = 1; Print["Group in array plot=y"], 
          ap = 0; Print["Group in array plot=n"]]; i = 0; 
        pobj = (
              BarChart[#1, AxesLabel -> xat, 
                  DefaultFont -> {"Times", 8}, 
                  DisplayFunction -> sp, PlotRange -> All, 
                  PlotLabel -> 
                    FontForm[
                      StringInsert["Exemplar ", 
                        ToString[
                          exsl[[i = 
                              i + 1]]], -1], {
                        "Times", 12}]]&)/@dsl; 
        If[1 == ap, err = 1; 
          While[err == 1, 
            npr = Input["Type vector of plots per array row"]; 
            If[nx < Plus@@npr, 
              Print["Error: sum plots per row > # plots"], err = 0]; ]; 
          Show[GraphicsArray[araydis[pobj, npr]], 
            DisplayFunction -> $DisplayFunction]]; ];

araydis[m,in]; arranges the graphic object plots in m for GraphicsArray according to the arrangement specified by in. For example if m had 9 plots, and in={2,3,4} then the 1st row will contain 2 plots, the next 3 plots, and the last row 4 plots as displayed by Graphics Array. This function differs from araydis3 in that  araydis allows specification of the number of plots in each row of the array  while araydis3 does not.
Length[in] <= Length[m].
Mod. Date: Feb 20/97

araydis[m_, in_] := 
      Module[{pa, ma, nt}, pa = {}; nt = Length[in]; ma = m; 
        Do[AppendTo[pa, 
            Take[ma, in[[i]]]]; 
          ma = Drop[ma, in[[i]]];, {
            i, 1, nt}]; Return[pa]; ];

Example Input Data: tcL holds exemplar and answers for the 3 character problem T,C, and L; hdw3c holds the hidden cell wts read from file TCLhw, and otw3c holds the output weights read from file TCLow as written by bpnn.

tcL = {{{ .9,  .9,  .9,  .1,  .9,  .1,  .1,  .9,  .1}, { .1,  .1, 
             .9,  .9,  .9,  .9,  .1,  .1,  .9}, { .1,  .9,  .1,  .1,  .9, 
             .1,  .9,  .9,  .9}, { .9,  .1,  .1,  .9,  .9,  .9,  .9,  .1, 
             .1}, { .9,  .9,  .9,  .9,  .1,  .1,  .9,  .9,  .9}, { .9,  .9, 
             .9,  .9,  .1,  .9,  .9,  .1,  .9}, { .9,  .9,  .9,  .1,  .1, 
             .9,  .9,  .9,  .9}, { .9,  .1,  .9,  .9,  .1,  .9,  .9,  .9, 
             .9}, { .9,  .1,  .1,  .9,  .1,  .1,  .9,  .9,  .9}, { .9,  .9, 
             .9,  .9,  .1,  .1,  .9,  .1,  .1}, { .9,  .9,  .9,  .1,  .1, 
             .9,  .1,  .1,  .9}, { .1,  .1,  .9,  .1,  .1,  .9,  .9,  .9, 
             .9}}, {{ .9,  .9,  .9,  .9,  .1,  .1,  .1,  .1,  .1,  .1,  .1, 
             .1}, { .1,  .1,  .1,  .1,  .9,  .9,  .9,  .9,  .1,  .1,  .1, 
             .1}, { .1,  .1,  .1,  .1,  .1,  .1,  .1,  .1,  .9,  .9,  .9, 
             .9}}};

tcL1 = tcL[[1]];

hdw3c = << "TCLhw";

otw3c = << "TCLow";

Example Run to reproduce bpnn results (*no new exemplars*)

bpnresp[{hdw3c, otw3c}, tcL1]

Example Input Data: tcL2 holds exemplar and answers for the 3 character problem T,C, and L. The 5th exemplar is T with cell (1,3) set to .1 instead of .9, i.e, the T is missing one shaded area. There are 13 exemplars.

tcL2 = {{{ .9,  .9,  .9,  .1,  .9,  .1,  .1,  .9,  .1}, { .1,  .1, 
             .9,  .9,  .9,  .9,  .1,  .1,  .9}, { .1,  .9,  .1,  .1,  .9, 
             .1,  .9,  .9,  .9}, { .9,  .1,  .1,  .9,  .9,  .9,  .9,  .1, 
             .1}, { .9,  .9,  .1,  .1,  .9,  .1,  .1,  .9,  .1}, { .9,  .9, 
             .9,  .9,  .1,  .1,  .9,  .9,  .9}, { .9,  .9,  .9,  .9,  .1, 
             .9,  .9,  .1,  .9}, { .9,  .9,  .9,  .1,  .1,  .9,  .9,  .9, 
             .9}, { .9,  .1,  .9,  .9,  .1,  .9,  .9,  .9,  .9}, { .9,  .1, 
             .1,  .9,  .1,  .1,  .9,  .9,  .9}, { .9,  .9,  .9,  .9,  .1, 
             .1,  .9,  .1,  .1}, { .9,  .9,  .9,  .1,  .1,  .9,  .1,  .1, 
             .9}, { .1,  .1,  .9,  .1,  .1,  .9,  .9,  .9,  .9}}, {{ .9,  .9, 
             .9,  .9,  .1,  .1,  .1,  .1,  .1,  .1,  .1,  .1}, { .1,  .1, 
             .1,  .1,  .9,  .9,  .9,  .9,  .1,  .1,  .1,  .1}, { .1,  .1, 
             .1,  .1,  .1,  .1,  .1,  .1,  .9,  .9,  .9,  .9}}};

tcL2 = tcL2[[1]];

bpnresp[{hdw3c, otw3c}, tcL2]

Notes to Above Bar Plots

Exemplar 5, which has the missing shaded cell in its character matrix, at the hidden cell level is treated very much like T in its unrotated form (Exemplar 1) and T rotated 270 degrees (Exemplar 4). This can be seen by the similarity of the pattern for Exemplars 1and 4,  and Exemplar 5. At the output level (last set of bar plots), the network categorizes Exemplar 5 as a T without any problem even though it has never seen this exemplar before.