UseMPW

Sample Notebook
v1.0, April 1993

Roland Jakschewitz
 
rlj-nbg@infohh.rmi.de
GER.XSE0027@applelink.apple.com

UseMPW is an external program running on Macintosh computers under System 7,
which "pipes" MPW (Macintosh Programmer's Workshop) commands from Mathematica
to MPW ToolServer, and "pipes" the resulting output back into Mathematica. The
communication between Mathematica and UseMPW is via MathLink, between UseMPW
and ToolServer via Apple Events. ToolServer can either be on the local machine
or on any machine on the network.
 
Main purpose is to speed up lengthy Mathematica computations like parsing, by
delegating them to available MPW tools - like GNU awk, or MPW Search or
Replace. These tools all provide full-fledged regular expressions, which is
something not supplied by Mathematica.
Another advantage is that special purpose MPW tools need not be modified to be
MathLink compatible, and can be used from MPW or Mathematica - or from any
other application which can send Apple Events to ToolServer (like e.g.
FileMaker Pro 2.0 and others).

After modifying the path in SubSubSection
(Un)Install:Using UseMPW:which path to take:
to the directory containing the external command and the supporting package,
you should be able to "Evaluate Initialization" and proceed with the Usage
Samples.

(Un)Install

Testing UseMPW
(in case the external command will be modified)

Normally the supporting package UseMPW.m installs the link when the package is
evaluated (by Needs or Get). For the purpose of debugging (modifications to)
the external command UseMPW.m should be modified not to auto-install the link.

lnk = Install[ "UseMPW" ]

Debugging

lnk = Install[ "UseMPW", LinkMode->Connect ]

Check

?UseMPW

Uninstalling

Uninstall@lnk

Uninstall@Links[][[1]] (* if lnk is not defined *)

Links[]

Using UseMPW

which path to take (unless you like to hunt for files in StandardFiles dialogs)

Setting the current directory AND appending to $Path will load the package and
launch the external command without any user interaction - since I don't move
my files as often as I use them, this seems to be just the right amount of user
interface.

You will have 2 path symbols, if the external command is not in the same
directory as the package:

myUseMPWpath =
"Daten:Work:Math:MathLinkTrainingCourse:MMaToMPW:UseMPW.Source:";

>>> set to the directory where the external command UseMPW is located <<<
this works for automatic find of UseMPW (but not for UseMPW.m)

SetDirectory[ myUseMPWpath ]

>>> set to the directory where the package UseMPW.m is located <<<
this works for automatic find of UseMPW.m (but not for UseMPW)

AppendTo[ $Path, myUseMPWpath ]

read in the package (auto-installing the external command)

Needs["UseMPW`"]

$ContextPath (* should contain UseMPW` *)

?UseMPW

?MPW

Uninstall

Uninstall /@ Cases[ Links[], _["UseMPW", __] ]

Usage Samples

Preliminaries

Which are MPW's default settings?

Options @ MPW

Aliases used (necessary for following examples):

MPW["alias wc count; alias ls files; alias grep search; alias cd directory;
alias l 'files -x ktm'"]
MPW["alias cat catenate; alias rm delete"]

ToolServer local or on the network

This works only on first invocation of MPW after loading the package, on later
calls, the option will be ignored, since there is already a connection to a
ToolServer.

MPW[ "l", BrowseNetwork->True]

All other samples (start and) use ToolServer on the local machine.

Raw Output

the external function UseMPW without "MPW" encapsulation
>>> (supply directories present on your machine) <<<

UseMPW[ "cd `WhereIs -s Daten: -d UseMPW.Source`;l", 0 ]

UseMPW[ "cd `WhereIs -s Daten: -d UseMPW.Source`;l", 0 ][[3]]

UseMPW[ "l nonExistantFile", 0 ][[4]]

UseMPW[ "cat E.[ch]", 0 ][[3]]

Type->Raw: print output as one string (as in MPW Shell)
>>> (supply directories present on your machine) <<<

MPW[ "cd `WhereIs -s Daten: -d UseMPW.Source`;l", Type->Raw ]

MPW[ "rm E.o;l", Type->Raw ]

MPW[ "grep -s New E.[ch]", Type->Raw ]

ReturnMode Settings

Showing Output

MPW[ "l nonExistantFile" ] (* default setting: Info *)

Showing Status only (3 ways)

MPW[ "l nonExistantFile", ReturnMode->Diagnostic ]

MPW[ "l nonExistantFile", ReturnMode->FullStatus ]

MPW[ "l nonExistantFile", ReturnMode->Status ]

Type Settings

Default Options, format with TableForm

MPW[ "l" ] // TableForm

Type->String with default (RecodLists->True) setting:
needs additional First[] to get rid of outer string bracket

MPW[ "l", Type->String ]

First@MPW[ "l", Type->String ] // TableForm

Type->Strig with additional RecordLists->False

Drop[ MPW[ "l EaE", Type->String, RecordLists->False ], 2 ]

Composite type specification
>>> choose your own search string instead of "New" <<<

searchCommand = "grep -s 'New' E.[ch]";

Take[#, {1,2}]& /@ MPW[ searchCommand,
   Type->{Word, Word, Word, Word, String},
   RecordLists->False
      ] // Union

MPW[ searchCommand,
    Type->{Word, Word, Word, Word, String},
    RecordLists->False ]

#[[2]]& /@ % // Union

Using MPW Output in Mathematica Expressions

Accessing "columns"

Drop[ MPW[ "l" ], 2 ];

Plus@@(ToExpression@#[[2]]& /@ %)

%/. K->1024

Null Return

MPW[ "ls > tempSample" ]

MPW[ "rm tempSample" ]

Use output of MPW command for construction of new command string

File name is second word, so: print first file that contains string "AE"
>>> choose your own search string instead of "AE" <<<

MPW[ "cat " <> MPW[ "grep -s 'AE' E.[ch]", Type->Word ][[1,2]], Type->Raw ]

Checking MPW tools with Mathematica :-)
>>> works best in directories with MathLink template files and C sources <<<

searchCommand = "wc `ls E.[ch] E.tm`";

this is what it looks like in MPW

MPW[ searchCommand, Type->Raw ]

doing the transformation of output lines 'by hand'

{#[[1]], ToExpression@#[[2]], ToExpression@#[[3]]}& /@ MPW[ searchCommand ]

keep numbers of lines and characters only

Drop[#,1]& /@ %

check our sum against count's (of which wc is just an alias)

Plus@@Drop[%,-1] == Last@%

another way of doing the check

let ReadList do the transformation (Note the RecordLists option!)

Drop[ MPW[ searchCommand, Type->{Word,Number,Number}, RecordLists->False ], -1
]

keep numbers of lines and characters only

Drop[#,1]& /@ %

and add them up

Plus@@%

Huge Output

Test memory overflow in server due to huge output

Unless the application heap size of UseMPW has been increased, the following
command will exceed UseMPW's buffering capacity - leading to termination of the
ToolServer script.
 
If there were a way of sending the single parts of the output to Mathematica
successively (without knowing in advance how many parts there will be), the
size of UseMPW's application heap could be substantially reduced - and memory
overflows due to huge output  could be avoided completely.

MPW[ "cat {CIncludes}E.h", ReturnMode->All, Type->Raw ];

InputForm@Last@%

Use awk for faster parsing - and Get the resultant file

As Nancy Blachman writes in Mathematica: A Practical Approach (Chapter 14.1.3),
"Data can be parsed (read in and interpreted) much faster using C, awk, or
shell programming."
My experience with the Macintosh version of Mathematica (2.1) shows that rather
than reading in the output of an awk script using ReadList, it is slightly
faster redirecting awk's output to a file and reading that file with Get[] (or
<<). In the latter case the awk script has to produce a Mathematica List of
records instead of a plain list.
In the following example the original data to be parsed are in FileMaker export
format (newline-separated records of TAB-separated fields of ASCII data in
German number format (decimal comma, triples of places separated by  points)
and German date format, and numbers with leading zeroes):

SetOptions[ MPW, Type->Raw ];

MPW @ "head {TmpDir}allForMath" (* 1st 10 lines *)

The awk script looks like this

MPW @ "cat {TmpDir}FMtoMath.awk"

And the output - one complete Mathematica expression - looks like this:

MPW @ "head {TmpDir}t.out" (* 1st 10 lines *)

At a size of 30K, Get[] is about 20% faster than doing a ReadList[]:

MPW @ "rm {TmpDir}t.out;l {TmpDir}allForMath"
MPW @ "echo {TmpDir}"

this is the fastest way of reading the data, with awk creating 1 Mathematica
expression:

Timing[ MPW["gawk -f {TmpDir}FMtoMath.awk {TmpDir}allForMath >{TmpDir}t.out" ];
]
Timing[ a = <<"Maxl:MPW:usr:tmp:t.out"; ]
MPW @ "l {TmpDir}t.out"

a[[7]]

Slower ways of doing it

slightly slower reading the list:

MPW @ "rm {TmpDir}tl.out"

Timing[ MPW["gawk -f {TmpDir}FMtoMathlist.awk {TmpDir}allForMath
>{TmpDir}tl.out" ]; ]
Timing[ b = ReadList["Maxl:MPW:usr:tmp:tl.out", Expression]; ]

b[[7]]

and a little slower still, if no temporary file is created (how can this be??):

Timing[ c = MPW["gawk -f {TmpDir}FMtoMathlist.awk {TmpDir}allForMath",
   Type->Expression,
   RecordLists->False
   ];
]

c[[7]]