Tools for Application Package Testing

E. C. Martin

Introduction

MathSource includes some new tools to aid application developers.

Guidelines for Mathematica Documentation, Packages, and Notebooks

Package, Notebook, and test file templates

Testing.m for automating function testing and EscapingSymbols.m for detecting missing usage messages or spuriously exported symbols in packages.

This notebook describes the use of the packages Testing.m and EscapingSymbols.m.  Other packages used by the Software Quality Assurance (SQA) department at Wolfram Research, Inc. will be added to MathSource in the future, so developers are advised to query MathSource on a regular basis.

Testing packages

The package Testing.m is used by the SQA department to facilitate testing for correctness and regression testing (i.e., testing whether the results obtained under one version of Mathematica agree with those obtained under another).  The package defines
many useful functions, only a few of which are described here.  More information on the
package can be found in the Discussion comment of Testing.m.

Testing your own packages

Running the tests in an existing test file

The "Guidelines" document gives an example package and an example test file for that package.  In this example., the NumberTheory`ContinuedFractions` package is loaded in the first test of the test file ContinuedFractions1.mt.  (By convention, Mathematica test files have the suffix .mt.)  The test file is "run" using the TestRun, RunFile, and EndRun functions defined in Testing.m.  The results of the run are written to stdout using a special notation.  Note that all test numbers are enclosed in square brackets except for 4 which is enclosed in colons.  This means that all six tests passed: tests 1-3, 5, 6 passed immediately and test 4 passed when the expected ouput was reevaluated.  See the Discussion comment of Testing.m for details of this notation.

<<Testing`
TestRun[]
RunFile["ContinuedFractions1.mt"]
EndRun[]

Several files are created by the test run.  testfailure lists all failed tests.  testlog gives a complete log of the test run.  testrecord maintains a record of all test runs;  this is useful if you are running a number of test files at one time.  testtiming gives the timing for each step in the test run. 

Here the testfailure file is displayed .  Recorded in the file are the time at which the tests started, the version of Mathematica used in testing, the test file that was read, the time of test completion, CPU time, and information on each test failure, such as  expected output and actual output.

!!testfailure

Creating a test file interactively

In the previous example, a test file was run that was created using a text editor (perhaps the editor included in the Notebook front end).  If you were testing a package that does series expansions and you had a math handbook with a table of such expansions, you might create a test file using the input and expected output as given in the table.  This is testing for correctness.  In contrast, if you  have confidence in your package and  would like to characterize its behavior before embarking on some new code development that may "break" your package, you might want to do regression testing.  Testing.m provides functions for creating a test file interactively, so you do not have to use a text editor as you would for the previous example.  Here a test file ContinuedFractions2.mt is created using the functions TestMake and EndMake.

<<Testing.m
TestMake["ContinuedFractions2.mt"]
<<NumberTheory`ContinuedFractions`
ContinuedFraction[6789/1234, 4]
EndMake[]

When the test file is displayed, you can see that the input and output of the Mathematica expressions evaluated between TestMake and EndMake were recorded using the Test function syntax defined in Testing.m.  The author of the test file and the date it was generated are included in a comment.

!!ContinuedFractions2.mt

The new test file can now be run.  The square bracket notation of the report sent to stdout indicates that both tests passed.

TestRun[]
RunFile["ContinuedFractions2.mt"]
EndRun[]

Submitting your packages to Wolfram Research, Inc. for testing

With each revision of Mathematica, a great effort is made to maintain backward-compatibility.  However, it is not always possible to do this and take advantage of bug fixes and new functionality.  If you are concerned about whether your package will be compatible with the version of Mathematica currently under development and are a heavy user of Mathematica, one option is to volunteer for beta testing.  Since only a limited number of developers can be beta testers, another option is to submit your package, together with test files characterizing package behavior, to mathtest@wri.com.  This mailbox is regularly monitored; submissions will be added to the test suite run each evening, and failures will be reported to the submitter.  Test files should be written using the functions provided in Testing.m on MathSource.

Note that this is a way for a developer to do regression testing on a pre-beta version of Mathematica; it is not a way to debug Mathematica code.  WRI welcomes the opportunity to learn of possible incompatiblities between a revision and outside packages early in the development cycle.

Escaping symbols

It is generally considered poor programming practice to introduce symbols in a context visible to the user that do not have usage messages.  The package EscapingSymbols.m contains a utility function that will help you avoid this practice. 

The function EscapingSymbols compares the list of all symbols visible to the user prior to evaluating its argument to the list of symbols visible to the user after evaluating its argument.  Any new symbols lacking usage messages are called escaping symbols.

If the argument to EscapingSymbols loads a package, this function can help the author determine which symbols need usage messages or which portions of code need to be changed to avoid introducing spurious symbols.  Sometimes code located outside the private context can be rewritten without using "index variables".  In other cases, the code can be relocated within the private context without any change in functionality.

A package with an escaping symbol

This is a package that exports symbols representing pitch/frequency equivalents.  Since a similar usage message definition is needed for each combination of pitch class and octave, the definitions are made in a Do loop.

!!BadNotes.m

To evaluate this package, the package is read in as an argument to EscapingSymbols.  The spurious index variable introduced ouside the private context is detected and  EscapingSymbols returns the symbol in a list.

<<EscapingSymbols`
EscapingSymbols[<<BadNotes`]

Rewriting the package to avoid escaping symbols

The defininition of usage messages can be rewritten without the use of an index variable.  Usually code that makes use of indexing can be written more clearly using functional programming.

!!Notes.m

Now  EscapingSymbols returns an empty list.

<<EscapingSymbols`
EscapingSymbols[<<Notes`]