Quality Engineering with 
and in Mathematica

An inside look on how Mathematica is used to test Mathematica.

Arnoud Buzing, manager of SQA 
(Software Quality Assurance).

Wolfram Research, Inc.

Introduction

Tracking bugs

Writing test cases

Reviewing test failures

Administration of tests

Build and test timeline

6pm - Check in code revisions. Revision system (CVS) begins to update.
7pm - Revision system update complete, component builds (kernel, frontend, documentation) begin.
11pm - Component builds mostly complete, layout builds begin.
Midnight - Layout build complete, testruns begin.
10am up to mid afternoon - testruns complete

Animate[Graphics[{{RGBColor[.8,.9,1],EdgeForm[Black],Disk[]},{Black,Opacity[.7],Disk[{0,0},.03]},Table[With[{u={Sin[2 Pi i/12],Cos[2Pi i/12]}},{Text[Style[i,13],.92 u],Line[{0.97 u,u}]}],{i,1,12}],Opacity[.8],Thickness[.02],Line[{{0,0},.65 {Sin[2Pi m/(12 60)],Cos[2 Pi m/(12 60)]}}],Thickness[.01],Line[{{0,0}, .85{Sin[2Pi m/60],Cos[2 Pi m/60]}}]},ImageSize->200,PlotLabel->Style[Framed[DateString[{0,0,0,Floor[m/60],Floor[Mod[m,60]],0},{"Hour12",":","Minute"}]],14,FontFamily->"Helvetica",Background->RGBColor[1,.875,.537]],ImageMargins->8],
{m,0,12 60},DefaultDuration->10]

User interface testing

The problem

Too much manual and repetitive checks.

The challenge

Testing major components of the frontend user interface in an automated fashion.

RobotTools

The solution, part 1

Taking control of user events with the kernel and JLink

The solution, part 2

Interacting with the frontend.

RobotTools examples

MouseOver

MUnit testing package

The problem

Kernel testing package not well adapted to do frontend testing.

The solution

Testing package with better frontend integration, including stylesheet to help writing tests, report generation with and in the frontend.

Simple MUnit test examples

Needs["MUnit`"]

A successful test:

Test[2^100,1267650600228229401496703205376]

An intentional test failure for demonstration purposes:

Test[2^100,1267650600228229401496703205376-1]

Test for an expected message:

Test[1/0,ComplexInfinity,{Power::infy}]

An intentional message failure

Test[1/0,ComplexInfinity,{}]

Test with a non-standard equivalence function:

Test[Plot[Sin[x],{x,0,1}],_Graphics,EquivalenceFunction->MatchQ]

res=Test[2 3,6]

TestInput[res]

ExpectedOutput[res]

ActualOutput[res]

Using MUnit with notebooks

MouseOver

Some actual examples

MouseOver

MouseOver

MouseOver

Administration of notebook tests

The challenge

Executing thousands of notebook tests in an orderly fashion and create useful reports from the generated data.

Conclusion

New benefits

Ability to examine more test cases, in a more reliable way than before.

New issues

New user interface testing system requires maintenance and troubleshooting.