Title slide

Hello, and welcome to my talk on quality engineering with and in Mathematica. 
My name is Arnoud Buzing and I am the manager of the Software Quality Assurance department at Wolfram Research.

Introduction slide

Our department is responsible for tracking bugs and issues in Wolfram products, in particular Mathematica, writing test cases, reviewing test failures and administrating testruns on these products.

We take great pride in delivering a high quality product, but since Mathematica is a large and complex system with many components, issues do come up. We typically handle about 30 new reports a day ranging from very minor issues to major concerns. A number of reports come via the prerelease testing program, but the main chunk of reports come from internal users and automated regression testing.

Today I would like to talk about a few of our automated testing systems. I will give a brief general overview of our build and testing process and then discuss the test automation of the user interface in detail. This is particularly relevant since version six contains a serious overhaul of the user interface.

Build and test timeline slide

Perhaps one good way to demonstrate the testing process is to see what processes are happening at what time of the day. During the day, up to six o'clock in the afternoon, developers work on their projects, implementing new features and fixing issues that have been reported at some point in the past. Changes are made to the code and are checked into a revision control system called CVS, which allows us to see who made what change at what time. This also allows us to undo any mistakes and build different versions of Mathematica.

From roughly six o'clock to seven o'clock the revision control system updates itself to point to the most up to date revisions of a given source file. After this a number of build processes begin the compile the kernel, the frontend and other basic components of Mathematica, like the documentation. This process usually takes until about 11pm. At this point a layout build process is started which assembles a full version of Mathematica.

When the layout build is complete, usually about 30 minutes later, the build system signals the testing system which then begins to run kernel, frontend and documentation tests. The kernel tests are the most extensive from an automation standpoint. Every kernel is tested with over 100,000 test cases which are organized into roughly 15 test suites. Every kernel test suite addresses a particular unit in the kernel, such as Symbolics, Numerics, ImportExport and so on.

The testruns typically complete in the morning depending on the speed and number of machines that are available for a given platform. Upon completion any new failures are emailed to a relevant developer and SQA engineer for review. The review determines whether the failure constitutes a new bug introduced the day before, or if it constitutes a 'false failure' due to a documented design change.

If the failure constitutes a new bug, a new bug report is opened with the analysis of the problem. This report is then assigned to the developer responsible for the given area. The developer fixes the problem, either by reverting code changes or by refining and improving the problematic code.

If the failure is a false failure, the test is updated to conform to the new documented design. At this point the whole cycle starts again. Developers works on the code and commit their new revisions.

User Interface testing

One thing I just skipped over in describing the build/test cycle is how testing the user interface, commonly known as the 'frontend', is handled. Up to and including version 5.2 a lot of the user interface testing was done manually, typically following predetermined lists of things to check. Aside from being a fairly expensive process, because of all the manual and fairly repetetive labor involved, it was also limited in  a number of ways. People can only type and move the mouse up to a certain speed. Also given the same recipe of user interface steps to follow people tend to each do them slightly differently.

With the development of many new features in the frontend, most notably the graphics and the controls it became even more pressing to improve this manual process.

So we looked for ways to increase the speed of user interface testing as well as create a more precise user interface testing language, so that, should there be a failure, it would be wholly reprodicible. We tried a number of third party tools, such as event recording and playback software, but they tend to be highly fragile and they would limit us to testing only the most basic elements of the frontend user interface.

RobotTools

So we started to experiment with event controlling software about four years ago, in particular a java class called simply 'Robot'. We used the kernel and JLink to create an instance of this class which then allowed us to take control of the mouse and keyboard.

The initial experiments were encouraging but also revealed many limitations. To truely interact with the frontend in an intelligent way, it's not enough to just be able to control the mouse and keyboard. You also need information about where to click, drag and type. Initially there was no simple way to get this information, and only very rudimentary methods were applied. For example, for a while we used screen shots of a given notebook and analyzed it by the presence of certain background colors or font colors to drive the mouse and keyboard events. This wasn't very effective though and the prospects of this project where dim at the time.

The next breakthrough came when the user interface group provided us with a useful testing hook. This testing hook added the ability for the kernel to query the frontend about all the global locations of the typeset structures displayed in a given notebook.

Note that all the typesetting in the frontend is organized into rectangular areas on the screen. For example the numerator of a fraction is displayed in a precisely described rectangle in a notebook as is the denominator which is displayed in another rectangular area. Being able to query the frontend for this information was a quantum leap forward in the test automation of the user interface. Now we were able to click and drag in any given part of a displayed expression and manipulate it ways very much the same as a user would do.

At this point we started to have most of the tools required to automate large parts of testing we had previously done by hand. We controlled the generation of mouse and keyboard events, we knew where to direct these event in the notebooks and after issueing these events we could read in the modified parts of the notebooks to check against a given expected answer.

I will now demonstrate a few examples of interacting with the frontend with this tool.

RobotTools examples slide

[Show all examples of RobotTools in detail]

The first thing that needs to happen is to load the package, which is called RobotTools.

In this first example we want to test if pasting a snapshot of a simple Manipulate works. To do this we create a new notebook which contains the Manipulate. Using a new notebook with just one expression helps to simplify testing and improves reproducibility later on. After the notebook appears we can query the frontend about the locations of certain controls. For example the control at the top right of a Manipulate is known as a popup menu and the corresponding typeset name is PopupMenuBox. The GetBoxRectangles function is the actual kernel function which calls the frontend. The information returned by this function is always a pair of coordinates. The first coordinate is the top left position and the second coordinate is the bottom right position. With this information we can now move the mouse to a given location, for example to the center or mean of these coordinates.

The steps just described can also be combined into one mouse function with a scaling function, as shown here. After clicking in a given location we can then type certain keys to navigate the popup menu and selecting the snapshot pasting item. Finally, we make a selection in the notebook and read back the interesting parts so we can test against it. Here is the whole example when run all at once. [and again].

In this second example we just want to drag the slider of the manipulate. Note how scaling works in this example. The first slider is relevant to the dragging operation. You can give symbolic names like Left, Center which takes the mouse to the left most part of the slider and centered in the vertical direction. Or you can give any scaled coordinate running from zero to one. In this particular example the global variable 'res' is affected by any slider movement and we use this to verify that the slider did indeed move. Note in particular that nothing is checking the appearance of the manipulate. For this part the human eye is still required, although we have a small set of tests which verify appearance via screen shots.

The third example shows a few simple graphic manipulation operations such as drawing a polygon and then rotating, resizing and skewing it. This particular example also shows how you can drag the mouse outside of any zero to one scaling range as well.

MUnit testing package slide

One of the tools that was still lacking, or suboptimal, was a good package to administrate the tests. The kernel testing package could theoretically do the job, but it is highly optimized for the kernel. So a new testing package, with a project name 'MUnit', was implemented. It was designed to work in closer coordination with the frontend and works particularly well with frontend tests written in notebooks using a special testing stylesheet. Like the kernel testing package it allows for test input, an expected output, possible expected messages. It also allows for any given equivalence function. Here are some examples:

Simple MUnit test examples slide

[Examples of MUnit]

First it should be noted that the MUnit package acts very much like the kernel testing package, in the sense that it can test any Mathematica expression. This first example simply shows how to test that the one hundredth power of two matches the expected output. If we intentionally change the expected output to be wrong we get a test failure. For expressions which evaluate with a given message, like one over zero, we place the message in the third argument of the test function. This next example show an intentional message failure by omitting the expected message. Finally, one can change the default EquivalenceFunction from SameQ to something else, like MatchQ. This is useful when you want to test just for a certain expression structure, but are not interested in an exact comparion. Also note that the test result contains all relevant information about a given test.

Using MUnit with notebooks slide

[More examples of MUnit in a stylesheet]

The very same test cases can also be written directly in a notebook using the MUnit stylesheet. This stylesheet provides a number of visual feedbacks to facilitate reviewing test failures. In particular note the extra success or failure cells below a test input and output pair.

The Clear button in the docked cell removes all the result cells. The 'Test Notebook' button runs all the test cases in the notebook and generate a report to review test failures. [do it]

In particular note the 'Jump to test' buttons which take you to a given test case. This can be useful for larger test notebooks. Additionally the 'Replace ExpectedOutput with ActualOutput' button allows you to update any false test failures. [correct one]

Some actual examples slide

Now I would like to show a few actual test notebooks. This first notebook tests a number of distinct Manipulate cases taken from the documented specification and the documented examples. It tests the Bookmark menu and various controls that are possible within a Manipulate. [run it]

The second notebook tests rotating three dimensional graphics at different magnifications, and with different graphics primitives. We have similar notebooks to test three dimensional zoom and pan as well.

The third notebook tests a variety of syntax coloring bugs that have been reported since the feature was implemented.

Administration of notebook tests slide

Having the tools to test the user interface, one more system was needed to tie it all together. We quickly started to accumulate many test notebooks derived from closed bug reports and documentation. As such we needed a system that could regularly administrate these test notebooks and present the results in an orderly fashion. We settled on a very general notebook testing system which essentially runs any Mathematica command on a given set of notebooks. In fact one of the original design objectives was to test a large set of user notebooks obtained by searching the web with Google. We eventually ended up with over 20,000 user notebooks. It turns out that this is far too many to test on a daily basis, at least with the current resources.

In any case, using this system we now administer about 2,200 user interface tests on a daily basis. Like the kernel tests, these notebook tests are now being organized into test suites, that is, groups of related tests. We have tests that check the menu items, 
ones that check issues with typeset expressions, ones that check the new user interface controls and so on. 

A few areas are still lacking or still under development, in particular the testing of dialogs.

Conclusion slide

Being able to run many user interface tests in a precise and reproducible manner has proven to be very useful. By reviewing the test failures we are now able to report any new problems in an unambiguous way. We now routinely write new user interface test cases based on bugs that have been fixed. We now regularly add new test cases for new features in the frontend. These are the new benefits. Of course there are still outstanding issues to be resolved. Having a large infrastructure for user interface test automation requires a non-zero maintenance and configuration effort. This is a cost we did not have before. However, this cost is relatively small compared to the benefits of the test automation.

Thank you for your attention during this talk. Does anyone have any questions?