Built-in Unit Testing

Raj Balaebail

Manager, Quality Assurance

Wolfram Research Inc.

Overview

To-do or not-to-do ?

Previous effort - MUnit with Workbench

Notebook Interface

VerificationTest

TestReport

Conclusion & Future Plans

Notebook Interface

FilePrint["/Users/rajb/Desktop/WTC_2014/Test.wlt"]

Steps

Conversion of regular notebook to a testing notebook

To convert a regular Wolfram Language notebook to a testing notebook, go to Menu -> Window -> Toolbar -> Testing

Click on the ‘Convert to Testing Notebook’ button in the toolbar

Add tests

To add more tests just type in to the notebook or use the ‘New’ button at the top left

Run tests

To run all tests in the notebook click the ‘Run’ button

To run individual tests, evaluate the input cell or the cell group

Test Details

After running a test, you could look at relevant details by clicking on the ‘Details’ dropdown on the right of the result of the test

To add expected messages or options click on the relevant button at the bottom of each test

For failed tests, you could quickly replace the existing output/messages by clicking on the ‘Replace Output/Messages’ at the bottom of each test

You could additionally rerun the test by clicking on ‘Rerun’ at the bottom of each test

Summary Bar

After running tests in the notebook a dynamic summary bar is placed at the top of the notebook

Any changes to test results upon addition, rerunning, etc., will be accurately reflected in the test summary bar

Clicking on each bar in the test summary bar will take you to the respective test

You can also order the summary bar according to order or status of results

One can quickly navigate only the failures by clicking on the chevron next to ‘Failures’ in the summary bar

More

Assign automatic test ids to all tests by clicking on the ‘Assign TestIDs’ in the ‘More’ dropdown

Save the tests in to a plain text .wlt format by clicking on the ‘Save as .wlt’ in the ‘More’ dropdown

Convert any input instantly in to a test by selecting the cell/cell-group of input cells and clicking on the ‘Convert Selection to Test’ in the ‘More’ dropdown

VerificationTest

VerificationTest[1+1,2]

VerificationTest[1+1,2]//InputForm

Part[%,1]//Dataset

VerificationTest

VerificationTest[1>0]

VerificationTest[2+2,4]

VerificationTest[1/0,ComplexInfinity,{Power::infy}]

Options

Options[VerificationTest]//Column

SameTest

VerificationTest[Plus@@Range[10],_Integer,SameTest->MatchQ]

VerificationTest[(10.^(-13)),10.^-12,SameTest->(Chop[#1-#2,(10^(-12))]==0&)]

MemoryConstraint

ByteCount[#]&/@{Range[5],{1,2,3,4,5}}

VerificationTest[Range[5],{1,2,3,4,5},MemoryConstraint->(QuantityUnit[150])]

VerificationTest[Range[5],{1,2,3,4,5},MemoryConstraint->(QuantityUnit[200])]

TestID

VerificationTest[Range[5],{1,2,3,4,5},TestID->"myTest"]

TimeConstraint

Simplify[2Sin[10x+11y+z]Cos[10x+10y+10t]]//Timing

VerificationTest[Simplify[2Sin[10x+11y+z]Cos[10x+10y+10t]],2 Cos[10 t+10x+10y] Sin[10 x+11 y+z],TimeConstraint->(QuantityUnit[3])]

VerificationTest[Simplify[2Sin[10x+11y+z]Cos[10x+10y+10t]],2 Cos[10 (t+x+y)] Sin[10 x+11 y+z],TimeConstraint->(QuantityUnit[3.5])]

Properties

test=VerificationTest[1/0,ComplexInfinity,{Power::infy}];(properties=test["Properties"])//Column

Dataset[Association[{#->test[#]}&/@properties]]

TestReport

TestReport[{VerificationTest[1+1,2],VerificationTest[1/0,ComplexInfinity]}]

TestReport["/Users/rajb/Desktop/WTC_2014/Test.wlt"]

TestReport["/Users/rajb/Desktop/WTC_2014/Test_NB.nb"]

%//InputForm

Options

Options[TestReport]//Column

SameTest

TestReport[{VerificationTest[1+1,2],VerificationTest[1.+10^-14,1]},SameTest->(Chop[#1-#2,(10^(-12))]==0&)]

MemoryConstraint

ByteCount[Range[10^2]]

TestReport[{VerificationTest[Range[10^2],_List,SameTest->MatchQ]},MemoryConstraint->(QuantityUnit[900])]

%["TestResults"]

TestReport[{VerificationTest[Range[10^2],_List,SameTest->MatchQ,MemoryConstraint->(QuantityUnit[1])]},MemoryConstraint->(QuantityUnit[900])]

TimeConstraint

TestReport[{VerificationTest[Simplify[2Sin[10x+11y+z]Cos[10x+10y+10t]],2 Cos[10 t+10x+10y] Sin[10 x+11 y+z]]},TimeConstraint->(QuantityUnit[3])]

%["TestResults"]

Properties

(properties=(report=TestReport["/Users/rajb/Desktop/WTC_2014/Test.wlt"])["Properties"])//Column

Dataset[Association[{#->report[#]}&/@properties]]

Conclusion and Future Plans

First step towards integrating unit testing in to Wolfram Language

Seamless transition from tests in a regular notebook to a testing notebook and .wlt test files

Tests that can be run anywhere, be it on Workbench, desktop, or cloud products, efficently

Backward compatibility for tests already written

Further enhance the variety of different types of supported tests and syntax

Support handling of special forms and typesetting