Mathematica as a Tactical Weapon in Large Corporate Warfare

Abstract

As a new employee in a large corporation (Electronic Arts), one needs to make a difference to get noticed. What better tool than Mathematica to quickly point out the mistakes your coworkers are making and separate the productive ones from the lazy bunch.

In a light-hearted style, Sarah will show how she connected Mathematica to key applications used in software development, including the source control database (Perforce), issue tracker (DevTrack), and project management tools (Microsoft Project). The Mathematica code is executed automatically every hour or every day, and the results are generated as large printable PDF documents and in the form of web pages where tooltips provide the details. This code uses the latest and future features of Mathematica.

The high-level aspects of the language enabled Sarah to focus on the satisfaction of her customers, iterating on custom graphic visualizations. Within Electronic Arts, those graphics are now defining new standards in software architecture, project scheduling, activity monitoring, and the level of separation between distinct but related development lines.

This talk will show how Mathematica is much more than a symbolic programming language, as it enables rapid prototyping, connecting to multiple information systems while demonstrating robust and superior output capabilities.

In view of the success of those graphics at Electronic Arts, Sarah now needs to hire another Mathematica programmer, so if you know anyone interested, please make sure you attend the talk!

Mathematica as a Tactical Weapon in Large Corporate Warfare!

Sarah Flannery

Worldwide CTO Group, Electronic Arts

Overview

Powerful Rapid Protyping Tool

Report Generation: Walk-through of techniques used in building a .m file to be run at regular intervals to create a report.

Examples

Start Up - Basic Constructs

Read in Data

Manipulate Data

Visualize Data

Pack in as much information as possible

Create Reports or Provide User Means to do so

Run the Reports Frequently

Send Notifications that Reports are completed

General Tricks Box

Examples

Visualizing Microsoft Project Schedules

I need the following functions to get .html files on my local machne to be opened by my web browser. The function will open any file in the application associated with the file's extension.

view[fileName_] := NotebookLocate[{URL[StringJoin["file:///", StringReplace[fileName, "\\" -> "/"]]], None}]

Making a 6.0 Button. Simple!

Button["Microsoft Project Schedule Example", view["C:\\Documents and Settings\\Sarah Flannery\\Desktop\\techconf\\MSProjectScheduleExample\\index.html"], Appearance->None]

Visualizing Game Runtime Architectures

The Button expression for the following is hidden by double-clicking on the Output cell's bracket. Delete the output label by selecting it and hitting delete.

Button["Example Project Architecture Report", view["C:\\Documents and Settings\\Sarah Flannery\\Desktop\\techconf\\ArchitectureExample\\index.html"], Appearance -> None]

Engineering Activity

An "out of the box" view: Number of Changelists by User (over a two week period)

Note that only every second row is labelled. This doesn't really tell us anything interesting and that first entry is an automated process not a person. It would be nice to remove entries like that...

Moving towards a different view - see patterns?

Shows per user checkins patterns over many days with daily team totals also.

Zooming in on that

This was a first iteration. Less than one day's work. A nice reference point to help think about how we could show more information. Note that the dots appear at the time the checkin occured. Dots on a date line show someone checking in work at midnight - not a healthy work schedule!

[removed image]

This webpage shows a verson of the above where checkins are now colored according to the type of work that was done, the volume of work and it also incorporates information from another database showing if users are building their sandboxes and running (and passing) unit tests. The page also has mathematica generated keys for the graphic, etc.

Button["Engineering Tasks Webpage 1", view["C:\\Documents and Settings\\Sarah Flannery\\engineeringprocess\\peopleP4checkins.html"], Appearance->None]

Code Development Lines

The Key: ML is Mainline, IB is Integration Branch or development line, SB is sanbox and CTG is Central Tools Group. Arrows show the direction of integration. We want to understand how close the IB is to the ML - how often are integrations happening back and forth?

A view over two weeks

Apologies that the graphics appear cropped off to the left - the top line in blue is ML, the black line represents the IB and the purple line represents many sandboxes

Zooming In...

Test Integration - Which line is the furthest behind Mainline? (Represented in red). This graphic also contains other information from a release engineering database, including information about hot fixes and when a build has been promoted to LKG or Last Known Good status.

Start Up - Basic Constructs

The bulk of what follows has been introduced through the use of examples. Here I show some code snippets and functions that I find myself using all the time.

Start: Create a notebook or just a .m (easy to use source code control to compare revisions)

Riffle the thing with meaningful Print statements - you'll need these to debug the cron job later.

Print["Talk began ", DateString[]]

For debugging, use the great new ProgressIndicator

Module[{x}, 
x=1;
PrintTemporary[ Dynamic[ProgressIndicator[x,{1,Length[set]}]] ];
 (* Do Something *)
    f& /@ set ;]

or

?Monitor

Print["Exporting and Analyzing GAME X Rig block files beginning ", DateString[]]

I need to use this function to ensure that string characters are preserved - Exporting as .m does not do the trick.

saveResult::usage="saveResult[path_,varName_,var_] saves out a copy of var to a file named \"varName.m\".
Read back in the result using var=Import[\"varName.m\"]";
saveResult[path_,varName_,var_]:=
Module[{saveFile},
saveFile=OpenWrite[ToFileName[{path}, varName<>".m"]];
WriteString[saveFile,ToString[var,InputForm]];
Close[saveFile]];

Use flags from outset to allow you to either recalculate something afresh or import a saved off value, for example. So that you can iterate on the format of the report rather than the work it does.

These flags can be set in the files (could be a txt file run by the Kernel - see later) that call the .m and then you can use constructs like this to be sure that the .m can always run:

If[Not[ValueQ[$RefreshData]], $RefreshData=False];

For Unit Tests - highlight failed tests

?$PrePrint

unitTestPrePrintFunction[True]:=Graphics[{Green, Rectangle[{0,0},{100,.5}]}, ImageSize-> 700]
unitTestPrePrintFunction[False]:=Graphics[{Red, Rectangle[{0,0},{100,10}]}, ImageSize-> 700]
unitTestPrePrintFunction[x_]:= FE`LimitOutput[x]

$PrePrint:=unitTestPrePrintFunction

Examples:

3===4

2===2

Read in Data

DatabaseLink`

Run

Maya:

Set the directory to the exectuable directory...

SetDirectory["C:\\Program Files\\Alias\\Maya6.5\\bin"]

This function will call Maya to export Maya Binary files to a XML format more convenient for analysis.

MB2RF3[{f1_,f2_,f3_}]:=Run[
StringJoin["maya.exe -batch -command \"source \\\"y:/pipeline/bin/SporeBatchExport.mel\\\"; SprBatchExport(\\\"",
f1,"\\\", \\\"",
f2,"\\\", 1, \\\"",
f3,"\\\");\" -log \"EXPORTRESULTS.TMP\" 2>&1"]]

I can sync files out of Perforce (p4) before performing some code analysis

If[$SyncToHead, 
Module[{dir},
dir=Directory[]; SetDirectory[$SporeTopLevelDirectory];
If[
Run[StringJoin["p4 sync //depot/SporeProjects/Content/Editors/RigBlocks/... > rigblockssync.txt"]]===0, 
Print["Sucessfully synced //depot/SporeProjects/Content/Editors/RigBlocks/... ", 
Length[StringSplit[Import[ToFileName[{$SporeTopLevelDirectory}, "rigblockssync.txt"]], "\n"]], " files affected"];]
SetDirectory[dir]]];

Manipulate Data

Rules are your friend!

(# -> DateString[{2006,#,1,0,0,0},{"MonthName"}])&/@Range[12]

RegularExpressions - Perl in Mathematica

StringCases["C:\\Documents and Settings\\sflannery\\SampleImage01.gif", RegularExpression["\\\\([^$\\\\]*)\\."]:> "$1"][[1]]

Visualize Data

Make use of Opacity

Many options...

<<DiscreteMath`GraphPlot`

Simple Things - still useful - doesn't all have to be custom graphics. Sometimes getting the data is the most difficult part.

[Multi ListPlot Image Removed]

Architecture Layer Diagrams

Sample Game & Scrambled Game

Development Lines Dashboard

Spark Lines example

Want to see the current value and some history of what the value has been in the recent past.

Perforce Checkins and Sandbox Builds

Microsoft Project Schedules

Pack in as much information as possible

Examples of great dashboards

Love Tooltips in 6.0

Where you flag bad patterns - email someone who should care or provide them with a button to do so. Example:

StringJoin["<a href='mailto:",#[[1]],"@ea.com?Subject=Potential Problem with ",#[[2]],".mb'>",#[[1]],"</a>"]&/@lastToSumbitItemPairs

Don't forget to add a key to the graphic

Create Reports or Provide User Means to do so

Use webMathematica

Preserve the tooltips so that you can extract pixel coordinates and create a html imagemap

This works in Beta 4

ExportWithRectangles[e_] := MathLink`CallFrontEnd[FrontEnd`ExportPacket[Cell[BoxData[ToBoxes[e]]], "MGF", Verbose -> True]]

TagForImageMap[e_, t_] := GiveSetting[FrontEnd`ImageMapRectangle[t], e];

samlpeG=Graphics[{Blue,TagForImageMap[Rectangle[{0,0},{1,1}],BlueTag],Red,TagForImageMap[Rectangle[{1,1},{2,2}],{RedTag, UniqueID}]}, ImageSize-> 20]

Rest@ExportWithRectangles[samlpeG]

This will work in the next beta

Annotation[Rectangle[{1,1},{2,2}],"MyRect","Region"]

Automatically create directories that you might need:

(If[FileType[#]=!=Directory, CreateDirectory[#]])&[ToFileName[{$ExportDirectory, "images"}]]

Save off images like this instead of a straight export (Cropping Bug Workaround)

Export[filePath,Cell[BoxData[ToBoxes@g], "Text", ShowStringCharacters->False], "GIF"]

Export pdfs for printing...

Finally a concrete example of how to construct html

Export[$ExportLocation,
  StringJoin[
   "<html><head><title>Spore Rig Blocks</title>",
(* You can even use the team's own style sheet *)
   "<link href='http://ears-wiki/sporecultwiki/css/ow.css' rel='stylesheet' type='text/css'></head>",
   "\n<body style='margin-left : .3in; margin-right : 1in;'>",
   "<h1>Spore Rig Blocks</h1>",
   "\n<p>", ToString[Length[allFilesToExportAsRF3]], " files were exported afresh for this analysis.</p>",
   "\n<p>Names highlighted in red do not appear among the list of allowed names (case sensitive) in the wiki <a href='http://maxisweb.rws.ad.ea.com/SporeCultWiki/ow.asp?AnimClipNamingConvention'>naming convention</a> pages.</p>",
   "\n<p><a href='#potentials'>", ToString[Length[Union[filesWithBadlyNamedHandles]]], " files</a> have handle names that do not correspond with their associated template. 
   Names are shown in orange when the file has no template associated with it.</p>",
   "\n<p><a href='#potentials'>", ToString[Length[Union[potentialProblemFiles]]], " files</a> have been identified as possibly not conforming to the convention for naming animations run by the engine.</p>",
   
(* here we create a bulleted index for the page that jumps to bookmarks created below. *)
   StringJoin["<p><ul>\n",
    StringJoin["<li><a href='#", StringTake[#[[1, 1]], 2], "'>", ToLowerCase[StringTake[#1[[1, 1]], 2]] /. preffixToEditorNameRules, (* " [",ToString[Length[#]],"]", *) "</a></li>\n"] & /@ lists, "</ul></p>"],
   
   "<br><br><br><br>", StringJoin["<a name='", StringTake[#[[1, 1]], 2], "'></a><h3>", ToLowerCase[StringTake[#[[1, 1]], 2]] /. preffixToEditorNameRules, "</h3>\n", makeHTMLTable[#]] & /@ lists,
   
   "\n<br><br><p align='right'>Generated ", DateString[], ". By Luc Barthelet &amp; Sarah Flannery.</p>",
   "</html>"], "Text"]

Run the Reports Frequently

Cron Jobs/Scheduled Tasks

The text file

AppendTo[$Echo, "stdout"];
$Prototyping = False;
maxDaysToView = 20;
lastDate = Date[];
Developer`UseFrontEnd[Get["c:\\CTO_Docs\\_CTO\\Docs\\SWEng\\EngineeringProcessRelated\\MARSDevLinesDashboard.m"]];
Developer`UseFrontEnd[Quit[]];

The .bat file

@ECHO OFF
p4 set P4PORT = XXXX:1666
p4 set P4CLIENT = XXXXXXXXXXXXXXX
cd "C:\Program Files\Wolfram Research\Mathematica\6.0 Prerelease"
math < "C:\\crons\GameDashboards.txt" > "C:\\crons\GameDashboards_OUT.txt"

There is Mathematica Help online for this.

Send Notifications that Reports are completed

Use JavaMail

Get["C:\\SWEng\\Utilities\\JavaMailEmail.m" ]

General Tricks Box

UnsortedUnion[l_]:=Reap[Sow[1,l],_,#1&][[2]]

view[fileName_]:=NotebookLocate[{URL[StringJoin["file:///",StringReplace[ToFileName[{Directory[]},fileName],"\\"->"/"]]], None}]

DateNormalization[date_Real, ___]:=date;
DateNormalization[date_List, initialDate_]:= N[( FromDate[date]-FromDate[initialDate])/ 86400]

Hit "Don't Create Auto Save Package" but want to change your mind?

SetOptions[InputNotebook[], AutoGeneratedPackage->True]

Questions

Print["Completed at", DateString[]]