Using Mathematica in a scripting environment

Konstantin Kouptsov

Wolfram Research

What is a script?

• Collection of commands, which are typically entered in the command line

• Is input to a certain interpreter, typically sh, csh or bash

• Commands are executed sequentially, which makes the script a program

• Uses a (top-level) interpreter commands; does not require “write-compile-link-run” cycle

• Is a cheap way to do system programming.

Mathematica script

Mathematica script is a collection of arbitrary Mathematica commands stored in a plain text file. This saves a lot of typing if these commands are to be executed multiple times.

...
variances=Variance/@N[values];
correlation=Table[Correlation[values[[i]],values[[j]]],{i,1,Length[values]},{j,1,Length[values]}];
volatility=Sqrt[variances];
cov=Table[correlation[[j,k]]*volatility[[j]]*volatility[[k]],{j,1,Length[variances]},{k,1,Length[variances]}];
...

Mathematica kernel (math or MathKernel) is the interpreter.

As of M8, script files can be made executable and started as any other program:

#!/usr/local/bin/MathematicaScript -script

...
variances=Variance/@N[values];
correlation=Table[Correlation[values[[i]],values[[j]]],{i,1,Length[values]},{j,1,Length[values]}];
volatility=Sqrt[variances];
cov=Table[correlation[[j,k]]*volatility[[j]]*volatility[[k]],{j,1,Length[variances]},{k,1,Length[variances]}];
...

Why and how this work: see my prevous talk: http://library.wolfram.com/infocenter/Conferences/7912/

Front End

• Journal of your work

• Structured document, ready for printing

• Interactive controls

• Formula typesetting

• Plots and images,...

Why script?

• Suitable for noninteractive evaluation:

• Does not require GUI, e.g. X

• No document to typeset

• No dynamics

• No mouse interaction

• No graphics to display

• Smaller performance overhead.

Better integration within a Unix workflow

• Can be used on equal footing with other programs (scripts or binary executables)

• Called from other scripts or binaries

• exec

• Command-line arguments

• stdin, stdout, stderr

• Pipes

• Capture return code

• Process/job control

• Scripting allows using Mathematica as a language for extension of system commands (along with bash, perl, python,...)

• Scripting is available on Windows too, but Windows does not have a convenient #!-like facility.

Typical uses

• Purely computational jobs in science or industry (data mining, number crunching) - running long time and unattended

• high-performance clusters or cloud sevices

• Tasks submitted to a queue for evaluation (batch evaluation) or distributed among nodes in a cluster.

• Automation, housekeeping tasks, running test suites e.g. from job scheduler (cron).

Advantage: superb computational capabilities and high-level mathematical components such as special functions, NDSolve, Integrate/NIntegrate, built-in linear algebra, statistics,...

Drawback: noticeable kernel startup time — has little importance for large-scale long-running tasks.

Task automation - an example

A system which would need to run a computation periodically, not requiring human intervention.

E.g. keeping an eye on my investment portfolio.

It would actively analyze market trends and risks, and alert me when my attention is needed.

Scalable: may run on low-end linux box, but depending on sophistication of the algorithms may require more computing power.

Project outline:

• Run periodically every workday after market close

• Download and store quotes locally

• Recalculate the value of the portfolio

• Check conditions and alert by sending an email

Job scheduler

cron is a time-based unix job scheduler, responsible for system housekeeping tasks (compressing logs, backups, cleaning up junk in temp dir, running indexers).

Runs as a daemon (or system service) on the background – not attached to a console

As part of the operation, cron daemon can run user-specified tasks.

crontab is used to specify when and what is to be run

Job scheduler - setting up

See user’s crontab entries:

kkouptsov@math~$ crontab -l

# Edit this file to introduce tasks to be run by cron.
#
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
# and what command to run for the task
#
# To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use '*' in these fields (for 'any').#
# Notice that tasks will be started based on the cron's system
# daemon's notion of time and timezones.
#
# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
#
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
#
# For more information see the manual pages of crontab(5) and cron(8)
#
# m h  dom mon dow   command

Adding a new task

kkouptsov@math~$ crontab -e

# m h  dom mon dow   command

27 18 * * 1-5 /home/kkouptsov/bin/update

Putting ‘>/dev/null 2>&1’ at the end will redirect stdin/stdout:

kkouptsov@math~$ crontab -e

27 18 * * 1-5 /home/kkouptsov/bin/update >/dev/null 2>&1

The update script

The update script could be a Mathematica script.

But we may also want to do some more settings:

#!/bin/bash

export PATH=/home/kkouptsov/bin:/bin/:/usr/bin:/usr/local/bin
export LD_LIBRARY_PATH=/lib:/usr/lib:/usr/local/lib
 
cd /home/kkouptsov/invest
exec ./do-update

The update script–2

For a system without X windows one may encounter the following:

kkouptsov@s50:~$ math
Mathematica 8.0 for Linux x86 (32-bit)
Copyright 1988-2011 Wolfram Research, Inc.

In[1]:= Import["http://www.google.com"]

GConf Error: Failed to contact configuration server; the most common cause is a missing or misconfigured D-Bus session bus daemon. See http://projects.gnome.org/gconf/ for information. (Details -  1: Not running within active session)

Out[1]= Web Images Videos Maps News Shopping Gmail More ?

To fix this, we may want to start some additional processes:

#!/bin/bash

export PATH=/home/kkouptsov/bin:/bin/:/usr/bin:/usr/local/bin
export LD_LIBRARY_PATH=/lib:/usr/lib:/usr/local/lib

eval `dbus-launch --sh-syntax --exit-with-session`
export DBUS_SESSION_BUS_ADDRESS
export DBUS_SESSION_BUS_PID 

cd /home/kkouptsov/invest
exec ./do-update

or like this:

#!/bin/bash

export PATH=/home/kkouptsov/bin:/bin/:/usr/bin:/usr/local/bin
export LD_LIBRARY_PATH=/lib:/usr/lib:/usr/local/lib

cd /home/kkouptsov/invest
exec dbus-launch --exit-with-session ./do-update

(This is not a complete solution: it starts dbus-daemon, but leaves it after termination of the script.)

Mathematica script: do-update

#!/usr/local/Wolfram/Mathematica/8.0/SystemFiles/Kernel/Binaries/Linux/MathematicaScript -script

(* Mathematica commands *)
Print["Hello"]

Linux has a limit of 127 characters on the “interpreter line”. (Other OSes have similar limitation.)

kkouptsov@math~$ ln -s /usr/local/Wolfram/Mathematica/8.0/SystemFiles/Kernel/Binaries/Linux/MathematicaScript /usr/bin/ms

Much better:

#!/usr/bin/ms -script

(* Mathematica commands *)
Print["Hello"]

The output from Print[] commands in the script will be appended to the body of email sent by cron upon script termination.

Some error and warning messages will be appended too.

If job specification in crontab included IO redirection, Print[] command has no effect.

Sending results

An alternative to using system mail:

#!/usr/bin/ms -script

SendMail[
	"From" -> "kkouptsov@wolfram.com",
	"To" -> "kkouptsov@wolfram.com",
	"Subject" -> StringJoin["Updates for ", 
		DateString[{"MonthName"," ","Day",", ","Year"}]],
	"Body" -> ToString[1],
	"Server" -> "mail.wolfram.com",
	"UserName" -> "kkouptsov",
	"Password" -> "secret"
]

Database support

Loading packages and database communication:

#!/usr/bin/ms -script

<< DatabaseLink`

SQLConnect[] := Module[
	 {db},
	 db = OpenSQLConnection[JDBC["com.mysql.jdbc.Driver", 
     "jdbc:mysql://localhost/invest" ],
     "Username" -> "kkouptsov",
     "Password" -> "secret"]];
     db
]
     
DbTables[] := (db = SQLConnect[]; tables = SQLTableNames[db]; CloseSQLConnection[db]; tables)

(* do the calculations *)

...

(* sending results *)

SendMail[
	"From" -> "kkouptsov@wolfram.com",
	"To" -> "kkouptsov@wolfram.com",
	"Subject" -> StringJoin["Updates for ", 
		DateString[{"MonthName"," ","Day",", ","Year"}]],
	"Body" -> ToString[DbTables[]],
	"Server" -> "mail.wolfram.com",
	"UserName" -> "kkouptsov",
	"Password" -> "secret"
]

Collecting data

• FinancialData (built-in)
• “free” sites (e.g. Yahoo Finance, Google Finance, MSN Money Central, ...)
• “paid” places (Norgate Investor Services, EODData, CSI, Quotes Plus, ...)

#!/usr/bin/ms -script

...

data = NormalizeQuote[#,Flatten[{
	DateString[{"Year","-","Month","-","Day"}],
	FinancialData[#,"OHLCV"]}]]&/@sym

db = SQLConnect[];
SQLInsert[db,"quote",
	{"exchange","symbol","date","open","high","low","close","volume"},#]& /@ data;
CloseSQLConnection[db];

...

GetQuotes[sym_,{fromY_,fromM_,fromD_},{toY_,toM_,toD_}] := Block[
	{url},
	url="http://ichart.finance.yahoo.com/table.csv?s=" <> ToString[sym] <>
	"&d=" <> ToString[toD] <>
	"&e=" <> ToString[PaddedForm[toM,2,NumberPadding->"0"]] <>
	"&f=" <> ToString[toY] <>
	"&g=d&a=" <> ToString[fromD] <>
	"&b=" <> ToString[PaddedForm[fromM,2,NumberPadding->"0"]] <>
	"&c=" <> ToString[fromY] <>
	"&ignore=.csv";
	Rest[Import[url,"CSV"]]]

UpdateQuotes[symbol_] := Module[
	{data, count},
	data=GetQuotes[symbol,{2000,1,1},Take[DateList[],3]];
	db = SQLConnect[];
	SQLInsert[db,
	"quote", {"exchange","symbol","date","open","high","low","close","volume"},
		NormalizeQuote[symbol,#]]& /@ data;
	CloseSQLConnection[db];
]

...

nyseList = Import["http://www.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=NYSE&render=download","CSV"];

...

Resources for analysis

• Financial functions

• Time Series

• Probability and Statistics

• Report Generation

• Other resources, e.g. GPU

...

Conclusion

This is just a sketchy illustration of one application of Mathematica scripting.

More on writing scripts: http://reference.wolfram.com/mathematica/tutorial/MathematicaScripts.html

Previous talk explaining script internals: http://library.wolfram.com/infocenter/Conferences/7912/

Thank you.