Wolfram gridMathematica

Cluster Integration Package

Schoeller Porter, Charles Pooh, Onkar Singh

Wolfram Research, Inc.

What is the Cluster Integration Package?

A package that provides a mechanism for interfacing gridMathematica with cluster and grid management systems.

- Launch gridMathematica compute kernels through the management system rather than ssh

- The management system handles load balancing on the cluster, you do not need to know the names of the nodes.

- Allows monitoring kernel status and collecting system information.

Why Did We Create It?

- Using the mangment systems to launch gridMathematica compute kernels is easier and more robust than with ssh.

-  On many managed clusters, only software that works with the management system is allowed to run on that cluster.

- You asked us to.

Which Management Systems Do We Support?

- Version 1.0 supports Microsoft Windows Compute Cluster Server 2003. This is available now and can be downloaded (for free) from
	http://www.wolfram.com/products/gridmathematica/extras.html

- Version 1.1 will add support for Sun Grid Engine. This is in testing now, and will be available by Supercomputing 2006 (November).

- Version 1.2 will add support for Platform LSF. This is currently planned for Q1 2007.

Interfaces

The Cluster Integration Package accesses the management systems through their native interfaces (eg, not the command line)

- Compute Cluster Server: .NET/Link via CCPAPI.DLL

- Sun Grid Engine: J/Link via DRMAA

- Platform LSF: J/Link via LSF Java APIs (planned)

How We Did It

1. Reserve a number of processors from the management system.

2. Create MathLink links in the manager node, one for each processor requested.

3. Backfill the reservation with tasks to launch a kernel on each processor and connect back to the waiting links.

4. Use ConnectSlave[ ] in the Parallel Computing Toolkit to initialize the links.

What to Expect in the Future . . .

- Support for additional grid and cluster management systems.
   What would you like to see?

- Access to more information about the hardware in the cluster: memory, processor speed and utilization, platform, etc.

- Graphical tools for launching/managing kernels.

- Better handling of job queueing.

A Demo of the Cluster Integration Package

- On Microsoft Compute Cluster Server 2003 (CCS)

- On Sun Grid Engine (SGE)