Implementation of a BufferedImage Translator

Mariusz Jankowski   
University of Southern Maine
mjkcc@usm.maine.edu

J/Link, a recent addition to Mathematica, promises the user an effective link to the vast functionality of the Java 2 platform.

Sun Microsystems packages for two-dimensional image processing:

- Java 2D 
	- Java Advanced Imaging (JAI)

Image representations in Java and Mathematica:

- BufferedImage (Java)
	- Raster (Mathematica)
	- ImageData (Digital Image Processing)

This talk is about a translator between the Java BufferedImage object and Mathematica's ImageData object.

The JAI provides a set of object-oriented interfaces that support a simple, high-level programming model which allows developers to manipulate images easily. There are approximately 80 operators, many with native support for optimized performance.

Examples:

- contrast enhancement
	- cropping
	- scaling
	- geometric warping
	- frequency domain processing

This loads the package and launches the Java runtime.

<< JLink`

InstallJava[]

$HistoryLength = 1;

The Digital Image Processing package introduced a convenient new Mathematica object called ImageData which encapsulates information about an image including the pixel values, the particular organization of the data and the color format.

This loads the ImageProcessing package and an example image.

<< ImageProcessing`

Here, I expose the internal structure of an ImageData expression.

(img = ImageRead["rgb.tif"]) // InputForm

The BufferedImage class introduced in JDK 1.2 plays a similar role to ImageData in the Java environment.

Here, I show how to construct a BufferedImage object. This determines the dimensions of the image, instantiates a BufferedImage object of appropriate size and data type, and populates the BufferedImage with raw data extracted from the Mathematica ImageData expression img.

{h, w} = ImageDimensions[img];
src = 
        JavaNew["java.awt.image.BufferedImage", w, h, 
          BufferedImage`TYPEUINTURGB];
r = src@getRaster[];
r@setPixels[0, 0, w, h, 
        Flatten[Reverse[
            img[[1]]]]]

Here is the equivalent code in Java syntax.

// This defines a new BufferedImage object
BufferedImage src = new BufferedImage(w, h, BufferedImage.TYPE_INT_RGB);
// This populates src with raw image data.
Raster r = src.getRaster();
r.setPixels(0, 0, w, h, imgData);

The getPixels method in Raster may be used to reverse the operation, namely return the raw pixel values to Mathematica. The original picture may be reconstructed by appropriately partitioning the list of pixel values.

r@getPixels[0, 0, w, h, JavaNew["[I", w*h*3]]

ReleaseObject[PeekObjects[]]

The BufferedImage class was introduced in JDK 1.2 and encapsulates information about an image including the pixel values, the particular organization of the data and the color format.

Raster - encapsulates image data as a rectangular grid of pixels.

DataBuffer - contains the actual raw image data (for example, DataBuffer.TYPE_FLOAT).

SampleModel - defines how the raw image data is organized in memory (for example, ComponentSampleModel).

ColorModel - is needed to interpret how the raw pixel values are mapped to a specific color.

ComponentColorModel - allows pixels to be represented by arrays of Java primitive types.

ColorSpace - defines specific color spaces such as the standard RGB space (ColorSpace.CS_sRGB) and a grayscale color space (ColorSpace.CS_GRAY).

To construct a BufferedImage object I use the JAI RasterFactory class to create compatible SampleModel, ColorModel and  DataBuffer objects. The following example illustrates the general method.

Here is an example color image.

img = 
        ImageRead["C:\\Documents and Settings\\mjkcc\\My \
Documents\\Data\\beans.tif"];

Show[Graphics[img]];

This loads some necessary classes.

LoadJavaClass["java.awt.Transparency"];
LoadJavaClass["java.awt.image.DataBuffer"];
LoadJavaClass["java.awt.color.ColorSpace"];
rf = JavaNew["javax.media.jai.RasterFactory"];

In constructing the Raster object of BufferedImage I use DataBuffer objects of type Byte, Integer, Float and Double, and ComponentSampleModel classes that permit two different interleaving methods that are equivalent to the interleaving methods found in DIP.

This creates a pixel-interleaved SampleModel for integer-valued RGB color pixels (0 ≤ r, g, b ≤ 255).

sm = 
        rf@createPixelInterleavedSampleModel[DataBuffer`TYPEUBYTE, 128, 120, 
            3];

This populates the DataBuffer with the raw image data in img.

db = 
        JavaNew["java.awt.image.DataBufferByte", 
          Flatten[ 
            Reverse[Map[Reverse, 
                img[[1]], {2}]]], 
          128*120];

In the ColorModel class hierarchy, the class specifically of interest is the ComponentColorModel class which was introduced in Java 2D and extended in JAI. It allows pixels to be represented by arrays of Java primitive types. A ColorModel is defined in part by a choice of ColorSpace. The ColorSpace class consists of the standard RGB space (ColorSpace.CS_sRGB), a grayscale color space (ColorSpace.CS_GRAY) and several other.

This creates a compatible ColorModel for integer-valued RGB color pixels (0 ≤ r, g, b ≤ 255). I use the standard RGB color space defined in Java2D.

cm = 
        rf@createComponentColorModel[DataBuffer`TYPEUBYTE, 
            ColorSpace`getInstance[ColorSpace`CSUsRGB], False, False, 
            Transparency`OPAQUE];

Together the DataBuffer and SampleModel define a Raster object.

r = rf@createWritableRaster[sm, db, Null];

Finally, the BufferedImage object is constructed from the ColorModel cm and Raster r.

src = 
      JavaNew["java.awt.image.BufferedImage", cm, r, False, 
        JavaNew["java.util.Hashtable"]]

One of the strength of Java is the relative ease with which fully portable user interfaces can be designed. The combination of Mathematica, J/Link, and Java AWT or Swing forms a very productive and powerful environment for creating custom user interfaces that either complement the Mathematica front-end, or if desired, replace it. Here is a simple example of an image viewer completely created from within Mathematica. In this example I use MathFrame and MathCanvas, J/Link subclasses of the AWT Frame and Canvas classes, respectively.

This instantiates the frame and canvas objects and sets several properties.

frm = JavaNew["com.wolfram.jlink.MathFrame"];
frm@setTitle["MDC2003"];
frm@setLocation[200, 100];
frm@setSize[300, 300];
canvas = 
        JavaNew["com.wolfram.jlink.MathCanvas"];
frm@add[canvas];

This paints the "peppers" image on the canvas.

canvas@setImage[src];

This brings the Java window to the foreground.

frm@validate[];
JavaShow[frm];

Here is an example of a simple image processing operation in a pure Mathematica (namely, Digital Image Processing) implementation.

DiscreteConvolve[img, BoxFilter[11, 11]]

This shows the original and processed images.

Show[GraphicsArray[{Graphics[img], Graphics[%]}], 
        ImageSize -> {270, 135}];

Here, I demonstrate the same operation implemented with calls to JAI.

Here, I load the JAI interface.

jai = JavaNew["javax.media.jai.JAI"];

Here I define the convolution kernel, the desired operation, and return the result.

ker = 
        JavaNew["javax.media.jai.KernelJAI", 11, 11, 
          Flatten[BoxFilter[11, 11]]];
pb = 
        JavaNew["java.awt.image.renderable.ParameterBlock"];
pb@addSource[src];
pb@add[ker];
dst = jai@create["convolve", pb];

This translates the Java object to a Mathematica expression using BufferedImageToImageData and displays the result.

Show[
        GraphicsArray[{Graphics[img], 
            Graphics[BufferedImageToImageData[dst]]}], 
        ImageSize -> {270, 135}];

Most other JAI image operators may be implemented in a similar manner.

Next, I demonstrate how to extend the border and add it to the convolution operator in the form of rendering hints.

LoadClass["javax.media.jai.BorderExtender"];
extender = 
        BorderExtender`createInstance[
          BorderExtender`BORDERUCOPY];
hints = 
        JavaNew["java.awt.RenderingHints", JAI`KEYUBORDERUEXTENDER, 
          extender];
dst = jai@create["convolve", pb, hints];

This shows the result.

Show[
        GraphicsArray[{Graphics[img], 
            Graphics[BufferedImageToImageData[dst]]}], 
        ImageSize -> {270, 135}];

The package BufferedImage.m includes several utility functions to ease the creation of scripted JAI code sequences. For example, the smoothing operation shown earlier may be defined as follows.

This creates the parameter block and rendering hints.

pb = JParameterBlock[src, {ker}];
hints = JBorderExtender["COPY"];

This performs the operation.

dst = JCreate["convolve", pb, hints];

This translates the Java object to a Mathematica expression using BufferedImageToImageData and displays the result.

Show[
        GraphicsArray[{Graphics[img], 
            Graphics[BufferedImageToImageData[dst]]}], 
        ImageSize -> {270, 135}];

It is of interest to compare execution speed of Mathematica and Java/JAI implementations of an example operation, such as convolution. I use a 512x512 grayscale image and a 3x3 filter.

This loads the example image and the filter.

img = 
        ImageRead["C:\\Documents and Settings\\Mariusz\\My \
Documents\\Data\\boats.tif"];
fir = BoxFilter[3, 3];

This returns the execution time of a pure Mathematica implementation.

t1 = (Timing[DiscreteConvolve[img, fir];] // First)/
        Second

Here, I measure the execution time of the Java implementation. Note that this includes the time needed to transfer the image data from the Mathematica to the Java side and back.

This uses the Java system clock to measure the execution time (in units of milliseconds)

LoadJavaClass["java.lang.System"];
c = System`currentTimeMillis[];
ker = 
        JavaNew["javax.media.jai.KernelJAI", 3, 3, 
          Flatten[fir]];
pb = 
        JavaNew["java.awt.image.renderable.ParameterBlock"];
pb@addSource[src = ImageDataToBufferedImage[img]];
pb@add[ker];
dst = jai@create["convolve", pb];
tmp = BufferedImageToImageData[dst]; 
    ReleaseObject[{src, pb, ker, dst}];
t2 = (System`currentTimeMillis[] - c)/1000.

This gives the ratio of the execution times.

t2/t1

I now measure the portion of this time needed to transfer the image data.

n = 20;
c = System`currentTimeMillis[];
\(\(Do[
        JavaBlock[
src = ImageDataToBufferedImage[img]; 

          tmp = BufferedImageToImageData[src]; 
          ReleaseObject[{src}]], {n}];\)\)
t3 = (System`currentTimeMillis[] - c)/1000. /N[n]

Note that the transfer time itself is greater than execution time of the Mathematica implementation!

With one exception (color representation using ICC profiles), the functionality of the JAI API is a subset of the Digital Image Processing package. Therefore the primary reason for interest in the JAI image library is the potential of performance improvement from native Java implementations of imaging algorithms. Such indeed is the case with some, but not all of the functions. Here is a tabular summary of a representative comparison. The data was measured using a 512×512 grayscale image on a 1.2 GHz Pentium III (mobile) computer (all numbers in seconds):

(Note: JAI timing data does not include the time needed to transfer the image data from the Mathematica to the Java side and back which was measured at approximately 360 mseconds. )

The talk presented details of a recently developed bidirectional translator between a DIP ImageData object and a Java BufferedImage object. This translator was developed as part of the next version of the Digital Image Processing package.