Movie Reader for Mathematica
Frame by Frame Access to Movie Images

Joshua C. Martell - University of Southern Maine
jmartell7@hotmail.com

Introduction

Mathematica supports many image formats, but no movie media formats

Sun's Java Media Framework (JMF) adds multimedia to applets

Using J/Link, we can access JMF's media capabilities to read movie files

One of the many obstacles facing digital image research is actually getting the information into Mathematica. Often data from experiments comes in as digital video clips and must be either broken into frames using a third-party software program or captured and saved as individual files.  Either way, there is more manual labor involved than there should be.  What we would like is a way to natively open the files and read the image data one frame at a time.  Sun has developed Java Media Framework (JMF) software that allows one to add movie reading and writing capabilites to your Java applets as well as grab individual frames.   MReader is written in Java and accesses these capabilites directly from Mathematica providing an easy interface that is capable of opening a number of different formats.  Functionality should continue to expand as Sun continues to develop JMF.

Introduction to Java Media Framework

JMF is an application programming interface for incorporating time-based media into Java
applications and applets.  It includes functionality to play, stream, and record a number of different media formats.

Input can come from a variety of sources.  JMF supports a number of cameras and video capture cards as well file and web based sources.  From there, the data is decoded into a format appropriate for display or convered between formats.  You could also insert a custom codec for real time filtering and effects.  Finally, the data can be presented to the screen, saved, or transmitted as appropriate.  

We use JMF to load the file and decode it into a raw format. Then we intercept the data and using controls from the player to seek and capture individual frames of video.  It would also be possible to use JMF to capture images from a digital camera or from an internet stream.

How it Works

MReader,coded in Java, provides the link between Mathematica and JMF. Once the file is opened, controler objects are obtained to seek to the desired frame and to return it as a BufferedImage.

ml = new MediaLocator(SourceString);
ds = Manager.createDataSource(ml);
p = Manager.createPlayer(ds);
p.realize();  
fpc = p.getControl("javax.media.control.FramePositioningControl");
fgc = p.getControl("javax.media.control.FrameGrabbingControl");
p.prefetch();  
fpc.seek(1);  
b = fgc.grabFrame();  
bti = new BufferToImage((VideoFormat)b.getFormat());

The FramePositioningControl allows us to seek to the desired frame number. FrameGrabbingControl allows us to capture the image data and returns it as a Buffer object.

Some formats do not support FramePositioning and or FrameGrabbing controls, most notably MPEG formats, and until the controls are added to JMF, we cannot use these formats for frame by frame processing. Quicktime and AVI formats are best supported.

Reading A File

Say you are a researcher who has been observing some physical process over time and your camera hardware has generated a movie file of the event.  Before you can do your processing on it, you need to get at the individual images.  Here is an example of how you would do that using MReader.

<< JLink`
InstallJava[];
Off[General::spell1];
AppendTo[$ExtraClassPath, "C:\\Program \
Files\\JMF2.1.1\\lib\\"];

The previous installs Java and tells Mathematica where the class file is stored.  Alternately, it can be placed with Mathematica's Java class directory, and then no setup will be necessary.
  
The following makes a new MReader Object and opens the file given. Note the URL style pathname.

movie = JavaNew["MReader"];
movie@
      MOpen["file://C:/Documents and Settings/martell/My \
Documents/111-1164_MVI.avi"]

The following line gets the 5th frame of the movie and returns it as a BufferedImage.

img = movie@MGetFrame[5]

Now we have a BufferedImage object.  The data can be extracted, or we can use Java's image processing functionality directly.  Because these are Java objects, they will exsist in memory until we call  ReleaseObject[]. Alternately, we can use a JavaBlock[] around code so that the objects will be removed from memory.  The snippet below will extract the raw image data and display it as a Mathematica graphic.

\(\(\(JavaBlock[
img = 
        movie@MGetFrame[10]; 
r = 
        img@getData[]; 
c = r@getNumBands[]; 

      w = r@getWidth[]; 
h = r@getHeight[]; 

      pix = JavaNew["[I", w*h*c]; 
tmp = 
        Reverse[Partition[
            Partition[r@getPixels[r@getMinX[], r@getMinY[], w, h, pix], c], 
            w]];
]\)\(
\)
    \)\)

Show[Graphics[Raster[tmp/255. , ColorFunction -> RGBColor]], 
      AspectRatio -> Automatic, ImageSize -> {w, h}]

A Package Solution for BufferedImage Objects

<< ImageProcessing`

img = movie@MGetFrame[5]

Show[Graphics[BufferedImageToImageData[img]]]

ReleaseObject[movie]

Methods in MReader

MReader[] - Creates MReader object

MReader[filename] - Creates an MReader object and calls MOpen[filename]

MOpen[filename] - Opens file "filename" for reading and realizes the Player, returns true if succesful

MGetFrameCount[] - returns the total number of frames

MGetFrameRate[] - returns the frame rate

MGetFrame[int i] - Returns frame i as a BufferedImage object

MGetFrame[] - Returns the next frame

MSkip[int i] - skips i frames, used in conjunction with MGetFrame[]

Summary

MReader allows you to open movie files for random or frame by frame access in Mathematica through Sun's Java Media Framework.  The returned BufferedImage format is not native to Mathematica, but it is not difficult to access that data, and a package created by Dr. Jankowski makes the translation even easier. 

JMF continues to be updated and hopefully future work will allow even more media formats to be accessable.

Appendix

Supported File Formats:
	QuickTime: 
		Cinepak
		H.261
		H.263
	AVI:
		Cinepak
	
Sun's list of supported formats. Individual frame access may not be available for all formats listed.
http://java.sun.com/products/java-media/jmf/2.1.1/formats.html	

To download JMF or for more information, visit Sun's website at:  
http://java.sun.com/products/java-media/jmf/

Acknowledgments

The author would like to thank Mariusz Jankowski for his help throughout MReader's development, Paul Nakroshis for helping with file format testing, and my Lord and Savior, Jesus Christ.

Java and Java Media Framework/JMF are trademarks of Sun Microsystems.
Quicktime is a trademark of Apple.  
Dog image was taken from a sample movie from Apple's iMovie software.