Import and Export

P.J. Hinton
Mathematica Programming Group
Wolfram Research, Inc.

Mathematica 4 provides a facility for importing and exporting graphics, data, and text from the kernel.  This talk will survey some of the numerous options for fine-tuning interpretation and output, provide an opportunity for developers to ask questions, and offer suggestions for future supported formats.

Motivation

Despite its high degree of integration,  Mathematica is not completely self-contained system unto itself.  Users often need to transfer information to and from Mathematica.  Some examples of this include:

Possible sources of data that could be brought into Mathematica include:

Possible destinations of data from Mathematica:

Support for data interchange in past versions of Mathematica was limited.  In many cases, you had no other choice but to do some additional programming.

({{format->
operation ↓, data, graphics, sound}, {import, ReadList
roll your own, roll your own
MathSource items
limited FE capability, Miscellaneous`Audio`}, {export, roll your own
WriteMatrix, psfix/rasterps
Display/psrender
, roll your own
front end menu}})

({{Data Format, Front End, Kernel}, {BMP, Export/Import (Windows Only), Export}, {PICT, Export/Import (Macintosh Only), Export (Macintosh Only)}, {WMF/EMF, Export/Import(Windows Only), Export(Windows Only)}, {X Bitmap, None, Export}, {EPS, Export, Export}, {Adobe Illustrator, Export, Export}, {PDF, None, Export}, {JPEG, None, None}, {GIF, None, Export}, {TIFF, None, Export}, {AU, None, needed standard pkg}, {AIFF, None, needed standard pkg}, {WAV, Import/Export, needed standard pkg}, {TEX/LATEX, used Kernel function, limited Export}, {HTML, used Kernel function, limited Export}})

Goals

With Mathematica 4, we introduced a new framework for data interchange.  We aim to achieve the following goals:

Basic Syntax

We are now in the process of bringing these disparate conversion methods under a single interface that uses only two functions.

Import[channel, format]

Export[channel, expression, format]

The channel can be a filename, a stream, a list of streams, or a pipe.

If the channel is a string containing a filename with an extension that maps to a recognized format, it is not necessary to include the format argument.

Supported Formats

Variants of PostScript

({{Format, Import, Export, Documented, Conversion Method}, {EPS, ☺, ☺, ☺, \[MathematicaIcon] ExportPacket/top level code
\[KernelIcon] psrender  }, {EPSI, ☺, ☺, ☺,  psrender}, {EPSTIFF, ☺, ☺, ☺, psrender/top level code}, {EPSMETAFILE
(Windows Only), ☺, ☺, ☹, \[MathematicaIcon] Export packet/top level code
\[KernelIcon] psrender/top level code}, {PSImage, ☺, ☺, ☹,  psrender}, {PSImask, ☺, ☺, ☺,  psrender}, {PDF, ☹, ☺, ☺, psrender}, {Adobe Illustrator, ☹, ☺, ☺, psrender}})

It is possible to export graphics and typeset structures (cells and boxes) with these functions.  A front end must be present in order for typeset structures to be exported.

Importing PostScript uses an undocumented feature of psrender which translates PostScript commands into Mathematica graphics primitives.

Bitmap Graphics

({{Format, Import, Export, Documented, Method Used}, {GIF, ☺, ☺, ☺, \[MathematicaIcon] ExportPacket/MathLink binary/top level code
\[KernelIcon] psrender}, {TIFF, ☺, ☺, ☺, \[MathematicaIcon] ExportPacket/MathLink binary/top level code
\[KernelIcon] psrender}, {JPEG, ☺, ☺, ☺, ExportPacket/MathLink binary/top level code}, {Portable Pixmap
PPM, ☺, ☺, ☺, ExportPacket/MathLink binary/top level code}, {Portable Graymap
PGM, ☺, ☺, ☺, ExportPacket/MathLink binary/top level code}, {Portable Bitmap
PPM, ☺, ☺, ☺, ExportPacket/MathLink binary/top level code}, {Windows Bitmap
BMP, ☺, ☺, ☺, \[MathematicaIcon] ExportPacket/top level code
\[KernelIcon]psrender}, {X Window Bitmap
XBM, ☺, ☺, ☺, \[MathematicaIcon] ExportPacket/top level code
\[KernelIcon] psrender}, {RawBitmap, ☺, ☺, ☹, \[MathematicaIcon] ExportPacket/top level code}, {PNG, ⌚, ⌚, ☺, MathLink binary/top level code}, {MGF Cell, ☺, ☺, ☺, \[MathematicaIcon] ExportPacket/MathLink binary/top level code
\[KernelIcon] psrender}})

It is possible to export graphics and typeset structures (cells and boxes) with these functions.  A front end must be present in order for typeset structures to be exported.

If ExportPacket is listed, this means that the front end, if available, will be used to generate the rasterization of the graphic.  If psrender is listed, this means that the conversion will proceed without a front end using the psrender as the converter.

Three-Dimesional Object Description Language

({{Format, Import, Export, Documented, Method Used}, {AutoCAD DXF, ☹, ☺, ☹, top level code}})

Last summer, an intern worked on extending this feature to handle a larger subset of three-dimensional graphics features.  The extensions probably will not appear in Mathematica 4.

OS-Native Formats

({{Format, Import, Export, Documented, Conversion Method}, {PICT
(Macintosh only), ☺, ☹, ☺, \[MathematicaIcon] ExportPacket/top level code
\[KernelIcon] psrender  }, {WMF
(Windows only), ☺, ☹, ☺, \[MathematicaIcon]\[KernelIcon] psrender}})

These files are generated with the aid of proprietary operating system APIs.  It is unlikely that we will extend these formats to other platforms unless there is an overriding demand.

Sound

({{Format, Import, Export, Documented, Method Used}, {Microsoft WAV, ☺, ☺, ☺, top level code}, {AIFF, ☺, ☺, ☺, top level code}, {AU, ☺, ☺, ☺, top level code}, {SND, ☺, ☺, ☺, top level code}})

Mathematica sound objects may be exported and imported into any of these formats.  The new-and-improved Miscellaneous`Audio` package code is used.

Printer-Native Formats

({{Format, Import, Export, Documented, Method Used}, {HP PCL, ☹, ☺, ☺, psrender}, {TTY, ☹, ☺, ☹, psrender}, {Epson FX, ☹, ☺, ☹, psrender}, {Epson LQ, ☹, ☺, ☹, psrender}, {IBM Pro, ☹, ☹, ☹, psrender}, {IBM Pro 24, ☹, ☺, ☹, psrender}})

Support for these formats is also documented in the psrender manpages that ship with the Unix version of Mathematica.

Data

({{Format, Import, Export, Documented, Method Used}, {Text, ☺, ☺, ☺, top level code}, {Unicode Text, ☺, ☺, ☹, top level code}, {Words, ☺, ☺, ☹, top level code}, {Table, ☺, ☺, ☹, MathLink binary/top level code}, {HDF, ☺, ☹, ☹, MathLink binary/top level code}, {MAT, ☺, ☺, ☺, top level code}})

Numerical and textual data are supported in these format.  HDF export currently supports only the Scientific Data Set subset of the HDF format specification.

Markup Languages

({{Format, Import, Export, Documented, Method Used}, {TeX, ☹, ☺, ☺, \[MathematicaIcon]\[KernelIcon] top level code}, {HTML, ☹, ☺, ☺, \[MathematicaIcon]\[KernelIcon] top level code}})

These are just wrapper functions for TeXSave and HTMLSave.  In future releases, HTMLSave and TeXSave will cease to exist as functions in the System` context, callable only from Export.

Determining What Formats are Available

The following global session variables return a list of format names as Mathematica strings.

$ExportFormats

$ImportFormats

Options

Data Options

These options can be applied in general.

({{Option Name, Default Value}, {ByteOrdering, $ByteOrdering}, {CharacterEncoding, Automatic}, {ConversionOptions, {}}})

Graphics Options

Options that are specific to a particular format are to be given as suboptions to ConversionOptions.

({{Option Name, Default Value}, {ImageResolution, Automatic}, {ImageRotated, False}, {ImageSize, Automatic}})

Adjusting Output Through Private Options

Many formats for Import and Export can be tuned to your liking with private options.  Instead of crowding the System` context with a large number of new option names which may be of use only to one or two formats, we decided to group such options under an aggregate option named ConversionOptions.  The syntax looks like this:

ConversionOptions -> {
  "private option 1" -> value 1,
  "private option 2" -> value 2,
  "private option 3" -> value 3
}

Descriptions of private options – grouped according to data format – can be found only in the online version of the Mathematica reference guide.  At the end of the entries for Import and Export, you will find closed cell group sections titled “Additional Information.”  Click on the heading cell's group open/close icon to reveal the documentation.

Usage Examples

baseDirectory = ToFileName[{$HomeDirectory, "Conference"}];

SetDirectory[ToFileName[{baseDirectory,"Images"}]];

Importing a TIFF Image

Notice that Import is able to guess the format from the extension.

gr= Import["lenny5.tif"]

The function returns a Mathematica graphics object.

Head[gr[[1]]]

A raster primitive is used because we are dealing with a bitmap.

Show[gr,ImageSize->{194,265}, PlotRange -> {{0,194},{0,265}}];

Importing GIF Animations

Animated GIFs are imported as a list of graphics objects.

gr = Import["mwnotepd.gif"]

Rather than generating a long string of cells in the notebook, this expression generates the sequence of cells as a closed group that only needs to be double-clicked to do its work.

SelectionMove[EvaluationNotebook[], After, EvaluationCell];
NotebookWrite[
EvaluationNotebook[],
Cell[
CellGroupData[
Cell[
GraphicsData[
"PostScript",
Apply[
DisplayString[Graphics[##, ImageSize -> {80,80}, PlotRange -> {{0,80},{0,80}}]]&,
#
]
],
"Graphics",
AnimationDisplayTime -> 0.2
]&/@gr,
Closed]
]
]

It is possible to pull out indivual pieces of any animation using the ConversionOptions suboption "ImageIndex".  This example pulls out the 23rd frame in the animation.

Show[Import["mwnotepd.gif", ConversionOptions -> {"ImageIndex" -> 23}],ImageSize -> {80,80}, PlotRange -> {{0,80},{0,80}}];

It should be noted that some GIFs are generated with an optimization referred to as the delta technique.  We do not support the superimposition of incremental changes, so the result may not be what you would first expect.

gr = Import["mwcrtux.gif"];

Show[#, ImageSize -> {468,80},PlotRange ->{{0,468},{0,80}}]& /@ gr;

Importing Sound

Here we play the Mathematica startup sound.

Show[ Import["believe.wav"]]

Exporting GIFs with Special Effects

You can set specify that the GIF be interlaced or use a transparent color.

Export["special.gif",Plot3D[Sin[x]Cos[y],{x,0,2Pi},{y,0,2Pi}],ConversionOptions ->{"Interlaced" -> True, "Transparency" -> RGBColor[1,1,1]}];

Exporting JPEG with Lossy Compression

gr = Show[Graphics[{Hue[0],Disk[{0,0},5]}],AspectRatio -> 1]

Export["reddot_20.jpg", gr, "JPEG", ConversionOptions ->{ Quality ->20}]

Export["reddot_80.jpg", gr, "JPEG", ConversionOptions ->{ Quality ->80}]

{#,FileByteCount[#]}& /@ {"reddot_20.jpg","reddot_80.jpg"} //TableForm

Exporting a Bitmap of a Formula

One can supply box structures in the form of Cell and Notebook expressions to obtain bitmaps of typeset expressions.

ℒ{f(t)}(s) = (∫_0^∞)(e^(- s t))f(t) dt

cellexpr = (
NotebookLocate["LaPlace Transform"];
NotebookRead[EvaluationNotebook[]]
)

Export["laplacexform.gif",cellexpr, "GIF",  ImageSize ->2{365,45}]

The Integrator website uses a similar approach to generate GIFs of typeset expressions for results.

Exporting Data Tables

mat = Table[Random[],{4},{4}]

Export["test_table.txt", mat, "Table"]

!! test_table.txt

You can export numerical data in binary formats such as Matlab version 4 MAT format and Scientific Data Set interface HDF files.

The Table format for Import uses a variety of heuristics to determine what data types should be used to store each field.  It should not be construed as a wholesale replacement of ReadList (yet).  Instead, it should be used for small to moderate sized tables where the structure may not be regular in data type or in number of fields per record.

Underlying Architecture and File Layout

The data interchange mechanism is implemented using a combination of top-level internal code and MathLink code.

ToFileName[{$TopDirectory,"SystemFiles","Kernel","SystemResources",$SystemID},"Converters.mx"]

ToFileName[{$TopDirectory,"SystemFiles","Converters", "Binaries", $SystemID}]

The user-visible Import and Export functions are really just wrapper functions to lower-level Mathematica functions, which reside in the System`ConvertersDump` context.

The association between these higher and lower is defined with the undocumented RegisterConverter function.  This function is part of the Experimental` context, so it is liable to change in future releases.

The reason that we use MathLink binaries in our system is because we allows us to rely on several, well-established, freely distributed graphics libraries.  Formats for which libraries are used include HDF (hierarchical data format), TIFF, and JPEG.  In addition, we make use of some of the source code in the netpbm tools to handle GIF, Portable Bitmap, Portable Pixmap, and Portable Graymap formats.  We also use some custom-written C code for handling tabular data.

The libraries are statically linked, so the user does not need to worry about conflicts with any shared libraries on his or her system.

Future Development

Appendix – Getting Bitmaps of Cell and Notebook Expressions

Providing typeset resuts is done by generating bitmaps through the notebook front end.  This is a necessary evil since the kernel has no built-in methods for creating rendered versions of typeset expressions, and the front end's typesetting code is not easily componentized.  The interface for requesting bitmaps is undocumented, but it is used by HTMLSave and Export so it is likely to remain a part of the Mathematica system for some time to come.

ExportPacket - A Request for a Graphic

A bitmap can be requested from the front end using ExportPacket.  This is not a kernel function in the traditional sense.  It is just a wrapper that is sent to the front end via a MathLink connection.  The syntax is as follows.

ExportPacket[expression, format]

The packet is sent from the kernel to the front end using LinkWrite, and the result is obtained with LinkRead.

The value of expression can be any valid Cell or Notebook expression.  The format can be any of the following strings: "PostScript", "Bitmap", "MGF", "GIF", or "PPM".

The expression returned by the packet request is a three-element list.  The first element is a Mathematica string containing the graphic.  The second element is a 2 × 2 list of reals containing the high-resolution EPS bounding box values as they would appear in the header of the EPS value.  The third element is a real containing the value of the baseline.  Neither the bounding box nor the baseline values are of interest in bitmap conversions.

The packet supports an optional third argument for bitmap formats, which is a string containing a filename.  This instructs the front end to write the bitmap directly to a file on the front end's file system.  The return expression replaces the graphic in the first element with the full filename's path written as a FrontEnd`FileName object.

Care must be exercised in forming the expression that is sent with the packet.  The front end can be unpredictable about what line width it uses to compute breaks, so it is usually wise to specify an explicit value for the PageWidth option.  You may also force the front end to use a different style sheet through the StyleDefinitions option.

With the exception of "PostScript",  the default environment is determined by the setting for ScreenStyleEnvironment.  For "PostScript", PrintingStyleEnvironment is used instead.

The following options apply to all target formats.

PostScript

The "PostScript" format is essentially an Encapsulated PostScript file sans these header lines

%!PS-Adobe-2.0 EPSF-1.2
%%BoundingBox: lx ly rx ry

The following options apply specifically to this format.

Mathematica Graphics Format

The format targets "Bitmap" and "MGF" are synonymous.  They result in Mathematica system independent raster graphics.  The string contains binary data.  To use it in a Cell expression, you must first uuencode it.

The following option applies specifically to this format.

Compuserve GIF

The "GIF" target generates a binary string with the data that comprises the GIF file.  It is capable of generating only individual GIFs.

The following options applies specifically to this format.

Portable Pixmap

The "PPM" target corresponds to a binary string with the data that comprises a portable pixmap stored in RAWBITS format.  This is a 24-bit color lowest common denominator format that can be used as a starting point for conversion to other graphics formats.

Sample Code

This function will convert a Notebook or Cell expression to a GIF by having the front end write the result to a file.  The result is verified by importing the GIF back into the kernel.  This function will not work when the front end and kernel do not share the same file system.

typesetGIF[expr_Notebook|expr_Cell] :=
  Module[
    {tmpnam, retpkt, imgsz},
    tmpnam = Close[OpenTemporary[]];
    retpkt = (
      LinkWrite[
        $ParentLink, 
        ExportPacket[expr, "GIF", tmpnam]
      ]; 
      LinkRead[$ParentLink]
    );
    imgsz = 
      Map[
        Apply[Subtract,#]&, 
        Map[Reverse,Transpose[Part[retpkt, 2]]]
      ];
    Show[
      Import[tmpnam, "GIF"], 
      ImageSize -> imgsz, 
      PlotRange -> {{0, First[imgsz]}, {0,Last[imgsz]}}
    ];
    DeleteFile[tmpnam];
  ]

Appendix – Adding Conversion Functions

The mechanism for adding additional formats for Import and Export currently resides in the Experimental` context and is not documented.  It is highly likely that this interface will change for future revisions, so developers are discouraged strongly from taking advantage of this feature in products they will ship.  For your reference, the syntax for RegisterConverter is spelled out here.

RegisterConverter[
  Import|Export,
  "format name",
  {"*.extension"},
  "converter binary"|None,
  low-level function,
  "FunctionChannels" -> 
    {types of supported channels}
  "TextFormat" -> True|False 
]

Possible values of FunctionChannels list items include FileNames and Streams.  The option TextFormat indicates whether data streams should be handled in text or binary mode.