Getting the Most out of (or into) Import/Export

Dale R. Horton
Wolfram Research
daleh@wolfram.com

Abstract

The Import and Export functions are simple enough for basic purposes, but more sophisticated applications require a higher degree of control. We will discuss tricks-of-the-trade, common problems, and how to modify the export of notebooks to HTML and TEX. And what do all those ConversionOptions do, anyway? In addition, we will cover recent and future additions to Import/Export and how developers can add their own converters.

Initialization

SetDirectory[
        DirectoryName@
          ToFileName["FileName" /. 
              NotebookInformation@EvaluationNotebook[]]];

Changes to Import/Export

4.0

Created.

4.1

New formats:
	STL, CSV, DXF Import, techexplorer Hypermedia Browser Export

Improvements: 
	Speedup of Table Import, RLE Compression of BMP Import and Export, Many others

4.2

New formats: 
	SDTS Import, FITS, NB, XML/NotebookML

Improvements:
	HTML Export creates XHTML, Partial CSS support for HTML Export, Added Import of 16- and 32-bit BMP,
	Error checking of options, Import/Export recognizes MIME types, Many others

Future

More formats, Improvements on existing formats
Full CSS support for HTML Export
Special flavors of TEX Export in Publicon
Many others

Advanced uses of Import/Export

Options and ConversionOptions

Options[Import]

Options[Export]

ConversionOptions takes a list of suboptions which are format-specific. Each suboption is a string, not a symbol

ConversionOptions -> {"conversionoption1" -> 
          value 1,  ... }

Descriptions of ConversionOptions (grouped according to format) can be found only in the online HelpBrowser. At the end of the entries for Import and Export, you will find closed cell group sections titled Additional Information. Click on the open/close icon to reveal the documentation.

Some useful ConversionOptions

Time savers (usually Import)

Space savers (usually Export)

Guide Interpretation

Things that don't have Mathematica equivalents

Slow Table Import

fn = 
      Export["test", 
        Table[Random[], {100}, {100}], "Table"];

old  =  First@Timing[Import[fn,  "Table"]]

slow = First@Timing[Import[fn,  "Table"]]

By saying that the data repeats every N rows (N is given by the ConversionOption "DataBlockLength"), we can improve on this greatly.

fast = 
      First@Timing[
          Import[fn,  "Table",  
            ConversionOptions -> {"DataBlockErrorChecking" -> 
                  False}]]

BEWARE! If the data does not repeat (for example, if there is a header), then the imported table may contain errors.

If you know exactly what type of data the file contains, you can always do better with ReadList.

rl = 
      First@Timing[
          ReadList[fn,  Number,  RecordLists -> True]]

MathFonts in PostScript

A common problem with PostScript graphics involve characters that use the Math fonts.

MakeEPS[file_, opts___] := 
      Export[file,  
        Plot[x, {x, 0, 1}, opts, TextStyle -> {FontSize -> 14}, 
          DisplayFunction -> Identity]]

MakeEPS[ "test1.eps", 
      PlotLabel -> "(α)"];

The problem is that "(", "α", and ")" are Math font characters. If you export this graphic and view in an application or print to a printer that does not have access to the Math fonts, then a font substitution occurs and they are replaced by other characters. There are three solutions:

1) Make the Math fonts accessible to the renderer.

http://support.wolfram.com/mathematica/graphics/export/ghostscript.html
http://support.wolfram.com/mathematica/systems/allplatforms/manualfontdownload.html

2) Remove the Math fonts from the file.

http://support.wolfram.com/mathematica/graphics/textfonts/nomathfonts.html

MakeEPS[ "test2.eps",  PlotLabel -> "(α)",  
      FormatType -> OutputForm];

But some characters cannot be represented in OutputForm (like Greek characters).

3) Embed the Math fonts in the graphic.

http://www.mathsource.com/cgi-bin/msitem?0210-519

4) Use Symbol font. Turn off OperatorSubstitution.

MakeEPS[ "test3.eps", 
        PlotLabel -> 
          StyleForm["(α)",  FontFamily -> "Symbol", 
            PrivateFontOptions -> {"OperatorSubstitution" -> 
                  False}]];

Exporting typesetting

Not only can Mathematica export graphics, in can also export typesetting.

ex = {"f", "[", "x", "]"}; 
exprs  =  Table[Take[ex, i], {i, Length[ex]}]; 
cells = 
      Map[Cell[BoxData[RowBox[#]], "Input"] &,  exprs]; 
NotebookPut[
      Notebook[cells,  WindowSize -> {FitAll, FitAll}]];

Export["typing.gif", cells,  
      ConversionOptions -> {"Loop" -> 
            True,  "AnimationDisplayTime" -> 1}];

Testing with strings

In addition to Import and Export, there are ImportString and ExportString. Among other uses, these make great tools for testing Import and Export. For example,

ExportString[{{1/5,  1/2}}, "CSV"]

str = 
      ExportString[{{1/5, 1/2}}, "CSV", 
        ConversionOptions -> {"Numeric" -> True}]

\(ImportString[str, \ "Table", \ 
      ConversionOptions -> {"TableSeparators" -> {{"
"}, {"."}}, \ "NumberPoint" -> ","}]\)

Controlling Export to HTML and TEX

nb = 
      Notebook[{Cell[
            BoxData@SuperscriptBox["2", "2"],  "Input"]}]; 
NotebookPut@nb;

Controlling Math in HTML

All are implemented in version 4.2, some implemented in 4.1

Export["file.html",nb,ConversionOptions->{"MathOutput"->form}];

"MathOutput"			Viewer						Menu Item
"GIF" (default)			Any						HTML
"DisplayForm"			Any
"InputForm"			Any
"MathML"			W3C Amaya browser				HTML+MathML
"MathViewer"			Math Viewer applet
"techexplorer"			techexplorer Hypermedia Browser plug-in

Note: "MathViewer" and "techexplorer" are unsupported. In particular, Mathematica cannot know what width and height will be rendered by the viewer. It simply makes an educated guess based on how it is rendered in Mathematica.

For complete control of "MathOutput"->"MathML", use "MathMLWrapper".

\(StringTake[
      ExportString[nb, "HTML", \ 
        ConversionOptions -> {"MathOutput" -> "MathML", \ "MathMLWrapper" -> \((StringJoin["begin(", #, ")end"] &)\)}], {353, \(-187\)}]\)

Controlling Math in TEX

Export["file.html",nb,ConversionOptions->{conversionoption}];

ConversionOption				Type of Math Output				Menu Item
defaults						LATEX2ε						TEX
"TeXVersion"->"techexplorer"			techexplorer Hypermedia Browser plug-in	techexplorer
"TeXVersion"->"LaTeX2.09"			LATEX2.09
"DisplayMathToTeX"->False			EPS files

HTML with CSS

Version 4.2 only.

Export["Css.html", nb,  
        ConversionOptions -> {"HeadElements" -> "<link \
href=\"My.css\" rel=\"stylesheet\" type=\"text/css\" />"}];

You will need to create the CSS yourself. Mathematica cell styles are converted to HTML class attributes.

!! My.css

Exporting HTML 4.0 Transitional

Version 4.2 only.

HTML export creates XHTML 1.0 Transitional. Great care was taken to make sure that browser that do not support XHTML will render these pages correctly. However, if for any reason you must use HTML 4.0 Transitional, you can use the undocumented ConversionOption "Version".

StringTake[
      ExportString[nb, "HTML",  
        ConversionOptions  ->  {"Version" -> "HTML4"}], 
      114]

MarkupRules

Fully implemented in version 4.2, partially implemented in 4.1

StringTake[ExportString[nb, "TeX"], {1068, -19}]

\(test[myrule_]\  := \ 
      StringTake[
        ExportString[nb, "TeX", \ 
          ConversionOptions -> {"MarkupRules" -> {"Input" -> myrule}}], {1068, \(-19\)}]\)

test[{"start{", "}end"}]

test[{"start{", mid, "}end"}]

mid[SuperscriptBox[x_, y_]]  :=  
      x <> " to the power " <> y; 
test[{"start{", mid, "}end"}]

See the Additional Information for HTMLSave and TeXSave for a discussion in greater detail.

ConversionRules

ConversionRules work much like MarkupRules except that they are embedded in a stylesheet and take effect automatically when that stylesheet is used. Take a look at the HelpBrowser stylesheet, for example.

Complete control

If all else fails, you can download the source files for HTML and TEX Export. Follow the instructions at http://support.wolfram.com/mathematica/interface/export/sources.html.

Once installed, you can modify the source code to your specific needs. These sources are not supported and may not work properly with all versions of Mathematica.

Creating Converters

The mechanism for adding additional formats for Import and Export currently resides in the Experimental` context and is not documented.

\(RegisterConverter[
Import | 
        Export, 
	 (*\ 
        Import\ or\ Export\ converter\ *) 
"FormatName", 
	 (*\ 
        Name\ of\ format\ *) 
{"*.extension"}, 
	 (*\ 
        String\ Pattern\ for\ filename\ *) 
"binary.exe" | "package.m" | 
        None, 
	 (*\ 
        Loaded\ during\ first\ use\ *) 
ConversionFunction, 
	 (*\ 
        Function\ that\ does\ the\ conversion\ *) 
Path -> {"directory" .. }, 
	 (*\ 
        Where\ to\ look\ for\ binaries\ or\ packages\ *) 
"FunctionChannels" -> {FileNames | 
            Streams .. }, 
	 (*\ 
        What\ types\ of\ channels\ does\ the\ ConversionFunction\ support\ *) 
"TextFormat" -> 
        True | False
	 (*\ 
        ASCII\ or\ Binary\ format\ *) 
]\)

An example,

\(Experimental`RegisterConverter[Export, "MyFormat", {"*.myext"}, 
      None, MyFunction, "FunctionChannels" -> {FileNames}, \ "TextFormat" -> True]\)

MyFunction[filename_,  data_,  
        opts___] := 
(Print[{filename,  data,  opts}]; 
        Put[ data,  filename])

Export["test.myext",  {1, 2, 3},  ImageSize -> 30,  
      ConversionOptions -> {"myopt" -> myval}]

!! test.myext

ExportString[{1, 2, 3}, "MyFormat"] // InputForm

Summary

Import and Export have easy to use interfaces when used for simple tasks, but they are not limited to such tasks. Sophisticated conversions are also possible if the correct Options and ConversionOptions are used and some common problems are avoided. ConversionOptions are particularly important when Exporting HTML and TEX, where many users require very precise tools for controlling the conversion. In addition, Import and Export are constatly evolving functions as new formats are added and existing formats are enhanced.

References