version 1.0.2 Copyright 1994, Todd Gayley Original concept and program by Todd Gayley Maintained by John Fultz (jfultz@wolfram.com) This is a MathLink program for fast reading and writing of binary files. It is intended as a more or less exact replacement for the standard package Utilities`BinaryFiles`, which is slow and cannot be used under Windows. For more discussion of this program, see Volume 4, issue 2 (Spring 1994), of The Mathematica Journal. The most recent version of this source file will always be available on MathSource (anonymous ftp at mathsource.wri.com). You need Mathematica version 2.2 or later to use this program. Windows users must have at least version 2.2.2 (or they must have version 2.2.1 and request the MathLink Developer's Kit for Windows). Mathematica for DOS users are out of luck, as MathLink is not available under DOS. Todd Gayley, the original author of this program, no longer maintains FastBinaryFiles, but he has given me permission to update the code and field questions as necessary. If you have any questions, feel free to contact me at jfultz@wolfram.com. If you absolutely must get in touch with Todd, please send me any email you wish to send to him and I'll see what I can do. John Fultz jfultz@wolfram.com HOW TO BUILD THE PROGRAM ------------------------- A ready-to-run executable is available for Macintosh. There are two executables for Windows--binary22.exe works with all 2.2 versions of Mathematica for Windows, and binary30.exe works with 3.0 and later versions of Mathematica for Windows. All others will need to build the executable from the single source file binary.tm (Mac and Windows users can also compile this source on their own, of course). You should see the MathLink materials for your platform for more information on how to build template-based MathLink programs. It should be straightforward, however, to build this program. A few notes for various platforms are given below. A note for 3.0 users: Mathematica 3.0 users should look at the MathLink build instructions in the Help Browser (AddOns->MathLink Library->System-Specific Information). Macintosh You cannot use MPW, because the embedded string constants are too big for MPW C [I'm not sure if this is still true...I've shortened some of the strings. Anybody out there with MPW, feel free to tell me your experiences. -jf]. For THINK C, you will need to check the 'Separate STRS' box in the 'Project/Set Project Type' dialog. Windows The program must be built as a Win32 application (if you're running under Windows 3.1, FastBinaryFiles is completely Win32s-compatible). See the MathLink Developer's Kit for Windows for more information. I have only used Visual C++ to build it, but there should be no problems with Borland C++, Symantec C++, or any other compiler capable of creating Win32s apps (note that this is not, however, a C++ program). Unix This source file is ANSI C, so you will need an (at least mostly) ANSI-compliant compiler. The 'cc' compilers (which are called by default by the 'mcc' script) on some machines are not ANSI-compliant. Chances are good, though, that if the code compiles without errors, it will run normally. You can use the following command line to build the program: mcc binary.tm -o binary If this produces no errors, then the executable, named 'binary', is probably OK. If you get compiler errors, you may need to modify the mcc script so that it calls an ANSI C compiler, like the GNU C compiler 'gcc'. HOW TO USE IT -------------- Place the executable (named 'binary', perhaps) in a suitable directory. If you're on a Windows or Unix machine, you should choose a directory that's on your OS's PATH (or your home directory will also usually work fine on a Unix system). Then, in Mathematica, you launch it with the command In[1]:= Install["pathname"] where "pathname" is the full pathname of the program. It will take a few seconds to launch, as it sends a lot of code to Mathematica during its startup process. You are now ready to use it. You will note that a package context has been created: In[2]:= $Packages Out[2]= {FastBinaryFiles`, Global`, System`} Consult the documentation for Utilities`BinaryFiles` in the Guide to Standard Packages for specific documentation of the functions. The FastBinaryFiles` package works in essentially exactly the same way. DIFFERENCES FROM UTILITIES`BINARYFILES` --------------------------------------- There are a few minor differences in functionality between FastBinaryFiles and Utilities`BinaryFiles`. One of the most important is in the ByteOrder option, which controls whether integers are read and written with their high-order or low-order bytes first (the option values MostSignificantByteFirst and LeastSignificantByteFirst, respectively). FastBinaryFiles adds a new default value, Automatic, which automatically detects the appropriate byte order for the processor that is running the external program. Another difference concerns the options for the WriteBinary function. In Utilities`BinaryFiles`, you specify a function to perform the conversion of the data to byte form with the ByteConversion option. The default is the supplied function ToBytes. If you have a list of integers that you want written in the Int32 form, which is not the default form for integers in the ToBytes function, you need to use a line like: WriteBinary["filename", intlist, ByteConversion->(ToBytes[#, IntegerConvert->Int32]&)] To simplify the syntax, FastBinaryFiles allows you to use: WriteBinary["filename", intlist, IntegerConvert->Int32] or simply: WriteBinary["filename", intlist, Int32] In other words, all the options for ToBytes are also options for WriteBinary. In fact, my WriteBinary doesn't even use ToBytes for byte conversion (or any other function you might specify). Thus, you cannot write your own function to use with the ByteConversion option. Another important difference is in the way filenames are interpreted. When you give a filename to a built-in function like OpenRead, Mathematica uses its notion of a current directory and a $Path to locate the file. With FastBinaryFiles, the external program will not share these notions, and may even be running on a different computer with a different file system or operating system. To cope with this issue, the FastBinaryFiles code first expands the filename string you specify into a full pathname, which is what is actually passed across the link to the external program. This expansion is done in a way that attempts to mimic the behavior of the built-in functions. Under most conditions you need pay no attention to this issue. However, if you need to alter or suppress this filename expansion, you can do this with the FilenameConversion option in OpenReadBinary, OpenWriteBinary, and OpenAppendBinary. The default value for this option is Automatic, but you can specify Identity to turn off conversion, in which case the string you specify will be sent unaltered to the external program. You can also specify your own function, which must be written to take a string and return a string. An example where you would need to suppress filename conversion is when the external program is running on a computer with a different operating system than the one that is running the kernel. For example, if the kernel is running on a Macintosh, then any filenames you give will be expanded as if they were Mac filenames, which would be totally inappropriate if the external program were running under, say, Windows. Please alert me if you find any bugs, or have suggestions or comments of any kind. I can be reached by email at jfultz@wolfram.com. I hope you find FastBinaryFiles useful. I invite you to examine the source, for it contains many useful tidbits and tricks. Unfortunately, I have not had time to comment the source significantly. REVISION HISTORY ---------------- 1.0: Original release, both on MathSource and The Mathematica Journal Electronic Supplement. 1.0.1: Fixed bug in ToBytes and WriteBinary that caused Int16 and Int32 conversions to sometimes be incorrect. The Mac and Windows executables are now built with the version 2.2.2 MathLink library. 1.0.2: Made ByteOrder option applicable to Single and Double types (previously it only worked for integer types). Also fixed ReadListBinary for CString, which formerly would trigger $RecursionLimit errors by returning a nested list of depth greater than 256. Now compiles with 3.0 MathLink. Fixed bug where full pathnames wouldn't work when running under Mathematica 3.0 for Windows. Handled failed SetDirectory[] cleanly (this caused a number of SetDirectory messages to come up when opening binary files). John Fultz Front End Group Wolfram Research, Inc.