Binarica

Fast Binary I/O for Mathematica 4.X (28 Apr 03)

Frank J. Iannarilli, Jr.

Fast Binary Read

Point to your own directory, or to the test directory where you unpacked the Binarica distribution.

SetDirectory["c:"];

Replace the filename in the following cell with your own, or use "binaricatest" below to test operation of Binarica.  
Opening stream with  DOSTextFormat -> False is critical!

stream = OpenRead["binaricatest.dat", DOSTextFormat  ->  False];

Get file length, and set stream pointer to beginning, or beyond header - application-specific.
Position to desired read starting location within file...count is in Bytes, and is index base = 0.

SetStreamPosition[stream, Infinity];

len = StreamPosition[stream]

SetStreamPosition[stream, 0];

Reads len # of Bytes from stream into linear List "data" of element type Integer.  Each byte in the file is converted to a Mathematica machine Integer (typically 4 Bytes of storage).  
However, it's NOT in PackedArray format, as can be discerned by ByteCount[].  To reduce memory usage, we'll convert to a PackedArray after the ReadList[].

data  =  ReadList[stream,  Byte, len];

Close[stream];

The Developer` context provides the test PackedArrayQ and the conversion function ToPackedArray.  It also allows access to SystemOptions.  We set the option "UnpackMessage"->True, to diagnose when/whether our operations induce any data unpacking.  Unpacking PackedArray data indicates efficiency-killing processing is occurring internally.

<< Developer`

SetSystemOptions["UnpackMessage" -> True]

PackedArrayQ[data]

Length[data]

Wow, a non-PackedArray representation uses an enormous amount of memory...much more than the putative Length[data]*(4 bytes/Integer) would imply!!

ByteCount[data]

Do This!

data = ToPackedArray[data];

PackedArrayQ[data]

That's better for memory usage, but we're still devoting a machine Integer for each Byte...we'll cut that down upon conversion to machine Reals.

ByteCount[data]

Fast Conversion of 4-byte IEEE single-precision data

Currently works only for PC-endian data!

ieee4BytesToReal[] doesn't quite handle zero-valued IEEE numbers, and returns ~5.8x10^-39.  You can get true zeros by using Chop[data,tolerance] with appropriate desired tolerance.

This function ieee4BytesToReal[] takes a linear list of machine Integers, each containing one Byte from the binary file.  We test that the length of list is a multiple of 4, since it's intended to convert a 4-byte (in the binary file) sequence representing IEEE single-precision floating point into a Mathematica (machine-precision) Real.  ieee4BytesToReal[] is careful to employ operations that maintain the PackedArray representation of the input argument, and thus maintain speed and memory efficiency.

The internal logic of function ieee4BytesToReal[] is as follows.  The argument "bytes" is a length-N list of machine Integers, each representing one Byte from the binary file.  "bytes" is first Partition[]'ed into an (N/2)x2 list, and inner-product'ed with {1,256} to convert to an (N/2)-length list of machine Integers, each now effectively containing the concatenated 16-bits of 2 Bytes. We Partition[] this result again to yield an (N/4)x2 list of machine Integers, to be operated upon by our Map[] function.  We don't attempt to stuff all 4 Bytes into one machine Integer (e.g., by taking inner product with {1,2^8,2^16,2^24}) since this would, for negative-sign bit (bit 31 in IEEE format) cause Mathematica to assume a positive integer result larger than a machine Integer, and thus spoil the PackedArray representation.

Each {Integer,Integer} element of the (N/4)x2 list is converted, based on the bits of its IEEE single-precision representation into a Mathematica Real.  This is ((-1)^(sign.bit))*(1.+normalized-mantissa.bits)*2^(excessexponent.bits-127).  
In the mantissa normalization computation below, we first convert the numerator and denominator to machine Reals - this is because there is no means of doing the right-bitshift operation (e.g., using BitShiftRight[] or dividing by Integer 2^24) in machine Integer representation without precipitating the efficiency-killing unpacking. 
Similarly for the sign determination, we must first work in machine Reals, but then the exponentiation must work in machine Integer, as a Real x^y precipitates unpacking. Fortunately, IntegerPart[] doesn't precipitate unpacking!

ieee4BytesToReal[
          bytes_?(Function[p,  
                VectorQ[p, IntegerQ] && Mod[Length[p], 4] == 0])] := 
        Map[((-1)^
                  IntegerPart[(N[BitAnd[#[[2]],  2^^1000000000000000]]/
                        N[2^15])]*(1.  + (N[
                          BitAnd[#[[2]],  
                                2^^0000000001111111]*2^16  +  #[[1]]]/
                        N[2^23]))*2^(N[
                          BitAnd[#[[2]],  2^^0111111110000000]]/128.  - 
                      127. )) &, 
          Partition[Partition[bytes,  2] . {1,  256},  
            2]
];

data = ieee4BytesToReal[data];

PackedArrayQ[data]

Dimensions[data]

Mathematica internally stores (machine-precision) Reals as Double-Precision

ByteCount[data]

Verification

Dimensions[data]

When applied in testing with the file "binaricatest" supplied with Binarica distribution, "data" should yield the following result; see if your result matches...two cells below:

binaricatestdata = {2.070941686630249`, 1.9956313371658325`, 
          0.9599212408065796`, 3.687040328979492`, 1.786059021949768`, 
          0.004340367391705513`, 1.6525468826293945`, 1.6130051612854004`, 
          3.125802516937256`, 1.500507116317749`, 1.4632216691970825`, 
          1.4031329154968262`, 0.6713268160820007`, 1.309463381767273`, 
          2.5512454509735107`, 0.6184617280960083`, 0.5936647057533264`, 
          0.5787039995193481`, 1.6874006986618042`, 1.638415813446045`, 
          0.5377838611602783`, 0.5149210095405579`, 0.9990072846412659`, 
          0.9794345498085022`, 0.4693576395511627`, 0.9261201024055481`, 
          0.4478801190853119`, 1.7618390321731567`, 0.4299958050251007`, 
          0.4154859781265259`, 0.8132661581039429`, 1.5944182872772217`, 
          0.001966889249160886`, 0.38030141592025757`, 0.745495080947876`, 
          0.3654741942882538`, 0.3576338291168213`, -0.3500820994377136`, 
          0.3428046703338623`, 
          0.6718281507492065`, -0.003313235240057111`, 
          0.6434759497642517`, 5.877471754111438`*^-39, 
          0.0031226463615894318`, 0.6040529012680054`, 
          0.5956515669822693`, -0.2901807725429535`, 
          0.28335484862327576`, 0.2812332212924957`, 0.0013966804835945368`, 
          0.8142952919006348`, 0.0013488504337146878`, 
          5.877471754111438`*^-39, -0.5183261632919312`, 
          0.2535819113254547`, 0.7494741678237915`, 0.24417363107204437`, 
          0.24029307067394257`, 0.23773016035556793`, 0.4680294692516327`, 
          0.22807681560516357`, 
          0.4551337659358978`, -0.22472389042377472`, 
          0.22041480243206024`, -0.2194446176290512`, -0.0010827253572642803`, -0.21331526339054108`, 0.6257936954498291`, 
          0.20440985262393951`, -0.0020507765002548695`, 
          0.20003379881381989`, -0.0009981440380215645`, -0.19688630104064941`, 0.3839457333087921`, 
          0.19096128642559052`, -0.0009491551900282502`, 
          0.1845521181821823`, 
          0.18236635625362396`, -0.0018327709985896945`, 
          0.17719526588916779`, 
          0.3549247980117798`, -0.17684687674045563`, -0.3480624556541443`, 5.877471754111438`*^-39, 0.16946467757225037`, 
          0.16684368252754211`, 
          0.3336080312728882`, -0.16505171358585358`, 
          0.16093800961971283`, -0.1618024855852127`, -0.1602407991886139`, -0.3175579309463501`, 
          5.877471754111438`*^-39, -0.15663693845272064`, 
          0.15133695304393768`, 
          0.1538587212562561`, -2.6875889975296`*^13, 
          2.3757035708156875`*^-36, 0.00014684813504572958`, 
          4.20220993536`*^11, 4.493998254668341`*^-37, 
          4.333400930762054`*^-37, 2.6009008128`*^10, 5.06069694507698`*^-22, 
          4.700918862429199`*^-38, -1.403068288`*^9, 
          8.112548361629362`*^-23, 5.193728617976151`*^-23, 
          1.1559299121178469`*^-38, -2.86869472`*^8, -4.77693826003995`*^38, 2.610342`*^7, 7.198905915828391`*^-9, 
          3.5925749042320376`*^-9, 
          2.2279917910594804`*^-9, -1.677959875`*^6, 
          1.43491975`*^6, -2.801750898506981`*^36, -3.5115191632446418`*^-25, -1.7370478009504785`*^-25, 
          1.4987171442651064`*^-25, 9.429301089936004`*^-11, 
          7.9860111579233`*^-11, 4.9416807819498237`*^-26, 31067.623046875`, 
          31442.123046875`, -1.1552811735402444`*^34, 
          30484.62109375`, -8.999882887756581`*^33, 
          1.7347541262387902`*^-26, 1.7933272025258458`*^-26, 
          1.1066583742327563`*^-26, 1.0119556225609738`*^-26, 
          7.486510543441849`*^-12, -4006.327880859375`, -7.913188791740911`*^33, -5.735634308166982`*^-27, -3775.01513671875`, -5826.5302734375`, -4.1678166771334844`*^33, -1.8923685351019483`*^-26, 7303.0302734375`, 
          1.6154390683985077`*^-26, -10380.8115234375`, -12823.810546875`, -15631.310546875`, -21232.12109375`, 
          2.6805684611660464`*^-26, -39636.24609375`, -3.402896621709146`*^34, 
          1.0310373388389138`*^-10, -126918.4921875`, -4.9608991256875415`*^35, -482713.9375`, 
          3.779858088764621`*^-25, -1.902581133209418`*^36, -2.7951515`*^6, -2.5433115`*^6, -1.4600254`*^7, -1.0524734`*^7, -4.797772`*^7, -4.732685015034791`*^-24, -4.37249984`*^8, 
          5.6399092`*^7, -1.58521451845766`*^38, -5.785074618093072`*^38, -1.7320807481215157`*^-23, -3.23635136`*^8, -1.9060016`*^8, -1.5525168`*^8, 2.1324412`*^7, 
          4.020812043225738`*^-23, 1.0108944000214569`*^-38, 
          5.877515895013064`*^-39, -3.610594906179949`*^-23, -7.08448128`*^8, -8.67159936`*^8, 
          3.53904576`*^8, -4.56984512`*^8, 2.3789151787401043`*^-38, 
          1.397759872`*^9, -2.555854336`*^9, 
          1.8566405597695927`*^-22, -1.394876288`*^9, 
          3.551215104`*^9, -3.190570496`*^9, -5.650838108474301`*^-22, 1.848745728`*^9};

data == binaricatestdata

Useful Snippets for Future Extension of Binarica

\(\(Here\ are\ some\ snippets\ that\ might\ be\ \(useful : 


            data\)\  = \ 
        Map[\((#1[\([1]\)]\  + \ 256*#1[\([2]\)]\ )\) &, \ 
          Partition[data, \ 2]];\)
\ \  (*\ 
      takes\ original\ Byte\ List\ "data"\ and\ makes\ 16 - 
        bit\ "words", \ 
      writing\ over\ List\ "data" . \ \ Result\ is\ half - as - 
        long\ List\ of\ \(\(words\)\(.\)\)\ \ *) 
\)
\(BaseForm[2^^0000111111111111, \ 2]
\ \ \  (*\ 
      Good\ way\ to\ check\ binary\ digit\ representation\ *) 
\)
BitAnd[data[[1]],  2^^0000111111111111]
\(\(dataAnd\  = \ 
        Map[\((BitAnd[#, \ 2^^0000111111111111])\)\  &, \ 
          data];\)
\ \ \  (*\ 
      bit - masking\ *) 
\)
IntegerDigits[#, 2, 32]
\(2^^FromDigits[#, 2]

\)
ByteCount[data];