BigData:  Demystifying Large Datasets in Mathematica

Nick Lariviere

Wolfram Research

Outline

Fundamentals

Speed and Memory Performance

Tips and Tricks

Future Improvements

Fundamentals

What is BigData?

For the purposes of this talk “BigData” refers to primarily numerical data sets which push the limits of physical memory capacity on a single workstation.

FilePrint["file.dat"]

ReadList["ExampleData/financialtimeseries.csv",Record]//TableForm

ReadList["log.log",Record,10]//TableForm

Fundamentals

Formats & Functions

There are a number of different functions available for loading data into Mathematica, from a number of different formats.

TabView[{Style["Import",Bold,16]->Column@{Style[#,Bold,14]&@"Pros"->Column@{"user-friendly","supports many formats"}," ",Style[#,Bold,14]&@"Cons"->Column@{"high memory overhead","relatively long load times"}},Style["ReadList",Bold,16]->Column@{Style[#,Bold,14]&@"Pros"->Column@{"fast","flexible","control of types", "supports streams"}," ",Style[#,Bold,14]&@"Cons"->Column@{"not as user-friendly","works best with plain-text formats"}},Style["BinaryReadList",Bold,16]->Column@{Style[#,Bold,14]&@"Pros"->Column@{"better performance than previous two functions"}," ",Style[#,Bold,14]&@"Cons"->Column@{"generally the least user-friendly","can involve a lot of data processing"}},Style["Get(DumpSave)",Bold,16]->Column@{Style[#,Bold,14]&@"Pros"->Column@{"most efficient loading operation","preserves PackedArrays"}," ",Style[#,Bold,14]&@"Cons"->Column@{"files aren't transferable between platforms","need to be loaded by another mechanism first"}}}]

Grid[{{"Comma-Separated Values",".csv","a plain-text format that separates records with commas and new lines"},{"Tab-Separated Values",".tsv","a plain-text format that separates records with tabs and new lines"},{"Table",".dat","a plain-text format that separates records with tabs and new lines"},{"Plain Text",".txt","plain-text files can store data using arbitrary record separators"},{"Binary","*","Binary data format, can have varous or no file extension"}},Frame->All,Alignment->Left]

Speed and Memory Performance

ReadList vs Import

Quit[];

$HistoryLength=0;
file="sample.dat";
(basemem=MaxMemoryUsed[])/1024^2.;

This file is just under 2 MB, composed of 100,000 Real numbers between 0 and 10

FileByteCount[file]/1024^2.

AbsoluteTiming[ReadList[file,Real];]
(MaxMemoryUsed[]-basemem)/1024^2.

AbsoluteTiming[Import[file,"Table"];]
(MaxMemoryUsed[]-basemem)/1024^2.

Speed and Memory Performance

ReadList vs BinaryReadList

AbsoluteTiming[rldat=ReadList["rldat.dat",Real];]

AbsoluteTiming[brldat=BinaryReadList["brldat"];]

rldat==brldat

Speed and Memory Performance

There are a number of factors that effect speed and memory performance when loading data, beyond just the file-size or number of elements in a data file.  One often-overlooked element is the dimensionality of the data.

Effect of Dimensionality

This data is composed of Real numbers between 0 and 10, with 4.68 million elements in a single column:

This data is composed of Real numbers between 0 and 10, also with 4.68 million elements, but is arranged in a 23.4k by 200 matrix:

BarChart[Transpose@{{9.11,8.44},{58.78,13.50}},AxesLabel->"Timing in seconds",PlotLabel->Style["Impact of Dimensionality",Bold,16],ChartLabels->{Style[#,12]&/@{"4680k x 1","23.4k x 200"},None},ChartLegends->(Style[#,12]&/@{"ReadList","Import"})]

Tips and Tricks

Programming Helper Functions

During the initial stage of development it’s often easiest to use Import in place of ReadList or BinaryReadList, while you get a feel for the data and the application framework.  However when it comes to the deployment stage of the application you can often get a bit improvement in performance by customizing the data loading portion of your application with helper functions.

file="ExampleData/financialtimeseries.csv";
FileByteCount[FindFile@file]/1024^2.

(imp=Import[file]);//AbsoluteTiming

readTimeSeries[file_]:={First[#],ToExpression[Last[#]]}&/@ReadList[file,{Record,Record},RecordSeparators->{"\n","\r\n","\r",","}]

(rts=readTimeSeries[file]);//AbsoluteTiming

rts===imp

Tips and Tricks

Reading Part of a File

Large data files, especially those aggregated via scripts or logging systems, can sometimes take up so much room that they cannot be loaded into memory on the system all at once.  However, this doesn't mean that the data is inaccessible to Mathematica, and there are a number of methods available for reading out portions of the data from very large files.

ReadList can be used directly on large files to read in subsets of the data, if that data is at the top of the file:

ReadList[file,{Record,Record},5,RecordSeparators->{"\n","\r\n","\r",","}]

However, if that data is farther into the file, and it isn't feasible to load in all the data before the desired information, OpenRead and Find can be used to locate a particular element within the data, a date for example:

stream=OpenRead[file]

StreamPosition[stream]

Find[stream,"Jan 12"]

StreamPosition[stream]

Note that Find sets the StreamPosition to the entry immediately following the information specified, so ReadList run on the stream will start at the next record, January 13th:

ReadList[stream,{Record,Record},5,RecordSeparators->{"\n","\r\n","\r",","}]

Close[stream];

Tips and Tricks

Reloading Data

Save and DumpSave present two methods for reloading data that’s previously been loaded (or generated) in Mathematica for future kernel sessions.

ClearSystemCache[];
AbsoluteTiming[data=Import["file.dat"];]

DumpSave["temp.mx",data];
Clear[data];

ClearSystemCache[];
AbsoluteTiming[Get["temp.mx"];]

All files were generated using a PackedArray of RandomReals in a 585000x8 matrix.

Row[Panel/@{BarChart[{35.662986755371094`,214.03595733642578`,80.27977752685547`,624.9833526611328`},ChartLabels->{"DumpSave","Save","ReadList","Import"},PlotLabel->"Total Memory Overhead",AxesLabel->"Overhead in MB"],BarChart[{0.090844`5.409841242113703,20.653004`7.766528222647709,2.501283`6.849707824924688,11.789229`7.523030397234453},ChartLabels->{"DumpSave","Save","ReadList","Import"},PlotLabel->"Timing",AxesLabel->"Timing in seconds"]}]

Future Improvements

64-bit Minted Kernel

Mathematica 9 introduces a true 64-bit minted MathKernel which allows expressions in excess of the traditional 2 GB limit to be loaded into memory:

This is a 111 GB file of 15 billion pseudorandom, 64-bit, floating point numbers arranged in a 1-dimensional list.

Improvements to ReadList Performance

In addition to the 64-bit minted kernel, Mathematica 9 includes improvements made to the low-level data loading functions which improve their speed significantly over previous versions.

BarChart[Transpose[{{14.6,8.26,6.77,2.61,16.28,7.12,6.83},{5.43,2.99,4.53,1.61,5.,4.5,4.52}}],PlotLabel->Style["ReadList Timings for Version 8 and 9",Bold,18],ChartLegends->(Style[#,12]&/@{"Version 8","Version 9"}),ChartLabels->{Rotate[Style[#,12],20 Degree]&/@{"data.dat","smalldata.dat","real-0-10-234k-20-Table.dat","real-zero-585k-8-Table.dat","sci-0-1e-10-Table.dat","real-(0-1)-4680k-Table.dat","real-0-10-23k-200-Table.dat"},None},ImageSize->600,AxesLabel->Style["Timing in Seconds",12]]

Tutorial on Loading Numerical Data

Version 9 will include a detailed tutorial on different strategies for loading (primarily) numerical data, with tips on making helper functions, strategies for monitoring and measuring memory usage and timing, and advanced topics like using BinaryReadList on ASCII files.

Conclusion

Fundamentals

→ Select the right function and format for your application.

Speed and Memory Performance

→ Be mindful of factors that decrease performance, like dimensionality and $HistoryLength.

Tips and Tricks

→ Use helper-functions to improve performance in production code, and reload data using DumpSave/Get

Future Improvements

→ 64-bit minted kernel and improvements to ReadList timing should further improve performance with BigData.