File Input Data Package - Readme

Harry Bishop
Process Modeling Solutions, Inc.
26 May 2006

hrbishop@pmsi-va.com

Importing data into a notebook for manipulation and analysis is an important file task.  Mathematica comes with the Import function for doing just that.  Certain data however seems resistant to the approaches of this function, for instance part number identification with values such as 23E2300000 and 13D3400.  Import sees these values as scientific numbers with very unwanted results.  Additionally, structured data in the form of tables could be brought in with default values inserted where blank entries exist.  By using a header row and a format row before the first row of data, the FileInputDataDT package provides a means of producing a fixed length data set on each row with a specified format.

Usually a structured dataset is produced from a database.  By providing a set of tools for identifying the data columns and format of the data list the FileInputDataDT package sets up the pattern of a data list, variables for replacing the list with particular values and a function for displaying a data list matched to the format.  The package provides for txt and csv input file structures.  A set of utility functions allows convenience in checking for blank or empty data lists.

The package recognizes formats: word, date, number, serial, currency.  Word is the default format for either no format line or when the format is not recognized.  date reads the string date text and converts it into a date▫time datatype.  number reads any number.  serial reads the text string and unpads leading zeroes so that different versions of serial numbers can be compared.  currency recognizes the $ sign and reads the value as a number.  When a blank or empty value is read in the datafile the package inserts default values in the following way:
word is left blank, date is assigned the file date, serial is left blank, currency and number are assigned i.

The package PMSI`ReadDateTime` is made public to use all the date tools contained in this package, especially those related to reading text strings containing date data. This package can also be found in the Mathematica online library under Information Science and Technology > Information Technology at 
http://library.wolfram.com/infocenter/BySubject/InfoScience/InfoTech/

Loading the package FileInputDataDT

The package FileInputDataDT is placed in folder PMSI located along the filepath:  "/Applications/Mathematica 5.2.app/AddOns/Applications"/PMSI.
I use the Get request and select initialization as a cell property such as:

<< PMSI`FileInputDataDT`;

Exported functions and constants are found by using Names and each has a usage message to describe the arguments and return values.

Names["PMSI`FileInputDataDT`*"]

For questions or comments please email me at hrbishop@pmsi-va.com

Examples to use the package and possible tests

?FileInputDataDT

({{Date, Part ID, Inservice Hours, Machine, Cost, Note}, {date, serial, number, word, currency, word}, {26  May 06, 23  E400, 140.2, A, $500  .00, Purchased}, {5 Jan 2005, 023  E400, 100, A, $1  .00, Sold}, {February 4, 1854, 100  D3500, 10, B, $25  .00, Tested}})

In the above table the first row contains the title of each column.  The second row establishes the format for the data once read in.  Rows 3-5 contain data.  Because we wanted to demonstrate the full date read capability i.e. February 4, 1854 this same data would not read correctly from a csv file because the comma in the date would be interpreted as a separation of data elements. However, with appropriate data, a csv file would read correctly and FileInputDataDT would key off the .csv extension in the filename.

filePath = "/File Organizer/OR Tools/test file.txt";

{titleLine, pattVars, dataFile, fnAbbr, rowForm, vars} = 
        readDataFile[filePath, abbr -> "test"];

TableForm[Take[dataFile, {1, 3}]]

Using the setVarsToRules symbolizes the variable names and establishes the transformation rules from data list pattern to individual element.

testFileHeadings = titleLine; setVarsToRules[fnAbbr, vars];

Test variables can now be used to pick out particular elements of a data list.

testData = dataFile;

testData /. test▫PartID

displayTest = 
      displayDates[testData, test▫rowForm]

Only the displayed output is changed leaving the datafile unchanged for further analysis.

displayTest // FullForm

exportTest = 
      exportDates[testData, test▫rowForm]

The output from exportDates changes the data to be strings cementing the format as that seen in displayDates.  This allows export to textfiles for use in Excel or text editors while maintaining the database structure.

exportTest // FullForm

As a example of reading the same data with Import from a .csv file or a .txt file results  in, explosion of the part numbers in the csv case or one long string in the txt case.

trial = Import["/File Organizer/OR Tools/test file.csv"]

Import["/File Organizer/OR Tools/test file.txt"] // 
      FullForm