25. Files and Representations of Data

Last revision:  April 15 1998

In this section we consider files, directories, and Mathematica's operations with them.

Mathematica operations/structures introduced: 
     files
     Directory[]
     $Path
     FileNames[]
     FileDate[file]
     FileByteCount[file]
     RenameFile[file1, file2]
     CopyFile[file1, file2]
     DeleteFile[file]
     SetDirectory[]
     OpenRead[]
     OpenWrite[]
     OpendAppend[]
     InputStream[filename, number]
     OutputStream[filename, number]
     Close[fileref]
     Write[fileref, data]
     WriteString[fileref, data]
     !!filename
     Read[]
     ReadList[]

Files

Files are collections of data on a computer, typically residing on a permanent storage device such as a hard drive or a floppy disk.  Files can contain data of any data type, in any order; in this respect, they are completely general.  Their very generality, however, together with the privileges programmers enjoy in modifying them, leads us to issue a caveat or two at the outset. First, programs that read and write files usually write data in a certain form and, when reading data, expect to find it in a similar form; hence programmers must be careful when dealing with files, lest some data be read in which is completely meaningless but which will have bad consequences during a program's execution.  Similarly, programs have the power to rename, copy, and delete files, and programmers should write programs which do so only under well-defined circumstances, in order to prevent costly mistakes.  In what follows we will first insure that we know where our files will reside, then discuss how to create and alter their contents, and finally show an example of good programming practice---checking to see whether a given file is one with which we wish to work.

Directories and File Characteristics

Below we will treat the standard file operations, those of opening, closing, reading, and writing, but first we must make sure we know where the files we create will be located on our computer.  Files on computers are usually organized into directories, or folders, which are collections of files and directories.  In many respects directories are like contexts:  directories can contain (sub)directories just as contexts can contain (sub)contexts; directories hold files while contexts hold variables and functions; there is a default directory just like there is a default context; and there is a directory search path used in the same manner as a context search path.  First, let's find our current default directory, using "Directory[]".

Directory[]

Now let's find out our directory search path, "$Path".

$Path

If we ask Mathematica to read a file with a certain name, it will first look in the default directory, and, if it does not find the file there, it will proceed to the directories in "$Path", checking them one at a time.  Note that the default directory and the elements of "$Path" are all strings.

FileNames[]

"FileNames[]" instructs Mathematica to list the files in the default directory.

FileNames[]

If we don't want a complete list, but want all file names which contain certain characters, we can use wild-card expressions.  Wild-card expressions are to some extent system-dependent, but usually an expression with an asterisk ("*") in one or more places is a legal wild-card expression, denoting that in a file name,  any characters may appear in place of the asterisk.  For instance, "A*" would denote all file names beginning with "A".  Let's see if we have any such names in our default directory.

FileNames["A*"]

"FileNames[]" will even allow a list of wild-card expressions; we could ask for all file names beginning with "A" or "B" by entering

FileNames[{"A*","B*"}]

Even more, however, we can have "FileNames[]" list files in another directory, without switching default directories, simply by referring to files with their long forms (analogously to variable names' long forms).  On an IBM computer, then, we could list files at the top directory of the "C:" drive by executing

FileNames[{"C:*","C:*.*"}]

(This last list was to insure that we would get all files, whether they contained an extension or not.)

Information on files; copying, renaming, and deleting

Mathematica has many commands for determining information associated to files, as well as for changing that information.  Two of the more frequently used for determining information are 
"FileDate[file]", which gives the last-modified date of the file in the list form "{year, month, day, hour, minute, second}", and "FileByteCount[file]", which returns then number of bytes in the file, a measure of its size.  Several of the more frequently used for altering information are "CopyFile[file1, file2]", which copies file "file1" to "file2"; "RenameFile[file1,file2]" which changes the name of "file1" to "file2"; and "DeleteFile[file]", which deletes file "file".  Note that in the preceding commands, "file", "file1", and "file2" all must be strings.

Changing the default directory: Important!

To change the default directory, use the "SetDirectory[]" command.  Ask your instructor what a good place on your hard drive (or floppy disk) would be to store and read temporary files, and set your default directory to that directory.

Opening, Reading or Writing, and Closing

Opening and Closing; Stream Expressions

Before opening a file, one must decide whether the file will be opened to read data, write data, or append data.  Some computer systems allow us to do some combination of these to the same file, but we will consider only these three primitive functions.  If we open a file for reading, we will not be able to write to it (unless we close it and then open it back up); if we open a file for writing, we will erase any file that was present with the same name, create a new file of the same name, and will not be able to read from it (unless we close it and open it back up); and if we open a file for appending, we will add data on to the end of a previously-existing file and will similarly be prevented from reading data.  Note that it is important to distinguish between opening a file for writing and opening a file for appending.

The three open commands are as follows: "OpenRead[]", "OpenWrite[]", and "OpenAppend[]".  The arguments, as usual, are strings representing the file names.  After performing their functions, the three commands return an expression of the form "InputStream[filename,number]" (for reading) or "OutputStream[filename,number]".  In each case, Mathematica assigns each open file a unique number to distinguish it from other open files.  When we wish to refer to our open file later on, we will need to refer to it via its stream expression "InputStream..." or "OutputStream..."; hence we will usually assign this long expression to a variable.

Let's open a file for writing, calling it "F1".

fileref = OpenWrite["F1"]

Now, to close a file, we use the "Close[]" command, giving it as its single argument the stream expression.  In our case above, we've stored the expression in "fileref", so we can execute "Close[fileref]" to close the file.

Close[fileref]

Note:  if you forget to assign the stream expression to a variable, you cannot rectify the situation by opening a file again; Mathematica will not allow a file to be opened twice.  Hence you must close it.  What if you've forgotten the stream expression, knowing the file name but not the number?  Use "Streams[]", which provides a list of all open files (and other streams).

Now let's see if a file by the name of "F1" was really created.

FileNames[]

We have verified that "F1" exists, and now we'd like to verify that it has nothing inside.  We can do so by checking its size.

FileByteCount["F1"]

Finally, let's delete this file.

DeleteFile["F1"]

It is important to remember always to close a file after you are finished reading or writing it.  The operation of closing allows the computer system to "tidy up" details related to the file, and thus the chances of having your data damaged in the file are decreased.

Reading and Writing

Before getting started, we should agree on some terminology.  Writing to a file means that the program will send data to the file, which will then be stored on a floppy disk or hard drive.  Reading from a file means that the program will receive data from  the file, which will then be stored in variables.

We begin by opening a file to which we will write and from which we will then read.

fileref = OpenWrite["FILE1"];

To write to a file, we use the command "Write[file, ... ]" or "WriteString[file, ... ]".  In each case, the first argument is the stream expression, typically contained in a variable such as "fileref".  The succeeding arguments, separated by commas as usual, are the expressions, or names of expressions, that we wish Mathematica to write out.  The differences between "Write[]" and "WriteString[]" are twofold and important: the former writes out strings with quotation marks, and then ensures that any further data will be written on a new line in the file, while the latter writes out strings without quotation marks and allows further data to be written on the same line.  We'll perform examples of both, and check this behavior when we read the file back in below.

First, we'll write the number 1, the String "Hello", and then the result of the expression "Apply[Plus,{1,2,3}]", making sure that any further output will begin on a new line.

Write[fileref,1,"Hello",Apply[Plus,{1,2,3}]];

Without formally opening a file, there is a quick way to ask Mathematica what the entire contents of the file are:  "!!".  Execute "!!file", where "file" is the file name (without quotation marks!), and Mathematica will simply dump the contents of the file onto the screen.

!!FILE1

We notice that when we write to a file, there are no default spaces between the characters; if we want spaces, we'll have to write them out separately.

WriteString[fileref,"The answers are "];
WriteString[fileref,1];
WriteString[fileref," and "];
Write[fileref,2];

!!FILE1

Notice how important it was that the first three commands were "WriteString[]", and the last "Write[]"; in the first cases we wished to insure that the strings were written without quotation marks and the data remained on the same line, while at the end we wanted to make sure Mathematica moved to a new line after writing "2".

Now let's close the file and read its contents back in.

Close[fileref]

fileref = OpenRead["FILE1"]

In reading a file, we must decide how we want to read the text of the file back in.  For instance, if the file begins "12", is this a number, or part of a string beginning "12Hello"?  Mathematica needs to be told what sort of data to expect from the file.  We are allowed to specify several ways to convert a sequence of characters into a values of given Mathematica data types, as follows.

"Character" instructs Mathematica to read in a single character---even a blank---and put it in its own String.

"Number" instructs Mathematica to read several numeric characters, including decimal points, until it finds a blank or other non-numeric character, and then convert the string of characters into a number.  The number will be exact if no decimal point is encountered, approximate otherwise.

"Word" instructs Mathematica to read in characters until a "word separator"---typically a blank---is encountered and then to convert the sequence of characters into a String.  (The standard word separators are " " (blank) and "\t" (tab); these can be changed as an option in "ReadList[]" or "Read[]"; for more information, consult the Mathematica manual.)

"String" instructs Mathematica to read in characters until it reaches the end of a line in the file and then to convert the characters into one String.

"Byte" instructs Mathematica to read in a single byte from the file and to return its numeric value as an Integer.  Usually, the byte will be a single character, converted to its character code.

"Expression" instructs Mathematica to read in a complete Mathematica expression.

Once we've decided what to read in, we have to functions we can use to perform the read operation, "Read[]" and "ReadList[]".  Each require the the first argument be the stream expression and that the second argument be one of the conversion types listed above.  The difference is that "Read[]" reads in a single value of the conversion type, while "ReadList[]" attempts to read in several instances of the type, placing them into a list.  Without a third argument, "ReadList[]" will attempt to read in the entire file as values of the specified type; with a third argument "n", "ReadList[]" will stop after reading in "n" instances of the type.

First, let's read in a single Number, which we expect to be "1".

num = Read[fileref,Number]

Now let's read in up to three characters, which we expect to be a quotation mark, an "H", and an "e".

charlist = ReadList[fileref,Character,3]

We'll now read in the rest of the line as a String.

restOfLine = Read[fileref,String]

Our "place" in the file, which means the point at which Mathematica is keeping its finger in order to decide what data to read in next, is now at the beginning of the second line.  Let's read in the line, separated into words.  We'll use "ReadList[]" with no third argument because we're intending to continue reading words until we reach the end of the file.

finalWords = ReadList[fileref,Word]

Note that the fourth and sixth elements of the preceding list are Strings, not numbers.

Now let's close the file.

Close[fileref]

Checking That We've Got the Right File

One good programming habit, with regard to files, is always to write out some special string in the front of a file that your program may need to read back in.  In this way we can check whether or not a file, the name of which has been provided by the user, is in fact a file of the correct type for our program.  For instance, in our package "Sports`", we may wish to save the roster to a file and read it back in.  We could store the String "This file written by package Sports" in the front of the file and check for the existence of this string when we read a file back in.

Examples

Storing a List of Numbers

As our first example we show a method to write out a list of numbers into a file for easy access later.  We will define a function "storeInFile[list,file]", where the first argument is the list of numbers and the second argument is a String representing a file name.  Our function will open the file, store the data, and close the file.  Notice that the function writes out each element of the list individually, instead of writing the entire list all at once; this will permit us to read in the elements one at a time, if desired, without reading in the entire list.

Clear[storeInFile]
storeInFile[
		list_?((Select[Map[NumberQ,#],False]=={})&),
		s_String] :=
		Module[{myStream},
		   myStream = OpenWrite[s];
		   Do[
		      Write[myStream,list[[i]]],
		      {i,Length[list]}
		   ];
		   Close[myStream];
		];

We test it out with the following code.

storeInFile[{1,2,3,4,5},"MYFILE"]

!!MYFILE

Reading a List of Numbers

For our second example, we define a function to read in a list of numbers from a file.

Clear[readMyFile]
readMyFile[s_String] :=
  Module[{myStream,l},
     myStream = OpenRead[s];
     l = ReadList[myStream,Number];
     Close[myStream];
     l
  ];

We test this function out on the file "MYFILE" created above.  Again, note that if the function above had not written out the numbers individually, but instead had written the entire list all at once, our function here would get into trouble, attempting to convert "{" into a number.

readMyFile["MYFILE"]

Placing a Type Flag at the Front of the File

Finally, we'll rewrite the previous two examples, providing a String at the front of the file to give us assurance that the files we're reading are ones which we wrote.

Clear[storeInFile]
storeInFile[
		list_?((Select[Map[NumberQ,#],False]=={})&),
		s_String] :=
		Module[{myStream},
		   myStream = OpenWrite[s];
		   WriteString[myStream,"Specially created!\n"];
		   Do[
		      Write[myStream,list[[i]]],
		      {i,Length[list]}
		   ];
		   Close[myStream];
		];

Clear[readMyFile]
readMyFile[s_String] :=
  Module[{myStream,l,ss},
     myStream = OpenRead[s];
     ss = Read[myStream,String];
     If[ss=="Specially created!",
        l = ReadList[myStream,Number],
        Print["Invalid file!"]; l={}];
     Close[myStream];
     l
  ];

Note that since we were using "WriteString[]", we had to tell Mathematica explicitly to end the line in writing the special code out (using "\n"), but when we read the string back in, the end-of-line character "\n" is not contained in the resulting string ("ss" above).

On some systems we are not allowed to re-write onto a file that already exists (and sometimes the "DeleteFile[]" command does not work); hence we'll switch to "MYFILE2".

storeInFile[{1,2,3,4,5},"MYFILE2"]

!!MYFILE2

readMyFile["MYFILE2"]