Copyright (C) 1997 Rich Neidinger, John Swallow, and Todd Will.  Free for
distribution to college and university instructors for personal,
non-commercial use.  If these notebooks are used in a course, the authors
request $20 per student.

25. Files and Representations of Data

Last revision:  November 26 1996

In this section we consider files, directories, and Mathematica's
operations with them.

Files

Files are collections of data on a computer, typically residing on a
permanent storage device such as a hard drive or a floppy disk.  Files can
contain data of any data type, in any order; in this respect, they are
completely general.  Their very generality, however, together with the
privileges programmers enjoy in modifying them, leads us to issue a caveat
or two at the outset. First, programs that read and write files usually
write data in a certain form and, when reading data, expect to find it in a
similar form; hence programmers must be careful when dealing with files,
lest some data be read in which is completely meaningless but which will
have bad consequences during a program's execution.  Similarly, programs
have the power to rename, copy, and delete files, and programmers should
write programs which do so only under well-defined circumstances, in order
to prevent costly mistakes.  In what follows we will first insure that we
know where our files will reside, then discuss how to create and alter
their contents, and finally show an example of good programming
practice---checking to see whether a given file is one with which we wish
to work.

Directories and File Characteristics

Below we will treat the standard file operations, those of opening,
closing, reading, and writing, but first we must make sure we know where
the files we create will be located on our computer.  Files on computers
are usually organized into directories, or folders, which are collections
of files and directories.  In many respects directories are like contexts:
directories can contain (sub)directories just as contexts can contain
(sub)contexts; directories hold files while contexts hold variables and
functions; there is a default directory just like there is a default
context; and there is a directory search path used in the same manner as a
context search path.  First, let's find our current default directory,
using "Directory[ ]".

Directory[]

Now let's find out our directory search path, "$Path".

$Path

If we ask Mathematica to read a file with a certain name, it will first
look in the default directory, and, if it does not find the file there, it
will proceed to the directories in "$Path", checking them one at a time.
Note that the default directory and the elements of "$Path" are all
strings.

FileNames[]

"FileNames[ ]" instructs Mathematica to list the files in the default directory.

FileNames[]

If we don't want a complete list, but want all file names which contain
certain characters, we can use wild-card expressions.  Wild-card
expressions are to some extent system-dependent, but usually an expression
with an asterisk ("*") in one or more places is a legal wild-card
expression, denoting that in a file name,  any characters may appear in
place of the asterisk.  For instance, "A*" would denote all file names
beginning with "A".  Let's see if we have any such names in our default
directory.

FileNames["A*"]

"FileNames[ ]" will even allow a list of wild-card expressions; we could ask for all file names beginning with "A" or "B" by entering

FileNames[{"A*","B*"}]

Even more, however, we can have "FileNames[ ]" list files in another
directory, without switching default directories, simply by referring to
files with their long forms (analogously to variable names' long forms).
On an IBM computer, then, we could list files at the top directory of the
"C:" drive by executing

FileNames[{"C:*","C:*.*"}]

(This last list was to insure that we would get all files, whether they
contained an extension or not.)

Information on files; copying, renaming, and deleting

Mathematica has many commands for determining information associated to
files, as well as for changing that information.  Two of the more
frequently used for determining information are
"FileDate[file]", which gives the last-modified date of the file in the
list form "{year, month, day, hour, minute, second}", and
"FileByteCount[file]", which returns then number of bytes in the file, a
measure of its size.  Several of the more frequently used for altering
information are "CopyFile[file1, file2]", which copies file "file1" to
"file2"; "RenameFile[file1,file2]" which changes the name of "file1" to
"file2"; and "DeleteFile[file]", which deletes file "file".  Note that in
the preceding commands, "file", "file1", and "file2" all must be strings.

Changing the default directory: Important!

To change the default directory, use the "SetDirectory[ ]" command.  Ask
your instructor what a good place on your hard drive (or floppy disk) would
be to store and read temporary files, and set your default directory to
that directory.

Opening, Reading or Writing, and Closing

Opening and Closing; Stream Expressions

Before opening a file, one must decide whether the file will be opened to
read data, write data, or append data.  Some computer systems allow us to
do some combination of these to the same file, but we will consider only
these three primitive functions.  If we open a file for reading, we will
not be able to write to it (unless we close it and then open it back up);
if we open a file for writing, we will erase any file that was present with
the same name, create a new file of the same name, and will not be able to
read from it (unless we close it and open it back up); and if we open a
file for appending, we will add data on to the end of a previously-existing
file and will similarly be prevented from reading data.  Note that it is
important to distinguish between opening a file for writing and opening a
file for appending.

The three open commands are as follows: "OpenRead[ ]", "OpenWrite[ ]", and
"OpenAppend[ ]".  The arguments, as usual, are strings representing the
file names.  After performing their functions, the three commands return an
expression of the form "InputStream[filename,number]" (for reading) or
"OutputStream[filename,number]".  In each case, Mathematica assigns each
open file a unique number to distinguish it from other open files.  When we
wish to refer to our open file later on, we will need to refer to it via
its stream expression "InputStream..." or "OutputStream..."; hence we will
usually assign this long expression to a variable.

Let's open a file for writing, calling it "F1".

fileref = OpenWrite["F1"]

Now, to close a file, we use the "Close[ ]" command, giving it as its
single argument the stream expression.  In our case above, we've stored the
expression in "fileref", so we can execute "Close[fileref]" to close the
file.

Close[fileref]

Note:  if you forget to assign the stream expression to a variable, you
cannot rectify the situation by opening a file again; Mathematica will not
allow a file to be opened twice.  Hence you must close it.  What if you've
forgotten the stream expression, knowing the file name but not the number?
Use "Streams[ ]", which provides a list of all open files (and other
streams).

Now let's see if a file by the name of "F1" was really created.

FileNames[]

We have verified that "F1" exists, and now we'd like to verify that it has
nothing inside.  We can do so by checking its size.

FileByteCount["F1"]

Finally, let's delete this file.

DeleteFile["F1"]

It is important to remember always to close a file after you are finished
reading or writing it.  The operation of closing allows the computer system
to "tidy up" details related to the file, and thus the chances of having
your data damaged in the file are decreased.

Reading and Writing

Before getting started, we should agree on some terminology.  Writing to a
file means that the program will send data to the file, which will then be
stored on a floppy disk or hard drive.  Reading from a file means that the
program will receive data from  the file, which will then be stored in
variables.

We begin by opening a file to which we will write and from which we will
then read.

fileref = OpenWrite["FILE1"];

To write to a file, we use the command "Write[file, ... ]" or
"WriteString[file, ... ]".  In each case, the first argument is the stream
expression, typically contained in a variable such as "fileref".  The
succeeding arguments, separated by commas as usual, are the expressions, or
names of expressions, that we wish Mathematica to write out.  The
differences between "Write[ ]" and "WriteString[ ]" are twofold and
important: the former writes out strings with quotation marks, and then
ensures that any further data will be written on a new line in the file,
while the latter writes out strings without quotation marks and allows
further data to be written on the same line.  We'll perform examples of
both, and check this behavior when we read the file back in below.

First, we'll write the number 1, the String "Hello", and then the result of
the expression "Apply[Plus,{1,2,3}]", making sure that any further output
will begin on a new line.

Write[fileref,1,"Hello",Apply[Plus,{1,2,3}]];

Without formally opening a file, there is a quick way to ask Mathematica
what the entire contents of the file are:  "!!".  Execute "!!file", where
"file" is the file name (without quotation marks!), and Mathematica will
simply dump the contents of the file onto the screen.

!!FILE1

We notice that when we write to a file, there are no default spaces between
the characters; if we want spaces, we'll have to write them out separately.

WriteString[fileref,"The answers are "];
WriteString[fileref,1];
WriteString[fileref," and "];
Write[fileref,2];

!!FILE1

Notice how important it was that the first three commands were
"WriteString[ ]", and the last "Write[ ]"; in the first cases we wished to
insure that the strings were written without quotation marks and the data
remained on the same line, while at the end we wanted to make sure
Mathematica moved to a new line after writing "2".

Now let's close the file and read its contents back in.

Close[fileref]

fileref = OpenRead["FILE1"]

In reading a file, we must decide how we want to read the text of the file
back in.  For instance, if the file begins "12", is this a number, or part
of a string beginning "12Hello"?  Mathematica needs to be told what sort of
data to expect from the file.  We are allowed to specify several ways to
convert a sequence of characters into a values of given Mathematica data
types, as follows.

"Character" instructs Mathematica to read in a single character---even a
blank---and put it in its own String.

"Number" instructs Mathematica to read several numeric characters,
including decimal points, until it finds a blank or other non-numeric
character, and then convert the string of characters into a number.  The
number will be exact if no decimal point is encountered, approximate
otherwise.

"Word" instructs Mathematica to read in characters until a "word
separator"---typically a blank---is encountered and then to convert the
sequence of characters into a String.  (The standard word separators are "
" (blank) and "\t" (tab); these can be changed as an option in "ReadList[
]" or "Read[ ]"; for more information, consult the Mathematica manual.)

"String" instructs Mathematica to read in characters until it reaches the
end of a line in the file and then to convert the characters into one
String.

"Byte" instructs Mathematica to read in a single byte from the file and to
return its numeric value as an Integer.  Usually, the byte will be a single
character, converted to its character code.

"Expression" instructs Mathematica to read in a complete Mathematica expression.

Once we've decided what to read in, we have to functions we can use to
perform the read operation, "Read[ ]" and "ReadList[ ]".  Each require the
the first argument be the stream expression and that the second argument be
one of the conversion types listed above.  The difference is that "Read[ ]"
reads in a single value of the conversion type, while "ReadList[ ]"
attempts to read in several instances of the type, placing them into a
list.  Without a third argument, "ReadList[ ]" will attempt to read in the
entire file as values of the specified type; with a third argument "n",
"ReadList[ ]" will stop after reading in "n" instances of the type.

First, let's read in a single Number, which we expect to be "1".

num = Read[fileref,Number]

Now let's read in up to three characters, which we expect to be a quotation
mark, an "H", and an "e".

charlist = ReadList[fileref,Character,3]

We'll now read in the rest of the line as a String.

restOfLine = Read[fileref,String]

Our "place" in the file, which means the point at which Mathematica is
keeping its finger in order to decide what data to read in next, is now at
the beginning of the second line.  Let's read in the line, separated into
words.  We'll use "ReadList[ ]" with no third argument because we're
intending to continue reading words until we reach the end of the file.

finalWords = ReadList[fileref,Word]

Note that the fourth and sixth elements of the preceding list are Strings,
not numbers.

Now let's close the file.

Close[fileref]

Checking That We've Got the Right File

One good programming habit, with regard to files, is always to write out
some special string in the front of a file that your program may need to
read back in.  In this way we can check whether or not a file, the name of
which has been provided by the user, is in fact a file of the correct type
for our program.  For instance, in our package "Sports`", we may wish to
save the roster to a file and read it back in.  We could store the String
"This file written by package Sports" in the front of the file and check
for the existence of this string when we read a file back in.

Examples

Storing a List of Numbers

As our first example we show a method to write out a list of numbers into a
file for easy access later.  We will define a function
"storeInFile[list,file]", where the first argument is the list of numbers
and the second argument is a String representing a file name.  Our function
will open the file, store the data, and close the file.  Notice that the
function writes out each element of the list individually, instead of
writing the entire list all at once; this will permit us to read in the
elements one at a time, if desired, without reading in the entire list.

Clear[storeInFile]
storeInFile[
                list_?((Select[Map[NumberQ,#],False]=={})&),
                s_String] :=
                Module[{myStream},
                   myStream = OpenWrite[s];
                   Do[
                      Write[myStream,list[[i]]],
                      {i,Length[list]}
                   ];
                   Close[myStream];
                ];

We test it out with the following code.

storeInFile[{1,2,3,4,5},"MYFILE"]

!!MYFILE

Reading a List of Numbers

For our second example, we define a function to read in a list of numbers
from a file.

Clear[readMyFile]
readMyFile[s_String] :=
  Module[{myStream,l},
     myStream = OpenRead[s];
     l = ReadList[myStream,Number];
     Close[myStream];
     l
  ];

We test this function out on the file "MYFILE" created above.  Again, note
that if the function above had not written out the numbers individually,
but instead had written the entire list all at once, our function here
would get into trouble, attempting to convert "{" into a number.

readMyFile["MYFILE"]

Placing a Type Flag at the Front of the File

Finally, we'll rewrite the previous two examples, providing a String at the
front of the file to give us assurance that the files we're reading are
ones which we wrote.

Clear[storeInFile]
storeInFile[
                list_?((Select[Map[NumberQ,#],False]=={})&),
                s_String] :=
                Module[{myStream},
                   myStream = OpenWrite[s];
                   WriteString[myStream,"Specially created!\n"];
                   Do[
                      Write[myStream,list[[i]]],
                      {i,Length[list]}
                   ];
                   Close[myStream];
                ];

Clear[readMyFile]
readMyFile[s_String] :=
  Module[{myStream,l,ss},
     myStream = OpenRead[s];
     ss = Read[myStream,String];
     If[ss=="Specially created!",
        l = ReadList[myStream,Number],
        Print["Invalid file!"]; l={}];
     Close[myStream];
     l
  ];

Note that since we were using "WriteString[ ]", we had to tell Mathematica
explicitly to end the line in writing the special code out (using "\n"),
but when we read the string back in, the end-of-line character "\n" is not
contained in the resulting string ("ss" above).

On some systems we are not allowed to re-write onto a file that already
exists (and sometimes the "DeleteFile[ ]" command does not work); hence
we'll switch to "MYFILE2".

storeInFile[{1,2,3,4,5},"MYFILE2"]

!!MYFILE2

readMyFile["MYFILE2"]