Converting Raw Data into XML

An XML Example Using Mathematica 4.2

One of the useful tools in Mathematica 4.2 is the ability to manipulate XML data.  This notebook provides an example of how to use Mathematica to convert a regular row-column data set into an XML document and then export it for later use.  In this case, the data set is an XML form of the Ritter Catalog of Cataclysmic Variable Stars (Ritter, H., & Kolb, U., 1998, A&AS, 129, 83).  The data file was converted into a CSV file using Microsoft Excel and now Mathematica can import the file correctly.  You will probably need to modify the path to the file to reflect its location at your end.

data = Import["ritter.csv", "CSV"];

The next step is to define a list of tag names that will be used in the final XML file.

\(\(atts = {"NAME", "RA", "TYPE", "MAG1", "MAG3", "T1", "ORBPER", "THREEPER", "EB", "SPECTR2", "MASSRATIO", "INCL", "M1", "M2", "ALTNAME", "DEC", "TYPE2", "MAG2", "MAG4", "T2", "TWOPER", "FOURPER", "SB", "SPCTR1", "MRERROR", "INCLERROR", "M1ERROR", "M2ERROR"};\)\)

Since the original data file has table headers, which are not needed in the final XML form, we can strip these headers off.  The first two rows are the headers.

data = Drop[data, 2];

The data file provides a list of objects and their parameters.  In the original data set, every pair of rows corresponds to a single object.  We can use Mathematica to group the data into pairs.

data = Partition[ToString /@ Flatten[data], 28];

Next, we define a set of functions that allow us to convert each data entry into a corresponding SymbolicXML structure. Finally, the data set is converted into SymbolicXML.

ValToSymbolicXML[name_, val_] := 
        XMLElement[name, {}, {val}];
CVToSymbolicXML[cv_] := 
        XMLElement["OBJECT", {}, 
          MapThread[ValToSymbolicXML[#1, #2] &, {atts, cv}]];
DataToSymbolicXML[data_] := 
        XMLElement["CATALOG", {}, CVToSymbolicXML /@ data];
data = DataToSymbolicXML[data];

The original data set had many blank fields.  Not all of the objects have all of the parameters defined.  We can perform some "house cleaning" by defining a function to remove any elements that are undefined/blank or that have question marks in them.  Question marks were added to the original data set to denote end of record markers for those cases in which the final fields were blank.

data = 
        data /. {XMLElement[_, {}, {""}] -> Sequence[], 
            XMLElement[_, {}, {"?"}] -> Sequence[]};

The original data set used special characters in some of the fields to denote special meaning.  This notation is fine for visual inspection of the catalog, but to make the data easier to analyze in an XML format, these special characters are better reflected as attributes of XMLElements rather than as part of the content of the XMLElement.  The special character used in the original data set included colons, question marks, asterisks, less than, greater than, and B (denoting that a blue filter was used).  Next, we define a set of tests to determine if a data entry has a special character associated with it.

EndsInColon[s_String] := ":" === 
          StringTake[s, -1];
EndsInQuestion[s_String] := "?" === 
          StringTake[s, -1];
EndsInStar[s_String] := "*" === 
          StringTake[s, -1];
StartsWithLess[s_String] := "<" === StringTake[s, 1];
StartsWithGreater[s_String] := ">" === 
          StringTake[s, 1];
EndsWithB[s_String] := "B" === 
          StringTake[s, -1];

With the above tests defined, we can apply the tests using replacement rules.  The resulting data is then ready for exporting as a generic XML file.

data = 
        data /. {XMLElement[name_, atts_, {val_?EndsInColon}] :> 
              XMLElement[name, 
                Append[atts, "uncertain" -> "true"], {StringDrop[
                    val, -1]}]};
data = 
        data /. {XMLElement[name_, atts_, {val_?EndsInQuestion}] :> 
              XMLElement[name, 
                Append[atts, "uncertain" -> "true"], {StringDrop[
                    val, -1]}]};
data = 
        data /. {XMLElement["ORBPER", atts_, {val_?EndsInStar}] :> 
              XMLElement["ORBPER", 
                Append[atts, "spectroscopic" -> "true"], {StringDrop[
                    val, -1]}]};
data = 
        data /. {XMLElement[name_, atts_, {val_?StartsWithLess}] :> 
              XMLElement[name, 
                Append[atts, "valuetype" -> "max"], {StringDrop[
                    val, -1]}]};
data = 
        data /. {XMLElement[name_, atts_, {val_?StartsWithGreater}] :> 
              XMLElement[name, 
                Append[atts, "valuetype" -> "min"], {StringDrop[val, 
                    1]}]};
data = 
        data /. {XMLElement[name_, atts_, {val_?EndsWithB}] :> 
              XMLElement[name, 
                Append[atts, "filter" -> "B"], {StringDrop[
                    val, -1]}]};

Mathematica 4.2 has the ability to Export[ ] the SymbolicXML expression to regular XML format.

Export["ritterxml.xml", data]