Extending Mathematica's XML Capabilities with J/Link

Todd Gayley
Wolfram Research

Note: This package requires XML features present only in Mathematica 4.2 and later. You will not be able to run these examples unless you have that version (not available yet at the time of this writing). At the time that 4.2 ships, I will make the code for the XMLJava application described here freely available, probably from the J/Link area of the WRI site (www.wolfram.com/solutions/mathlink/jlink).

Introduction

This talk will present some of the capabilities of the XMLJava application. This application adds some useful functionality to Mathematica's XML-handling features.

The XMLJava application relies on open source Java implementations of XML technologies such as DOM, SAX, and XSLT. These Java implementations are called from Mathematica using J/Link. Java is a natural language to use for XML processing, as it has well-developed class libraries for everything related to XML, and most new work with XML appears first in a Java implementation.

The application will be made freely available as an unsupported tool and J/Link sample application.

Features Summary

Create Java DOM objects from XML files or URLs, or Symbolic XML expressions.
Create Symbolic XML from Java DOM objects.

SAX parsing of XML data, with SAX event handler functions written in Mathematica.

Perform XSLT tranformations on XML input from a file, URL, or Symbolic XML expression.

Perform XPath queries on XML input from a file, URL, or Symbolic XML expression.

Direct access to the DOM API, or any other Java API, via J/Link. Motivated users can easily extend the package or add new features. Any XML technology that works with DOM objects can be wired up easily.

Complete source code is provided.

Initialization

Needs["XML`"]

Needs["XMLJava`"]

sampleFile = ToFileName[{$TopDirectory, "AddOns", "Applications", "XMLJava", "SampleFiles"}, "stocks.xml"];

ImportXML[sampleFile]

Java DOM Objects

The key capability of XMLJava is its ability to construct Java DOM objects out of Symbolic XML and vice versa. Once we have the ability to go back and forth between DOM and Symbolic XML, a wide variety of XML operations are easy. For example, XSLT transformation and XPath queries can be performed on DOM objects.

DOM stands for Document Object Model, and it is a standard API for manipulating XML data. XMLJava uses the Java API for XML Processing (JAXP) implementation of various XML APIs.

The function that creates a DOM object is called MakeDOM. It can operate on a variety of inputs:
	- a raw string of XML data
	- an XML filename
	- a URL that points to an XML data source
	- a Symbolic XML expression

dom = MakeDOM[sampleFile]

One thing to do is to convert to Symbolic XML:

ToSymbolicXML[dom]

Developers will find that DOM objects are an extremely useful starting point for a wide variety of present and future XML technologies and manipulations. The XSLTransform and XPathApply functions are just a few lines of code that operate on DOM objects. Other XML functions should be similarly straightforward with a DOM object as the starting point.

XSLT

The XSLTransform function takes an XML input and an XSL stylesheet, performs the transformation, and returns the Symbolic XML expression that represents the resulting XML. The input can be a raw XML string, a filename, a URL, a Java DOM object, or a Symbolic XML expression. The XSL stylesheet must be given as a filename.

birdsFile = ToFileName[{$TopDirectory, "AddOns", "Applications", "XMLJava", "SampleFiles"}, "birds.xml"];
xslFile = ToFileName[{$TopDirectory, "AddOns", "Applications", "XMLJava", "SampleFiles"}, "birds.xsl"];

XSLTransform[birdsFile, xslFile]

XPath

Another XML technology that can operate on DOM objects is XPath. XPath provides a way to address parts of an XML document. For use from Mathematica, you can think of it as a query language for extracting parts of an XML document.

You can use the XPathApply function to apply an XPath expression to XML input and return the resulting portions. The input to XPathApply can be a raw XML string, filename, URL, or Symbolic XML expression.

This query extracts all elements with tag name "Species", wherever they appear:

XPathApply[birdsFile, "//Species"]

As a means of extracting desired portions of an XML document, XPathApply can be compared to the Mathematica functions Cases and Select. In fact, it is usually the case that extracting portions of an XML document that match a certain pattern is easier in the Mathematica language than using XPath. This is a reason why Mathematica is such a natural language for manipulating XML data. Here is the equivalent Mathematica code:

Cases[ImportXML[birdsFile], XMLElement["Species", __], Infinity]

It takes a bit of thought to come up with an example where the XPath query is significantly easier to write than the corresponding Mathematica code (at least, easier for an experienced Mathematica programmer). An example is the following: "Give me the common names of all the listed birds in the order Pelecaniformes". Here it is using XPathApply:

XPathApply[birdsFile, "//Order[@Name='PELECANIFORMES']//Species/text()"]

Here it is in Mathematica. It takes two separate calls to Cases:

pelicans = Cases[ImportXML[birdsFile], XMLElement["Order", {_ -> "PELECANIFORMES"}, _], Infinity];
Cases[pelicans, XMLElement["Species", _, {name_}] :> name, Infinity]

The XPathApply function will prove useful to those programmers who are fluent in XPath but not in Mathematica, or who already have appropriate XPath expressions supplied from another source, or who have queries that are sufficiently complex and specialized that they are most easily expressed in XPath.

SAX

SAX is another popular API for parsing XML. Instead of building up an in-memory tree structure (the DOM object), a SAX parser scans through an XML document firing callbacks to user-specified functions as parsing events occur ("startElement", "endElement", etc.) In many cases, it is more convenient to simply be notified of things of interest in the XML file, rather than being given a big tree (or Mathematica Symbolic XML expression) that you have to walk through and inspect. With SAX, you can parse documents much larger than your available system memory, and you can construct your own data structures using your callback event handlers.

The XMLJava application lets you use SAX to parse XML, and specify Mathematica functions to be called in response to parsing events. One example of what you can do in your callback handlers is build a Symbolic XML expression. Of course, this isn't perticularly useful since ImportXML does this for you. But it is a useful starting point.

SAXParser[birdsFile]

The SAXParser function takes two arguments and some options. The first argument is the XML input, which can be a raw string, a filename, or a URL. The second argument is a list of rules that associate Mathematica functions with SAX parsing events. If you leave out this argument, it defaults to using the handlers that build Symbolic XML, as in the line above.

Building an expression like the above example is an atypical use of SAX--you usually use SAX if you are interested in gleaning some information from a document, not acquiring a complete representation of it.

Here is a more realistic example of how you would use the SAXParser function. We want to obtain a list of all the common names of birds referred to in the birds.xml document. The common names appear as the text content of elements with tag name "Species". To extract this information, we will only need to be notified of the "startElement" and "characters" SAX events. Here are the handlers we define to be called when these events occur:

startElementHandler[uri_, localName_, qualifiedName_, attrList_] :=
	captureNextText = (localName == "Species")
	
charactersHandler[chars_List] :=
	(
		If[captureNextText, AppendTo[namesList, FromCharacterCode[chars]]];
		captureNextText = False
	)

Now we call SAXParser:

Block[{namesList = {}},
	SAXParser[birdsFile, {"startElement" -> "startElementHandler", "characters" -> "charactersHandler"}];
	namesList
]

Please note that I don't recommend that you go to the effort of using SAX for this sort of thing unless you really want to avoid building an entire DOM tree and associated Symbolic XML expression. It is usually much easier to just get the Symbolic XML expression and manipulate it with Mathematica code. Compare how simple it is to extract the names from the Symbolic XML:

Cases[ImportXML[birdsFile], XMLElement["Species", _, {name_}] :> name, Infinity]

Example: Headlines From the Internet

This example shows how to read headlines in XML format from the Slashdot Internet news site and build a table of hyperlink buttons in Mathematica that point to the stories. We will examine several ways to process the XML headline data.

slashdotNews = ImportXML["http://www.slashdot.org/slashdot.xml"]

Now let's extract the headline information we need from the XML from the Slashdot site. We want a list of pairs {"headline", "url"}. This will be fed into a function that builds the hyperlink buttons. First, we try the extraction using the Mathematica language operating on Symbolic XML:

headlines1 =
	Cases[slashdotNews,
		XMLElement["story", _, {___, XMLElement["title", _, {title_}], ___,
			XMLElement["url", _, {url_}], ___}] :> {title, url},
		Infinity
	]

Now for comaprison, we do the same thing using XPath:

dom = MakeDOM["http://www.slashdot.org/slashdot.xml"];
headlines2 =
	Partition[XPathApply[dom, "//story/title/text() | //story/url/text()"], 2]

For a third method, we will call methods in the DOM API directly. We are using "raw" J/Link here, making Java calls directly from Mathematica:

titles = dom@getElementsByTagName["title"];
urls = dom@getElementsByTagName["url"];
len = titles@getLength[];
headlines3 =
	Table[{titles@item[i]@getFirstChild[]@getNodeValue[], urls@item[i]@getFirstChild[]@getNodeValue[]},
		{i, 0, titles@getLength[] - 1}
	]

headlines1 === headlines2 === headlines3

Now build the buttons:

NotebookWrite[EvaluationNotebook[], Cell[{ButtonBox[#1, ButtonData :> {URL[#2], None}, ButtonStyle->"Hyperlink"]& @@@ headlines1}, "Input"]]

Lucent' s New Chip Is Just One Molecule Thick
Slashback :  Quiesence,  Jazz,  RAND
Sprint ION' s $100/mo,  8  Mbps Home Service Tanks
SkyOS Now Runs Linux Binaries Natively
Yellow Dog Linux 2.1 Shipping
EFF speaks out against MAPS
TiVo Gets In Deeper With Sony
Digital Cameras Go Disposable
Neighborhood Area Networks?
Gilmore Commission Recommends Secret ' Cyber Court'

Example: Processing Stock Quotes From the Internet

Stock quotes are available from the Internet in XML format from xmltoday.com. Here we get the information for IBM, MSFT, and SUNW:

quotes = ImportXML["http://www.xmltoday.com/examples/stockquote/getxmlquote.vep?s=IBM+MSFT+SUNW"]

Let's say we want to plot the intraday change as a function of volume for these stocks. Here is a complicated Cases function that will extract and manipulate the data we want, resulting in a list of triplets {volume, change, tickerName}:

data =
	Cases[
		quotes,
		XMLElement["stock_quote", _, {___, XMLElement["symbol", _, {ticker_}], ___,
			XMLElement["price", {"type"->"dayhigh", "value"->highval_}, _], ___,
			XMLElement["price", {"type"->"daylow", "value"->lowval_}, _], ___,
			XMLElement["volume", _, {vol_}], ___}] :> 
				{ToExpression[vol], ToExpression[highval] - ToExpression[lowval], ticker},
		Infinity
	]

<<Graphics`Master`
TextListPlot[data]

Now let's do the same thing using XPath to extract the information:

stockDOM = MakeDOM["http://www.xmltoday.com/examples/stockquote/getxmlquote.vep?s=IBM+MSFT+SUNW"];

highs = ToExpression /@ Last /@ XPathApply[stockDOM, "//stock_quote/price[@type='dayhigh']/@value"];

lows = ToExpression /@ Last /@ XPathApply[stockDOM, "//stock_quote/price[@type='daylow']/@value"];

tickers = XPathApply[stockDOM, "//stock_quote/symbol/text()"];

volume = ToExpression /@ XPathApply[stockDOM, "//stock_quote/volume/text()"];

Transpose[{volume, highs - lows, tickers}]

XSLTransform[input_, xsl_String] :=
	JavaBlock[
		Module[{dom, factory, transformer, resultDOM},
			dom = MakeDOM[input];
			LoadClass["javax.xml.transform.TransformerFactory"];
			factory = TransformerFactory`newInstance[];
			transformer = factory@newTransformer[JavaNew["javax.xml.transform.stream.StreamSource", toFileURL[xsl]]];
			resultDOM = JavaNew["javax.xml.transform.dom.DOMResult"];
			transformer@transform[JavaNew["javax.xml.transform.dom.DOMSource", dom], resultDOM];
			ToSymbolicXML[resultDOM@getNode[]]
		]
	]

XPathApply[input_, xpath_String] :=
	JavaBlock[
		Module[{dom, list},
			dom = MakeDOM[input];
			LoadClass["org.apache.xpath.XPathAPI"];
			list = XPathAPI`selectNodeList[dom, xpath];
			If[list === Null,
				{},
			(* else *)
				ToSymbolicXML[list@item[#]]& /@ Range[0, list@getLength[] - 1]
			]
		] // Map[Replace[#, XMLObject["XMLDocument"][_, elmnts_XMLElement, ___] -> elmnts]&, #]&
	]