Using Java to get stock data from the internet into Mathematica

An obvious use of Java is reading files off the internet. Some websites allow you to read files that contain historical stock data.  In this example, we will use J/Link to retrieve particular stocks from http://data.moneycentral.msn.com. As a common foundation for retrieval, first a function must be created that takes an arbitrary file specified by a URL (http://data.moneycentral.msn.com.) and copies it to a local temporary file.  Java has a class called java.net.URL that represents a URL and allows data to be downloaded.  In particular,there is an openStream method that returns a standard InputStream,which can be easily read from. Later, embellishments will be added that will process the file and convert it into a ListPlot of our particular stock. We begin by loading the jlink package.

<<JLink`

GetURL

The following pseudocode was written to demonstrate what will be needed in the code.

GetURL[url_String] :=
	open a temporary local file
	create a new Java URL object
	create a Java input stream from the URL object
	read from the URL, writing the data to the local file
	close the stream
	close the local file
	release the Java objects created
	return the local filename

Here is the fundamental code that J/Link needs in order to read files from the internet.

GetURL[url_String] :=
	JavaBlock[
		Module[{u, stream, numRead, outFile, buf},
			InstallJava[];
			outFile = OpenTemporary[DOSTextFormat->False];
			u = JavaNew["java.net.URL", url];
			(* This is where the error will show up if the URL is not valid.
			   A Java exception will be thrown during openStream, which
			   causes the method to return $Failed.
			*)
			stream = u@openStream[];
			If[stream === $Failed, Return[$Failed]];
			buf = JavaNew["[B", 5000]; (* 5000 is an arbitrary buffer size. *)
			While[(numRead = stream@read[buf]) > 0,
				WriteString[outFile, FromCharacterCode[Take[Val[buf], numRead]]]
			];
			stream@close[];
			Close[outFile]   (* Close returns the filename. *)
		]
	]

There is one interesting wrinkle in the code. You need to use the read method of the InputStream you get back from openStream. The read method is declared as public int read(byte[] b). The read method does not return the array of data; it takes the array as an argument and fills it. A Java method that takes an array of bytes would typically be called from Mathematica with a list of integers. If you create a list of 5000 integers, 0s for example, and send that as the argument to read, what would happen? An array of this data would be created in Java and passed to read. The read method would then fill the array with data, overwriting the 0s, and return the number of bytes read. This method works fine except for one problem: you cannot get the data back into Mathematica! You have no way of retrieving the array that was created from the list of numbers you sent.

The solution to this problem is to create an array of bytes on the Java side with JavaNew. Then you have a reference to the array that will be filled by read. You can pass this reference (buf in the code above) to the read method instead of a Mathematica list of numbers. Recall that all Java methods that take arrays can be called from Mathematica with either a Mathematica list or a reference to an array on the Java side. The same is true for strings.

To create a Java array object, you need to use JavaNew with the class name as the first argument and the length as the second argument. The actual Java class name for an array of bytes is [B, and J/Link makes no attempt to hide these cryptic array class names behind a prettier interface. To create a multidimensional array, pass a list of integers as the second argument. For example, JavaNew["[[I", {3, 4}] creates the array int[3][4].

When the array has been filled by read, calling Val forces the array reference buf to be converted into its Mathematica value representation, which is a list of integers. These numbers are then converted to a string via FromCharacterCode and written to the file. Section 1.2.4 of the J/Link User Manual discusses Val and other issues concerning how J/Link handles "by value" and "by reference" representations of Java objects.

Now that you have the GetURL function, you can easily write functions that read files directly off the internet into the kernel.

GenerateStockURL

Since we will be using http://data.moneycentral.msn.com, we will need to input an appropriate URL to GetURL in order to retrieve the exact stock data we want.  For example, if we would like to retrieve the Dow Chemical Stock from Jan 2000 through Dec 2000 we would need to use the following URL, http://data.moneycentral.msn.com/scripts/chrtsrv.dll?Symbol=DOW&C2=0&C1=2&C3=0&C4=0&C5=01&C6=2000&C7=12&C8=2000&C9=15&FileDownload=.  We will simplify this by creating a function that will create that URL for us

The following function will take a stock symbol, a starting month and year, and an ending month and year.  It will then replace certain values in the URL with your values.  The return value will be an appropriate URL.

GenerateStockURL[{symbol_, startmonth_, startyear_, endmonth_,  
            endyear_}] := 
        Block[{urlstring, f, 
            x}, 
If[(startmonth  ≥  
                    endmonth  &&  
                  startyear  ==  
                    endyear)  ||   (startyear  >  
                  endyear), (Print["Illegal data. Aborted."];  
              Abort[];)]; 
f[x_] := 
            If[StringLength[ToString[x]] == 1, "0" <> ToString[x], 
              ToString[x]]; 
urlstring = 
            ToString["http://data.moneycentral.msn.com/scripts/chrtsrv.dll?\
Symbol=stocksymbol&C2=0&C1=2&C3=0&C4=0&C5=startmonth&C6=startyear&C7=endmonth&\
C8=endyear&C9=15&FileDownload="]; 
StringReplace[
            urlstring, {"stocksymbol" ->  
                ToString[symbol], "startmonth" ->  
                f[startmonth], "startyear" -> 
                ToString[startyear], "endmonth" -> 
                f[endmonth], "endyear" ->  
                ToString[endyear]}]
];

GenerateStockURL[{DOW,  1,  2000, 12, 2000}]

CreateStockPlot

The following function will take your particular stock and either a high, low, open, or close parameter and create a ListPlot of your data.  This function simply illustrates a few basic Mathematica commands, which can be highly embellish for your particular needs.

\(\(CreateStockPlot[Stock_, \ HLOC_String] := 
        Module[{theurl, \ temporaryfile, \ importeddata, \ coredata, \ 
            closing}, 
theurl = 
            GenerateStockURL[Stock]; \ 
temporaryfile = 
            GetURL[theurl]; \ 
          importeddata = 
            Import[temporaryfile, "Table", 
              ConversionOptions -> \ {"TableSeparators" -> {{"", "
"}, {","}}}]; 
          coredata\  = \ 
            Part[importeddata, 
              Range[7, 
                Length[importeddata]]]; 

Which[
            HLOC == "Open", 
priceData = 
              Reverse[\(Transpose[coredata]\)[\([2]\)]], \ 
            HLOC == "High", \ 
            priceData = Reverse[\(Transpose[coredata]\)[\([3]\)]], \ 
            HLOC == "Low", \ 
            priceData = Reverse[\(Transpose[coredata]\)[\([4]\)]], \ 
            HLOC == "Close", \ 
            priceData = 
              Reverse[\(Transpose[
                    coredata]\)[\([5]\)]]]; 

ListPlot[priceData, PlotJoined -> \ True, 
            PlotStyle -> RGBColor[1, 0, 0], 
            PlotLabel -> 
              StringDrop[
                StringDrop[ToString[importeddata[\([4]\)]], 1], \(-1\)], 
            Ticks -> {Automatic, Automatic}, 
            AxesLabel -> {"Days", \ "Price"}, \ 
            ImageSize -> \ {400, 300}]
];\)\)

What this leads to:

CreateStockPlot[{DOW,  1,  2000, 12, 2000}, "High"]

CreateStockPlot[{MSFT,  1,  2000, 12, 2000}, "Low"]