Data Collections with Mathematica

Paul Wellin

Wolfram Research

wellin@wolfram.com

CellPrint[TextCell[
Row[{
Dynamic[Style[DateString[],FontFamily->"Helvetica"],UpdateInterval->1],
Style["  (GMT: "],$TimeZone,")"}],"Subsubsection",ShowStringCharacters->False,CellMargins->{{60,Inherited},{Inherited,Inherited}}]]

(GMT: -5.`)

The Problem with Data

Users of data are typically faced with the following issues:

difficult to locate and extract from sources

data from seemingly unrelated fields is scattered across the vastness of the Internet

different disciplines use different conventions for naming, units, methods, etc.

data comes in many different file formats

databases are not always easy to use

Scope of Data Collections

Mathematica 6 introduced the notion of computable data collections, gathering together data from a wide range of disciplines into a unified computational framework for immediate and direct use in calculations. In Mathematica 7 we now have over 20 data functions:

Physical and chemical data: ElementData, ChemicalData, IsotopeData, ParticleData, »

Earth and astronomical data: WeatherData, GeodesyData, CityData, CountryData, GeoDistance, AstronomicalData, »

Life science data: GenomeData, GenomeLookup, SequenceAlignment, ProteinData, »

Finance and economic data: FinancialData, CountryData, »

Mathematical data: FiniteGroupData, GraphData, KnotData, LatticeData, PolyhedronData, »

Linguistic data: DictionaryLookup, WordData, ExampleData, »

This tutorial will introduce data collections in Mathematica, starting with a brief discussion of their scope. We will then turn to syntax and usage, and finally look at several concrete examples.

What is a Data Function?

A data function works much the same way as any other function in Mathematica, but returns stored data.

The basic syntax is xxxData["str"].

FinancialData["AAPL"]

ChemicalData["CyclosporinA"]

WeatherData["Seattle","Temperature"]

GenomeData["COX6BP3"]

Basic Examples

The interface to the data collections is much easier than standard database querying methods.

CityData["Paris"]

CityData[{"Paris","IleDeFrance","France"},"Population"]

You can compute with them.

((AstronomicalData["Saturn","OrbitPeriod"])/(AstronomicalData["Earth","OrbitPeriod"]))

You can plot them.

DateListPlot[FinancialData["AAPL","2005"]]

You can manipulate them.

TabView[Map[#->PolyhedronData[#]&,PolyhedronData["Platonic"]]]

How they Work

Mathematica does not ship with (most of) the data; it is stored on our servers and pushed to the user over the internet.

The first time you use a data function, Mathematica contacts our data server, which sends basic indexing info.

The first time you query for an entity/property in a session, Mathematica downloads the most updated data from our server, which gets stored locally on your hard drive in a compressed file.

In a Mathematica session, after a data function is first used, data gets cached in memory.

Initial scope

There are three different types of data collections:

Pre-computed: most of the mathematical data is precomputed in-house: e.g., GraphData, KnotData, LatticeData, PolyhedronData

Aggregated from publicly available sources: e.g., AstronomicalData, CityData, CountryData, ChemicalData, ElementData, IsotopeData, ParticleData, WordData

Direct connection to web services-based provider: e.g., FinancialData

Integrated data sources: Documentation

Basic Syntax

XXXXData["Entity", "Property"] returns a Mathematica expression. Arguments are generally given as strings.

ElementData["Einsteinium","DiscoveryYear"]

WeatherData["Tokyo","WindSpeed"]

FinancialData["^DJI","Members"]

Sometimes data is not available from the sources for a particular entity.

CountryData["Cuba","PovertyFraction"]

ElementData["Helium","MohsHardness"]

Missing values are ignored in plotting functions.

mh=ElementData[#,"MohsHardness"]&/@ElementData[]

ListPlot[mh]

Units, Notes, and All That

XXXXData["Entity","Property","Annotation"] gives additional info. Annotation can be used to get information such as units, and notes on entity properties.

Units:

AstronomicalData["Earth","RotationPeriod"]

AstronomicalData["Earth","RotationPeriod","Units"]

WeatherData["KCMI","Temperature"]

WeatherData["KCMI","Temperature","Units"]

Significant digits using Mathematica’s built-in precision tagging:

d=ChemicalData["Aspirin","Density"]

Precision[d]

The decay width in MeV for the muon particle.

ParticleData["Muon","Width"]

Uncertainty interval:

ParticleData["Muon","Width","Interval"]

Notes:

ElementData["Helium","MeltingPoint"]

ElementData["Helium","MeltingPoint","Note"]

Scope

XXXXData[All] gives a list of entities.

ElementData[All]

AstronomicalData[All]//Length

Properties

XXXXData["Properties"] gives a list of properties.

ProteinData["Properties"]

WeatherData["Properties"]

GenomeData["Properties"]

Classes

XXXXData["Classes"] gives a list of classes.

AstronomicalData["Classes"]

GenomeData["Classes"]

Short[CountryData["Classes"],20]

Examples

In the remaining sections we will work through some examples from several different data collections. These will range from fairly basic to examples combining some more significant programming.

CountryData

Scope: 237 countries, 304 classes, 224 properties

CountryData[All]

Short[CountryData["Classes"],15]

Short[CountryData["Properties"],18]

Display the flag of Greenland.

CountryData["Greenland","Flag"]

Some miscellaneous data on Greenland.

Map[#->CountryData["Greenland",#]&,{"Languages","LifeExpectancy","MedianAge","PhoneLines","InternetUsers","MajorIndustries","GDP","ElectricalGridPlugImages","Population"}]//TableForm

Find the countries with the 10 highest rate of oil consumption.

TableForm[Take[SortBy[Select[({#1,CountryData[#1,"OilConsumption"]}&)/@CountryData[],FreeQ[#1,Missing["NotAvailable"]]&],Last],-10]]

CountryData["China","OilConsumption","Units"]

CityData

Scope: over 163,000 cities covered throughout the world.

CityData[All]

CityData["Properties"]

Find city names that occur most frequently in the United States.

data=Tally[Map[CityData[#,"Name"]&,CityData[{All,"UnitedStates"}]]]

Sort on the second element in each list and take the top 10.

top10=Take[SortBy[data,Last],-10]

Put the data in a formatted grid.

Grid[Prepend[top10,{"City Name","Frequency of Occurrence"}],
Spacings->{3,1},Alignment->{{Left,Center},Automatic},
Frame->All,Background->{None,{{{LightGray,Gray}},{1->LightYellow}}},BaseStyle->"Text"]

Bordering countries

In this example, we combine information from CountryData and CityData to display a network showing bordering countries of South America.

Gather the countries and the latitude/longitude coordinates of each capital.

countries=CountryData["SouthAmerica"]

capitals=Map[CountryData[#,"CapitalCity"]&,CountryData["SouthAmerica"]]

coordinates=Map[Reverse,CityData[#,"Coordinates"]&/@capitals]

Plot the relationships (sharing a border) as a graph.

p1=GraphPlot[
Flatten@Map[Thread[#->CountryData[#,"BorderingCountries"]]&,countries],VertexCoordinateRules->coordinates]

Plot the outlines of the countries.

p2=Graphics[Map[CountryData[#1,"FullPolygon"]&,countries]/.Polygon->Line]

Put it together.

Show[p1,p2]

ElementData

Scope: ElementData covers the entire periodic table, from hydrogen to element 118, Ununoctium.

ElementData[All]

Over 80 unique properties, including atomic weight, density, boiling & melting points, and many more.

ElementData["Properties"]

Use LatticeData to get detailed information on solid state structures.

ElementData["Titanium","CrystalStructure"]

LatticeData[%,"Image"]

Find how many elements were discovered in each country.

ChemicalData

Scope: currently over 34,000 compounds.

ChemicalData[]

Over 100 unique physical, structural, and safety properties, as well as chemical identifiers.

ChemicalData["Properties"]

Use string patterns to find a compound.

Short[ChemicalData["*chloro*"],10]

Find the melting point of ibuprofen.

ChemicalData["Ibuprofen","MeltingPoint"]

Find the structure diagram for riboflavin.

ChemicalData["Riboflavin"]

ChemicalData["Riboflavin","EdgeRules"]

ChemicalData["Riboflavin","VertexTypes"]

Find the 3D molecular structure for the cholesterol molecule.

ChemicalData["Cholesterol","MoleculePlot"]

ChemicalData["Cholesterol","AtomPositions"]

Create a table of various chemical properties (code for ChemicalTable at end of notebook).

ChemicalTable[{"Cyclohexanol","Acetone","DiethylEther","Tetrahydrofuran","DimethylSulfoxide"}]

GenomeData

Scope: nearly 40,000 genes from the human genome.

GenomeData[All]

Over 40 unique properties such as gene location, sequence properties, transcription, identification, and so on.

GenomeData["Properties"]

Nearly 200 classifications:

GenomeData["Classes"]

Here are the genes on human chromosome 22:

GenomeData["Chromosome22Genes"]

And here is one particular gene from this chromosome:

GenomeData["C22CTA250D109"]

GenomeData["C22CTA250D109","SequenceLength"]

Find the 15 shortest genes in the human genome:

Take[Sort[{GenomeData[#,"SequenceLength"],#}&/@GenomeData[]],15]

This finds the positions on the human genome at which this DNA sequence occurs.

GenomeLookup["CTCTCTAACTAAAT"]

WeatherData

Scope: access to over 17000 weather stations around the world; dozens of properties; current and historical data.

WeatherData[All]

WeatherData["Properties"]

Get information for Champaign IL using either the city name, a specific weather station, or latitude/longitude coordinates.

WeatherData["Champaign","Temperature"]

WeatherData["KCMI","WindSpeed"]

coords=CityData[{"Champaign","Illinois","UnitedStates"},"Coordinates"]

WeatherData[coords,"Humidity"]

Range of daily temperatures for the past three months.

DateListPlot[(WeatherData["KCMI",#1,{DateString[DatePlus[-90]],DateString[DatePlus[0]],"Day"}]&)/@{"MeanTemperature","MaxTemperature","MinTemperature"},Joined->{True,False,False},Filling->{2->{3}},Mesh->All]

FinancialData

Scope: over 189,000 entities from all over the world and more than 70 properties.

FinancialData[]//Length

FinancialData["Properties"]

Find the current net asset value for the Vanguard Windsor II fund.

FinancialData["VWNFX"]

Plot the log of the S&P500 for all available dates:

DateListLogPlot[FinancialData["SP500",All]]

Find members of the Dow Jones Industrial Average with low 50-day volatilities:

Select[FinancialData["^DJI","Members"],FinancialData[#,"Volatility50Day"]<0.25&];
FinancialData[#,"StandardName"]&/@%

First few members of the Frankfurt exchange:

frankMems=Take[FinancialData["Frankfurt","Members"],10]

FinancialData[#,"StandardName"]&/@frankMems

GraphData

Scope: over 3200 graphs, 157 classes, and 258 properties.

GraphData[All]

GraphData["Classes"]

GraphData["Properties"]

Characteristic polynomials for Foster (cubic symmetric) graphs.

GraphData["Foster*"]

GraphData["Foster098A","CharacteristicPolynomial"][x]//TraditionalForm

Compare its 2D representation and its toric structure in 3D.

GraphicsRow[
{GraphData["Foster098A","Image"],
GraphPlot3D[GraphData["Foster098A","EdgeRules"],Boxed->False]
},ImageSize->Large]

Look at all graphs and properties. (The function CollapseFriends is defined at the end of this notebook.)

Friends=First/@Split[GraphData[],CollapseFriends[#1]==CollapseFriends[#2]&];

Manipulate[Column[{GraphData[g],GraphData[g,p]}],{g,Friends},{p,Complement@@GraphData/@{"Properties","Classes"}}]

Embeddings:

GraphicsGrid[Partition[GraphData["TesseractGraph","AllImages"],6],ImageSize->10 72]

Find out more:

GraphData["TesseractGraph","Information"]

Tables of graphs with known embeddings:

GraphicsGrid[Partition[(Tooltip@@GraphData[#,{"Image","Name"}])&/@Take[GraphData["VertexCoordinates","Defined"],60],10],ImageSize->800]

Further Resources

Source information is available for the data collections by clicking the link at the bottom of the documentation page for each collection. For example, source information for ElementData can be found at reference.wolfram.com/mathematica/note/ElementDataSourceInformation.html. Additional information can be found at these sites:

Wolfram|Alpha: www.wolframalpha.com

Wolfram Library Archive: library.wolfram.com

Training from Wolfram Education Group: www.wolfram.com/weg

Initialization

ChemicalTable

ChemicalTable[chemicals_List]:=Module[{labels,style,data,img},
labels={"Name","Structural Diagram","Molecular Formula","Molecular Weight","Density (g (cm^(-3)))","Dielectric Constant","Boiling Point (°C)","Flash Point (°C)","NFPA Label"};
style:=Style[#,FontFamily->"Helvetica",FontWeight->Bold]&;
data=(Table[ChemicalData[#,property],{property,{"Name","ColorStructureDiagram","FormulaDisplay","MolecularWeight","DensityGramsPerCC","DielectricConstant","BoilingPoint","FlashPoint","NFPALabel"}}]&/@chemicals);
img=Text[Style[Grid[Transpose[Join[{style/@labels},data]],
Frame->All,
Background->{LightYellow,{{{GrayLevel[0.9],White}}}},
Alignment->{{Left,{Center}},Center}],
FontFamily->"Helvetica",Bold,13]
];
img/.{(ImageSize->_?NumericQ)->(ImageSize->110),(ImageSize->x_List)->(ImageSize->1.2*x)}
]