Working with NoSQL Using Wolfram Language

Mili Vishwakarma

Wolfram Technology Group

Outline

What is NoSQL

MongoDB basics

Configuring and connecting to MongoDB from Mathematica

Create, Retrieve, Update and Delete from MongoDB

Neo4j basics

Configuring and connecting to Neo4j

Neo4j operations on a simple database

Resources

What is NoSQL?

Data storage and retreival mechanism

Data is stored in any structure other than a table based

Can be a Key-Value pair , a graph based database, column based or document based

Example - Cassandra, MongoDB, Neo4j

MongoDB basics

MongoDB is a document based database

Every document is a BSON document which is binary representation of JSON

Documents consist of Field:Value pairs

{
               field1 : value1
               field2 : value2
               field3 : value3
               .
               .
              fieldN : valueN

}

Every field:value pair is called BSON element

A collection of BSON documents is called a BSON Collection

Example MongoDB Collection

Source : MongoDB Document, Release 2.4.9, http://docs.mongodb.org/manual/

Configuring MongoDB

Install MongoDB - http://www.mongodb.org/downloads?_ga=1.97695135.1435617592.1412267567

To Start MongoDB

Launch command prompt and change directory to MongoDB\bin

Start mongod.exe

Launch another command prompt and change directory to MongoDB\bin

Start mongo.exe

This will start the MongoDB server on local machine

To set up with username and password follow instructions from manual

Configuring MongoDB

In Mathematica

Download  a driver file from http://docs.mongodb.org/ecosystem/drivers/community-supported-drivers/

Copy the driver file to appropriate directories

If using Java based driver copy to

FileNameJoin[{$InstallationDirectory,"SystemFiles","Links","JLink"}]

If using a .NET based driver copy to

FileNameJoin[{$InstallationDirectory,"SystemFiles","Links","NETLink"}]

Connecting to MongoDB

Launch NETLink package

Needs["NETLink`"]

Start NETLink Runtime

InstallNET[]

ConnectToMongoDB[hostname_String]:=Module[{client,newclient},
LoadNETAssembly["MongoDB.Driver"];
client=LoadNETType["MongoDB.Driver.MongoClient"];
newclient=NETNew[client,"mongodb://"<>hostname]]

conn=ConnectToMongoDB["localhost"]

Create BSON Document and Element

Flow

Create collection

GetMongoServer[conn_]:=Module[{},conn@GetServer[]]

createCollection[conn_,db_String,collectionname_String]:=Module[{server,database},
server=GetMongoServer[conn];
database=server@GetDatabase[db];
database@CreateCollection[collectionname]
]

createCollection[conn,"mydb","UserData"]

Create BSON Document

CreateBsonDocument[]:=Module[{newdoc},
newdoc=NETNew["MongoDB.Bson.BsonDocument"]
]

{doc1,doc2,doc3,doc4}=Table[CreateBsonDocument[],{x,1,4}];

Create BSON Element

CreateBsonElement[fieldname_String,value_,conn_]:=Module[{server,bsonvalueload,bsonelement,bsonValue},
server=GetMongoServer[conn];
bsonvalueload=LoadNETType["MongoDB.Bson.BsonValue"];
bsonelement=LoadNETType["MongoDB.Bson.BsonElement"];
bsonValue=MongoDB`Bson`BsonValue`Create[value];
bsonelement=NETNew[bsonelement,fieldname,bsonValue]
]

{element1, element2,element3}={CreateBsonElement["name","Nina",conn],CreateBsonElement["age",45,conn],CreateBsonElement["status","A",conn]}

names={"David","Smith","Sarah"};

age={30,45,26};

status={"B","A","B"};

elementlist=MapThread[{CreateBsonElement["name",#1,conn],CreateBsonElement["age",#2,conn],CreateBsonElement["status",#3,conn]}&,{names,age,status}]

Insert BSON element into the document

InsertElementIntoBsonDocument[doc_,element_]:=Module[{},
doc@Add[element]
]

InsertElementIntoBsonDocument[doc1,#]&/@{element1,element2,element3}

docs=MapThread[InsertElementIntoBsonDocument[#1,#2]&,{{doc2,doc3,doc4},elementlist}]

Insert BSON document into collection

InsertBsonDocIntoCollection[conn_,db_,collectionname_,doc_]:=Module[{server,database,dbitem},
server=GetMongoServer[conn];
database=server@GetDatabase[db];
dbitem=database@Item[collectionname];
dbitem@Insert[doc];
]

InsertBsonDocIntoCollection[conn,"mydb","UserData",doc1]

InsertBsonDocIntoCollection[conn,"mydb","UserData",#]&/@{doc2,doc3,doc4}

Retrive the data

Get a list of collections from a database

Getcollection[conn_,dbname_]:=Module[{server,db,count},
server=conn@GetServer[];
db=server@GetDatabase[dbname];
count=db@GetCollectionNames[]@Count;
Table[db@GetCollectionNames[]@Item[i],{i,count-1}]]

Getcollection[conn,"mydb"]

Get data from a collection

GetData[conn_,db_,collecName_]:=Module[{server, database,dbitem,dbitemfindall,dbenum,list,i,},
server=conn@GetServer[];
database=server@GetDatabase[db];
dbitem=database@Item[collecName];
dbitemfindall=dbitem@FindAll[];
dbenum=dbitemfindall@GetEnumerator[];
countdoc=dbitem@Count[];
list={};
i=Boole[dbenum@MoveNext[]];
While[i==1,dbbsondoc=dbenum@Current;count=dbbsondoc@Elements[]@Count;j=1;While[j<count,AppendTo[list,{dbbsondoc@GetElement[j]@Name,dbbsondoc@GetElement[j]@Value@Value}];j++];i=Boole[dbenum@MoveNext[]];];
Flatten[Partition[list,countdoc],1]
]

dat=GetData[conn,"mydb","UserData"]

p=Partition[Map[#/.{a_,b_}-> a->b&,dat],3]

assoc=Association[#]&/@p

dataset=Dataset[assoc]

dataset[Select[#age>30&],"age"]

dataset[All,"name"]

Update data

Updating data

UpdateBsonDocument[conn_,db_,collection_,matchfield_,matchfieldvalue_,updatefield_,updatefieldvalue_]:=Module[{server,dbobj,dbitem,queryinsert,newdoc,multiflag,updateobj},
server=GetMongoServer[conn];
dbobj=server@GetDatabase[db];
dbitem=dbobj@Item[collection];
queryinsert=InsertElementIntoBsonDocument[NETNew["MongoDB.Driver.QueryDocument"],CreateBsonElement[matchfield,matchfieldvalue,conn]];
newdoc=NETNew["MongoDB.Driver.UpdateDocument","$set",NETNew["MongoDB.Bson.BsonDocument",updatefield,MongoDB`Bson`BsonValue`Create[updatefieldvalue]]];
multiflag=NETNew["MongoDB.Driver.UpdateFlags"];
updateobj=dbitem@Update[queryinsert,newdoc,multiflag@Multi]]

UpdateBsonDocument[conn,"mydb","UserData","status","A","age",30]

Neo4j Basics

Graph based database

Data is stored as nodes connected by directed, typed relationship with properties

Several nodes can be grouped under a label, say Person

A relationship describes how a node is connected

John KNOWS David and Johan

Johan KNOWS David

Accepts Cypher language queries

Configuring Neo4j

Install Neo4j Community Edition by downloading from - http://neo4j.com/download/

Start Neo4j server by going to Start-> All Programs-> Neo4j Community-> Neo4j Community

Create a folder for example ‘Neo4jdb’ at a location that has permissions to read and write.

In the Neo4j start up window browse to the folder ‘Neo4jdb’ and click on Start

Connecting to Neo4j

Download the Neo4j driver, we will be using Java driver in this presentation - http://neo4j.com/contrib/

Load JLink package and start the Java Runtime

Needs["JLink`"]

InstallJava[]

Add driver path to classpath

AddToClassPath["C:\\Users\\miliv\\Desktop\\neo4j-jdbc-master\\neo4j-jdbc-master\\target\\"];

Make a connection

ConnectToNeo[hostname_String]:=Module[{jc,drvmgr,prop,conn},
jc=LoadJavaClass["org.neo4j.jdbc.Driver"];
drvmgr=JavaNew["org.neo4j.jdbc.Driver"];
prop= JavaNew["java.util.Properties"];
conn=drvmgr@connect["jdbc:neo4j://"<>hostname,prop]]

connNeo=ConnectToNeo["localhost:7474"]

Neo4j Operations

Function to evaluate queries that do not return any data, for example creating a database

ExecuteQuery[conn_, query_String] := Module[{stmt, create},
  LoadJavaClass["org.neo4j.jdbc.IteratorResultSet"];
stmt = conn@createStatement[IteratorResultSet`TYPEUSCROLLUSENSITIVE, IteratorResultSet`CONCURUUPDATABLE];
  create = stmt@executeQuery[query]]

Function to evaluate queries that return some data

GetData[conn_,query_String]:=Module[{obj},
obj=ExecuteQuery[conn,query];
If[ClassName[obj]=="org.neo4j.jdbc.IteratorResultSet",
If[obj@isBeforeFirst[]==True,list={};
While[obj@next[]==True,data=obj@getString[1];AppendTo[list,data]]]];
list]

Neo4j operations on a simple database

Create a Person database

ExecuteQuery[connNeo,"CREATE (person1:Person { name : 'John', Country : 'USA' })
CREATE (person2:Person { name : 'David', Country : 'Canada' })
CREATE (person3:Person { name : 'Johan', Country : 'Sweden' }),
(person1)-[:KNOWS]->(person2),(person1)-[:KNOWS]->(person3),(person3)-[:KNOWS]->(person2)"]

Retrieve a single node by name

GetData[connNeo,"MATCH (person:Person { name: 'David' })
RETURN person"]

Get all names and countries

names=GetData[connNeo,"MATCH (person:Person)
RETURN person.name
ORDER BY person.name;"]

country=GetData[connNeo,"MATCH (person:Person)
RETURN person.Country
ORDER BY person.name;"]

namestr=Map[StringJoin["'",#,"'"]&,names];

Get all connections of ‘John’, ‘David’ and ‘Johan’

query=Map["MATCH (:Person{name:"<>#<>"})-[:KNOWS]->(person) RETURN person.name;"&,namestr];

connectionlist=MapThread[{#1,GetData[connNeo,ToString[#2]]}&,{names,query}];

connectionlistflatten=Flatten[#,2]&/@connectionlist;

numberlist=connectionlistflatten/.{"David"-> 1,"Johan"-> 2,"John"-> 3};

Graph[{2\[DirectedEdge]1,3\[DirectedEdge]2,3\[DirectedEdge]1},VertexShapeFunction->"Circle",VertexLabels->{1-> Placed["David",Below],2-> Placed["Johan",Above],3-> Placed["John",Above]},EdgeLabels->{2\[DirectedEdge]1-> "KNOWS",3\[DirectedEdge]2-> "KNOWS",3\[DirectedEdge]1-> "KNOWS"}]

Delete a node

ExecuteQuery[connNeo,"MATCH (n:Person)where n.name='John' OPTIONAL MATCH (n)-[r]-() DELETE n,r;"]

Delete all nodes and relationships

ExecuteQuery[connNeo,"MATCH (n) OPTIONAL MATCH (n)-[r]-() DELETE n,r;"]

Resources

MongoDB  - http://docs.mongodb.org/manual/

Neo4j - http://docs.neo4j.org/

Neo4j online courses - http://www.neo4j.org/learn/online_course

NET/Link - MouseOver

JLink - MouseOver

Thank you!

Copyrighted material Wolfram Technology Group - Wolfram Research Inc.