Mathematica At Enova

Vinod Cheriyan,  Mark Rozpadek
Enova International

Join the Conversation #WolframTechConf

Outline

Overview of Enova

Analytics Engine

High-level Architecture

Some Metrics

XMLMerge

Algorithm

Example

Enova - A Growing Global Online Lender

Our Analytics Team

51 Employees, six teams

24 new hires in the last 12 months, 8 interns

Mix of college hires and experienced analysts

Our teams are responsible for protecting and growing the profitability of the company through data-driven decision-making

Realtime Analytics is the Backbone of our Business

Analytics Engine

Some Metrics

•	Using our old system a model would take on average of about 1 – 1.5 months to deploy into our production environment
•	With Colossus the time has been reduced to 1.5 to 2 weeks to deploy into our production environment
•	Any minor changes or change for sampling percentage would previously take a week or two to deploy and have active
•	With Colossus we can implement these changes in a few hours

Some Metrics (Contd.)

Data from January 1st  2015 to February 12th 2015.  This only includes live decisioning models
•	Average Colossus response time per request: 0.0775867
•	Average time to get variables per request: 0.21593
•	Average requests per hour: 360.8
•	Total requests: 2,680,215
•	Timeouts for variables: 1,186 (this is due to leads data or credit reports)
•	Timeouts for Colossus: 0
•	Average number of variables per request: 32.6
•	Maximum requests per day to date: 53,502

The System Supports High Transaction Volume

With Very Fast Response Time Per Request

XML Merge

Goal of this section

Give an example of how we use Mathematica to solve a business problem.

Give an example of how functional programming can be used effectively.

Business Problem - Derive Structure of Credit Report XMLs

Setting

To assess creditworthiness of customers, Enova obtains credit reports from various credit reporting agencies.

The credit reports are provided as XMLs

We need to parse and store this data into a relational database

For this we, need to create the relational structure

Challenge

For many of the sources, metadata is not available as schema definitions (DSD or XSD)

Metadata is not available as a DSD or XSD

Often times, the metadata is available as PDF or Word documents

It is time-consuming to manually translate these specifications into

Consequently, modelers pick and choose a small subset of data that they can manually parse

Suggested Solution

If we can merge many different XMLs, we can get an approximation of the complete XML.

Solution - Merge XMLs to Create a “Complete” XML

Idea

If we merge a sufficiently large number of XMLs, the resulting XML will be close to the complete XML.

Not fool-proof

There can be an optional element or group of elements that are missed from the sample

But, practical

We can use Mathematica to do the merge

If written carefully, the merge can be really fast

Algorithm

Idea: Merge the XMLs recursively

Input: xml x1, xml x2

Check if x1 and  x2 have the same name; if not display error and STOP

Create an empty merged xml xm

Merge all attributes of x1 and x2 into xm

If x1 and  x2 are both simple, create a corresponding simple node in xm

For all the children elements of x1 and  x2 :

Create merged child element  xc

Determine multiplicity of child element

If muliplicity is 1, create one child element in xm

If mulitiplicity is greater than 1, create M child elements in xm

Example of a Merge

Example XML

Merged XML

XMLs are Stored as XMLElement

Mathematica Uses XMLElement to store XMLs

exampleXML = XMLElement["Accts",{},{XMLElement["Acct",{},{XMLElement["AcctNumber",{"type"->"savings"},{"3333333333"}],XMLElement["Bank",{},{"SomeWhereLand Central Bank"}],XMLElement["Comment",{"type"->"optional"},{XMLElement["Text",{"language"->"English"},{"This is a comment."}],XMLElement["Author",{},{"vcheriyan"}]}],XMLElement["acchistory",{},{XMLElement["lineitem",{"date"->"050114","balance"->"0"},{}],XMLElement["lineitem",{"date"->"060114","balance"->"10"},{}]}]}],XMLElement["Acct",{},{XMLElement["AcctNumber",{"type"->"checking"},{"123412341234"}],XMLElement["Bank",{},{"Bank of SomeWhereLand"}],XMLElement["extid",{},{"123"}],XMLElement["extid",{},{"456"}],XMLElement["acchistory",{},{XMLElement["lineitem",{"date"->"050114","balance"->"124"},{}],XMLElement["lineitem",{"date"->"060114","balance"->"100"},{}],XMLElement["lineitem",{"date"->"070114","balance"->"80"},{}],XMLElement["lineitem",{"date"->"080114","balance"->"90"},{}]}]}]}]

Try 1 - “Procedural” Approach

Timing results based on two implementations - procedural and rules matching

The idea was to translate the above algorithm almost as is into Mathematica

This resulted in a fairly long but readable function

Initial Version

mergeElements[target_XMLElement, source_XMLElement] := 
 Module[{
   targetSimple, sourceSimple, targetChildrenTally, 
   sourceChildrenTally, allChildren, sourceCount, targetCount, 
   nodelist,
   newChildElement,
   mergedElement,
   tempElement,
   i
   },
  (
   mergedElement = Null;
   debug["came into mergeElements with "];
   debug[target];
   debug[source];
   
   (* First Check if node names are the same *)
   If [getName[source] != getName[target], Print["Error!"]];
   
   (* Next, Check to see if we are merging simple-simple or complex-complex *)
   targetSimple = SimpleNodeQ[target];
   sourceSimple = SimpleNodeQ[source];
   If[ sourceSimple ⊻ targetSimple, 
     (Print[
        "Something wrong. Trying to merge simple node and complex node " <> source[[1]] <> " " <> target[[1]] ];
       mergedElement = Null;
       Return[mergedElement];
       )
    ];
   
   (* Start creating the merged element *)
     mergedElement = createElement[getName[source]];
     debug["creating mergedElement " <> ToString[mergedElement]];
   
   (* First, merge all attributes *)
     debug["list of attributes to merge :"];
     debug[Union[getAttributes[target], getAttributes[source], 
        SameTest -> (getAttributeName[#1] == getAttributeName[#2] &)] ];
     
     mergedElement = Fold[
       addAttribute[#1, #2] &,
       mergedElement,
       Union[getAttributes[target], getAttributes[source], 
        SameTest -> (getAttributeName[#1] == getAttributeName[#2] &)]
       ];

   If[sourceSimple && targetSimple,
    (
    	debug["Merging two simple nodes"];    
     	mergedElement = addValue[mergedElement, getValue[source]];
     ),
    (
   
     (* Process the children *)
     targetChildrenTally = Tally[getChildrenNames[target] ];
     	debug["targetChildrenTally: " <> ToString[targetChildrenTally] ];
     sourceChildrenTally = Tally[getChildrenNames[source] ];
     	debug["sourceChildrenTally: " <> ToString[sourceChildrenTally] ];
     allChildren = 
      Union[targetChildrenTally[[All, 1]], 
       sourceChildrenTally[[All, 1]] ];
      debug["allChildren " <> ToString[allChildren]];
     For[i = 1, i <= Length[allChildren], i++, (
       debug["Main for loop i = " <> ToString[i]];
       sourceCount = 
        Length[getChildrenNames[source, allChildren[[i]]]];
       targetCount = 
        Length[getChildrenNames[target, allChildren[[i]]]];
       debug[{sourceCount, targetCount}];
       Which[
        sourceCount == 1 && targetCount == 1,
        (
         debug["1 to 1 merge : " <> allChildren[[i]] ];
         mergedElement = addChild[mergedElement,
           mergeElements[
            Select[target[[3]], #[[1]] == allChildren[[i]] &][[1]],
            Select[source[[3]], #[[1]] == allChildren[[i]] &][[1]]
            ]
           ];
         ),
        Max[sourceCount, targetCount] > 1,
        (
         debug["one to many merge : " <> allChildren[[i]] ];
         nodelist = Join[
           Select[target[[3]], #[[1]] == allChildren[[i]] &],
           Select[source[[3]], #[[1]] == allChildren[[i]] &]
           ];
         newChildElement = mergeAll[nodelist];
         mergedElement = addChild[mergedElement, newChildElement];
         mergedElement = addChild[mergedElement, newChildElement];
         mergedElement = addChild[mergedElement, newChildElement];
         mergedElement = addChild[mergedElement, newChildElement];
         ),
        True ,
        (
         (* This means exactly one of (sourceCount, targetCount)=
         0 and the other is 1 *)
         debug["copy from source or target : " <> allChildren[[i]] ];
         tempElement = If[sourceCount == 0, 
         					Select[target[[3]], #[[1]] == allChildren[[i]] &][[1]],
         					Select[source[[3]], #[[1]] == allChildren[[i]] &][[1]]
         	];
         mergedElement = 
           addChild[mergedElement, mergeElements[tempElement, tempElement]
            ];
         )
        ]
       )];
     )];
   Return[mergedElement];
   )
  ]

Timing - Not Very Good

Timing[Do[mergeElements[exampleXML,exampleXML];, {10}]][[1]]/10

Try 2 - “Pattern Matching” Approach

Idea: Mathematica is very good with pattern matching. 
So why not make use of that?

For example:

Rather than explicitly check if we are trying to merge two elements of the same name, define it in the pattern

me[XMLElement[s_,attr1_, val1_], XMLElement[s_,attr2_, val2_] ]:= XMLElement[s, me[attr1,attr2], me[val1,val2]]

Define new patterns for merging attributes and values

me[s:{__Rule}, t:{__Rule} ]:=  Union[s,t, SameTest -> (#1[[1]] == #2[[1]] &)]
me[{s_String}, {t_String}] := {s}   (* note: The merged XML only requires a placeholder *)

The main function is now much cleaner:

me[s:{__XMLElement}, t:{__XMLElement}]:= Module[{stally, ttally, allchildren,
	scounts, tcounts, groupedAllChildren,
	childrenList, counttype, childrentally, newChildrenList,
	i},(
	
	stally = GatherBy[s,First];
	ttally = GatherBy[t,First];
	
	Map[(scounts[#[[1,1]]] := Evaluate[ Dimensions[#][[1]] ]) &, stally];
	Map[(tcounts[#[[1,1]]] := Evaluate[ Dimensions[#][[1]] ]) &, ttally];
	scounts[x_]:= 0;
	tcounts[x_]:= 0;
	allchildren = Union[ stally[[All,1,1]], ttally[[All,1,1]] ];
	groupedAllChildren = GatherBy[Join[s,t], Max[{scounts[#[[1]]], tcounts[#[[1]]]}]&];

	
	newChildrenList= {};
	
	For[i = 1, i <= Length[groupedAllChildren], i++, (
		childrenList = groupedAllChildren[[i]];
		childrentally = GatherBy[childrenList, First];
		
		counttype = Max[{scounts[#], tcounts[#]}]& [childrenList[[1,1]] ];
	
		
		newChildrenList = Join[newChildrenList, Switch[counttype, 
			1, Flatten[Map[me, childrentally]],_, Flatten[ConstantArray[Flatten[Map[me, childrentally]], 4] ] ]
		];	
				
	)];

Revised Version

The command now runs much faster!

Timing[Do[mergeElements[exampleXML,exampleXML];, {10}]][[1]]/10

Timing[Do[me[exampleXML,exampleXML];, {10}]][[1]]/10

Summary

Enova’s Analytics Engine, Colossus, runs on Mathematica

It supports our day-to-day operations efficiently and effectively

We also do some cool programming in Mathematica