Magic and Cognitive Dissonance: 
One-Touch Data Modeling

Katya Vladislavleva and Mark Kotanchek
Evolved Analytics

<<DataModeler`

Abstract

After a decade of development and industrial application, we are closing in our goal of one-touch conversion of real-world data (unbalanced, correlated, missing elements, etc. — i.e., ugly) into meaningful and insightful driver variable identification and models. Without requiring a Ph.D. in Computer Science or Statistics, it is now possible to easily develop robust nonlinear models (complete with trust measures), identify data outliers and interactively explore the model dynamics and response sensitivities.

Our presentation will use DataModeler to illustrate the ease and power of conversion of spreadsheet data into an interactive data story report (aka, computational document) using examples drawn from economics, wind turbine power generation and industrial process monitoring.

Data-driven Modeling and Cognitive Dissonance

trivialModels=SelectModels[
RobustModels@SymbolicRegression[input,response,TimeConstraint ->2, Parallelization-> False, IndependentEvolutions-> 2],
SelectionStrategy -> AllModels,
QualityBox -> {60,All}
];

trivialModels=RetrieveModelSets[ProjectName-> "trivialModels"];input ={1,2};
response ={1,2};allData = MergeInputResponseData[input,response];
byproductFolder=ToFileName[{$DataModelerExampleDataDirectory,"Soft_Sensor"}];parameterNames={"density","rate","flowA","vacuum","temp1","weight","flowB","temp2"}; responseName="byproduct";allNames=Flatten[{parameterNames,responseName}]; {colLabels,allTrainingData}=ImportDataMatrix["Byproduct_Training.xls",Directory->byproductFolder];
trainingResponse=(Subscript[allTrainingData, [[All,-1]]]);
Dimensions[trainingInputs=Most/@allTrainingData];
{colHeadersGDP,dataMatrixGDP} = ImportDataMatrix[
Import[FileNameJoin[{EvaluationNotebookDirectory[],"Predicting_GDP_Per_Capita_70_Countries_Full_list.xls"}]
][[1]],
DeleteNonNumericRows->0.5
];
defaultModelsGDP=SelectModels[RetrieveModelSets[ProjectName-> "defaultModelsGDP"], QualityBox-> {80,0.21}];
defaultEnsembleGDP=RetrieveModelSets[ProjectName-> "defaultEnsembleGDP"];
pPlotGDP=ParetoFrontPlot[defaultModelsGDP,PlotLabel->LabelForm["Alternative GDP per Capita Models",FontSize-> 16], ImageSize-> 320, PointSize-> 0.02];
vPlotGDP=VariablePresenceMap[defaultModelsGDP, AspectRatio-> 0.8,PlotLabel->LabelForm["Variable Presence in the models",FontSize-> 16]];
predictionGDP=Show[
ResponseSurfacePlot[CreateModelEnsemble[defaultEnsembleGDP, EnsembleStrategy-> AllModels],GraphicsArrayColumns->None,PlotRange-> ({0,Max@#}&@(Subscript[dataMatrixGDP, [[All,2]]])),
PlotStyle->{Lighter@Yellow,Specularity[White,20],Opacity[0.6]}],
ListPointPlot3D[(Subscript[dataMatrixGDP, [[All,{4,9,2}]]]),PlotStyle->{{Red,PointSize@Large}}],
PlotRange->All, ImageSize-> 350
];

TabView[{"Problem Setup"->Show[Import["/Users/katya/Documents/Reading & Learning/Presentations/My Presentations/2011WTC/graphics/BB.pdf"][[1]],ImageSize-> 700],"Goal"-> Show[Import["/Users/katya/Documents/Reading & Learning/Presentations/My Presentations/2011WTC/graphics/modeling.pdf"][[1]],ImageSize-> 600],
"Myth 1"-> 
GridTable[{{
GridTable[
ModelPhenotype[
trivialModels[[1;;14,All]]
],
Label -> LabelForm@"Some of the discovered models", RowLines-> False
],
Show[
ResponseSurfacePlot[trivialModels,DataVariableRange ->{{0,3}},GraphicsArrayColumns->None],
ListPlot[
allData,
PlotStyle-> {Red, PointSize[0.04]}
],
PlotLabel -> LabelForm@"A diverse set of models can exactly fit the data",
ImageSize -> 350, PlotRange-> {{0,3},{0,3}}
]
}},RowLines->None,ColumnLines->False,
Label -> LabelForm["There should be THE model...",FontSize -> 24,FontColor->Blue]
],
"Myth 2"-> GridTable[{{
UnivariatePlot[allTrainingData,DataVariableLabels->allNames,PlotLabel->LabelForm["Industrial Measurements of an Inferential Sensor",FontSize->16]],
CorrelationMatrixPlot[allTrainingData,DataVariableLabels->allNames,PlotLabel->LabelForm["VariableCorrelation",FontSize-> 16],ImageSize-> 300]
}},RowLines->None,ColumnLines->False,
Label -> LabelForm["Variables are uncorrelated...",FontSize -> 24,FontColor->Blue]
],
"Myth 3"-> GridTable[{{
ModelSelectionTable[
defaultEnsembleGDP , Label -> LabelForm["Ensemble Models Predicting GDP per Capita",FontSize -> 16]]},
{predictionGDP}},RowLines->None,ColumnLines->False,
Label -> LabelForm["The world is a polynomial...",FontSize -> 24,FontColor->Blue], ImageSize-> 300
],
"Myth 4"-> GridTable[{{
pPlotGDP,
vPlotGDP
}},RowLines->None,ColumnLines->False,
Label -> LabelForm["All variables matter...",FontSize -> 24,FontColor->Blue], ImageSize-> 200
]
}, Alignment-> {Center, Top}
]

Robust Data Modeling Technology is here

If there is information in the data we find it

Show[Import["/Users/katya/Documents/Reading & Learning/Presentations/My Presentations/2011WTC/graphics/dataModelsHeader.pdf"][[1]],ImageSize-> 800]

Some of the current projects

- Pandemics analysis and prevention strategies,

- Customer satisfaction prediction,

- Structure - property relationships,

- Economic modeling,

- Traffic Safety and Accident prediction,

- Wind power output prediction.

Definitions of success

The minimal subspace of variables explaining the response is identified

Response is predicted reliably and prediction quality is quantified

Decision tradeoffs in model complexity and prediction accuracy are proposed to the problem owner

Domain expert says: “Hmm.. That looks easy..”, or “Ha! Of course! It is obvious.”

Models have been deployed and have been bringing money since.

Success is in the Workflows

Today we will not be talking about collecting the data for data-driven knowledge discovery - it is an art on its own, but we will focus on the technology of retrieving insights from given numeric data and doing it magically with one touch.

TabView[{"Success is in integration"-> Show[Import["/Users/katya/Documents/Reading & Learning/Presentations/My Presentations/2011WTC/graphics/colorWheelWide.pdf"][[1]],ImageSize-> 700] ,
"Problem Solving is Iterative"-> Show[Import["/Users/katya/Documents/Reading & Learning/Presentations/My Presentations/2011WTC/graphics/ProblemSolvingCycle.pdf"][[1]],ImageSize-> 400],
"Data Science is a system"->Show[Import["/Users/katya/Documents/Reading & Learning/Presentations/My Presentations/2011WTC/graphics/DataScienceDiagram.pdf"][[1]],ImageSize-> 400]},  Alignment-> Center]

Magic and One-touch Data Modeling

Data-driven problem solving became a routine

TableForm[{{
Show[Import["/Users/katya/Documents/Reading & Learning/Presentations/My Presentations/2011WTC/graphics/plug-and-play.pdf"][[1]],ImageSize-> 270],Show[Import["/Users/katya/Documents/Reading & Learning/Presentations/My Presentations/2011WTC/graphics/magic.pdf"][[1]],ImageSize-> 330]
}}]

Example : Wind Energy Output Prediction

Joint project with Tobias Friedrich (MPI Saarbruecken, Germany), Frank Neumann and Markus Wagner (University of Adelaide, Australia)

Load the data & labels (October 2010-June 2011)

Import the data from disk

{myDataLabels,myDataMatrix}=ImportDataMatrix["windEnergyDataFull.xls"];
Dimensions@myDataMatrix

Explore variable labels of all data columns

myDataLabels

Explore the Data

SummaryStatistics

GridTable[SummaryStatistics@N@myDataMatrix,
Label->"Summary Statistics",
TableHeadings ->{myDataLabels,{"min","mean","max"}}
]

UnivariatePlot and BivariatePlot

UnivariatePlot[myDataMatrix,PlotLabel->None,
DataVariableLabels ->myDataLabels,ImageSize->600,AspectRatio->1]

DataDistributionPlot

DataDistributionPlot[myDataMatrix,PlotLabel->"DataDistributionPlot
 of Data Columns",DataVariableLabels->myDataLabels,ImageSize->500]

CorrelationMatrixPlot

As it should, the constant columns are automatically deleted.

CorrelationMatrixPlot[
myDataMatrix,
PlotLabel->LabelForm@"CorrelationMatrixPlot of Data Columns",
DataVariableLabels ->myDataLabels, ImageSize-> 450
]

The Linear Reference (what if the problem is trivial?)

Create input response data

{myInputs,myResponse}=SplitInputResponseData[myDataMatrix];

inputLabels=Most@myDataLabels

Create Linear Models

ModelSelectionReport[
linearModel = CreateLinearModel[
myInputs,N@myResponse,
DataVariables -> inputLabels
]
]

This is already a reasonable model

ModelPredictionPlot[
linearModel,
myInputs,N@myResponse, ToolTipFunction-> None
]

Let’s try to create linear models using polynomials of higher degrees (models are linear in the coefficients):

ParetoFrontPlot[
linearModelsOfHigherOrders=CreateLinearModel[
myInputs,myResponse,
DataVariables -> inputLabels,
PolynomialOrder -> #,
DataVariableRange-> SummaryStatistics@myInputs
]&/@Range[3]
]

Develop Models

Timing[
ParetoFrontPlot[testModels=SymbolicRegression[
 myInputs,N@myResponse,
FunctionPatterns-> BuildFunctionPatterns["ExtendedMath"],
DataVariables -> inputLabels,
StoreModelSet -> True,
ProjectName-> "windAllModels",
TemplateTopLevel-> BuildFunctionPatterns[Plus],
TimeConstraint-> 2000,
IndependentEvolutions -> 10
]
]
]

Load Developed Models

archivedModels = RetrieveModelSets[
ProjectName-> "windAllModels"
];
ParetoFrontPlot[
#, 
PlotLabel->LabelForm[{Length@archivedModels,"models archived"},Joined->True]
]&@archivedModels

Min[ModelQuality[archivedModels][[All,2]]]

ParetoFrontPlot[robustModels=RobustModels[archivedModels, QualityBox-> {250,0.4}]]

Extract Variable Importance

Let’s analyze variable presence and variable contribution in developed models:

VariablePresenceMap[archivedModels, AspectRatio-> 1.5]

VariablePresenceTable[archivedModels, QualityBox-> {All,0.3}]

reducedModels=RetrieveModelSets[ProjectName-> "windReducedModelsAllVars"];

{labels,myVariableContribution}=ImportDataMatrix["windVariableCont.xls",Directory-> EvaluationNotebookDirectory[]];

BoxWhiskerChart[100*Transpose@myVariableContribution, ChartLabels-> labels, AspectRatio-> 2, BarOrigin-> Left,ChartStyle-> Red, PlotLabel-> "Variable Conribution to the relative Error,%"]

Variable Combination Analysis

GridTable[
VariableCombinations[reducedModels],
TableHeadings-> {Automatic,{"Percent","Fraction","Cumulant","Count","Drivers"}}
]

VariableCombinationMap[reducedModels]

VariableCombinationTable[SelectModels[reducedModels, QualityBox-> {All,0.4}, SelectionStrategy-> AllModels]]

Validate and Test Models (July 2011)

Create Model Ensembles

ModelSelectionTable[quickEnsemble=CreateModelEnsemble[reducedModels,myInputs,myResponse,QualityBox -> {All,0.4},
AllowedVariables-> DriverVariables[reducedModels,0.335]]]

EnsemblePredictionPlot[
quickEnsemble,
myInputs,myResponse, ToolTipFunction-> None, EnsembleDivergenceFunction-> StandardDeviation]

Load Test Data (from July 2011)

{testDataLabels,testDataMatrix}=ImportDataMatrix["WindEnergyTESTData.xls"];Dimensions@testDataMatrix

{myTestInputs, testResponse}=SplitInputResponseData[testDataMatrix];

EnsemblePredictionPlot[
quickEnsemble,
myTestInputs,testResponse, ToolTipFunction-> None, EnsembleDivergenceFunction-> StandardDeviation]

Validate Solutions on unseen Data

ensemblePrediction=EvaluateModel[quickEnsemble,myTestInputs];ListLinePlot[{ensemblePrediction,testResponse}, AspectRatio-> 0.25, PlotStyle-> {Red, Blue}, PlotLabel-> LabelForm["Predicted (Red) vs. Observed (Blue) Energy Output over Time in July (Test Data)", FontSize-> 14], ImageSize-> 600]

ResponseSurfacePlot[quickEnsemble,DataVariableReference -> ModelMaximum]

ResponsePlotExplorer[quickEnsemble,
RangeExpansion-> {0.8,1.2},DataVariableReference -> ModelMaximum, ImageSize-> 350, ControlPlacement-> Left]

Perform Model - based Outlier Detection

Joint work with Philip Truscott (Singapore University of Technology and Design, Singapore)

GDP vs Taxes

Models were developed using four short symbolic regression runs

Quiet@ParetoFrontPlot[
defaultModels = SymbolicRegression[
dataMatrix,
TargetColumn-> 2,
ExcludedVariables->{8},
DataVariables->colHeaders,
TimeConstraint->30,
IndependentEvolutions-> 4,
StoreModelSet-> True, 
ProjectName-> "defaultModelsGDP"
],
PlotRange->Automatic
]

defaultModels=RetrieveModelSets[ProjectName-> "defaultModelsGDP"];VariablePresenceMap[defaultModels,QualityBox -> {100,0.2}, AspectRatio-> 1.2]

VariableCombinationMap[defaultModels,QualityBox -> {100,0.2}]

ModelSelectionReport[
defaultEnsemble = CreateModelEnsemble[
defaultModels, dataMatrix,
AllowedVariables->DriverVariableCombinations[defaultModels,1,QualityBox -> {50,0.1}],
QualityBox -> {50,0.1}
]
]

Show[
ResponseSurfacePlot[defaultEnsemble,GraphicsArrayColumns->None,PlotRange-> ({0,Max@#}&@(Subscript[dataMatrix, [[All,2]]])),
PlotStyle->{Lighter@Yellow,Specularity[White,20],Opacity[0.6]}],
ListPointPlot3D[(Subscript[dataMatrix, [[All,{4,9,2}]]]),PlotStyle->{{Red,PointSize@Large}}],
PlotRange->All
]

EnsemblePredictionPlot[defaultEnsemble,dataMatrix,ToolTipFunction->((Subscript[dataMatrix, [[#,1]]])&)]

EnsembleResidualPlot

It looks like we are capturing the behavior pretty well.

EnsembleResidualPlot[defaultEnsemble,dataMatrix,
ToolTipFunction->((Subscript[dataMatrix, [[#,1]]])&)]

DataOutlierAnalysis

Although outliers, these are not “far outlers” (which would have a distance greater than 3).

DataOutlierTable[defaultEnsemble,dataMatrix]

Conclusions

One touch data modeling works

It frees thinking time for researchers and proposes non - intuitive testable and falsifyable hypotheses

PhD in statistics or computer science is no longer a requirement to do data driven modeling

Intelligence, common sense, and constructive reasoning are still required for interpreting results (this is an art)

What do we want to change

Random designs, "One variable at a time"

Fundamental assumptions on model structure

Fundamental assumptions on linearity of relationships

(Ab)use of methods for solving problems they were not designed for (e.g. highly non-linear relationships, correlated inputs, etc)

When potentially successful projects are closed because they do not produce revenue immediately, or because results cannot be generated in a high-throughput fashion

When money is left on the table

When the experts and modelers spend their time on doing ill-structured labour-intensive  work for modeling data while they could be solving world' s greatest challenges

Decision paralysis among modelers (when the number of alternative technologies is overwhelming - people tend to fall back to easy conventional methods, e.g.one variable at a time, or linear regression)