Trustworthy Data Models

DataModeler and Real-World Modeling

Mark Kotanchek

Evolved Analytics LLC

Abstract

Data modeling is (generally) hard.

Using developed models in the real-world is nerve-wracking.

DataModeler makes life better: automatic variable selection, appropriate complexity models, human insight and model prediction trustability metrics during use.

Demonstration focus today will be the use of model trustability to guide data collection for adaptive data collection for design-of-experiments and emulator development.

We will demonstrate a subset of the DataModeler capabilities along the way.

The Official Abstract

Data modeling is intriniscally hard: What is the relative importance of the potential input variables? What variable combinations are important? What does the model mean? Does the model capture the fundamentals or is it chasing perturbations caused by noise?

The problem gets even worse when it becomes time to use the models. There is an implicit assumption that the the future looks just like the past — i.e., the modeled system has not changed its fundamental behavior and the model is not being asked to predict in new operating regions. Unfortunately, real-world application will likely violate these assumptions. This reliance on the future behavior closely resembling the past behavior is analogous to driving a car using only the information from the rear-view mirrors as guidance! Similarly, even though using a data-driven model may be necessary or better than the alternatives, there is a certain amount of trepidation in that use.

The DataModeler add-on package for Mathematica helps to take some of the risk and trepidation out of empirical modeling. Using state-of-the-art genetic programming algorithms to perform symbolic regression, compact and human-interpretable models can be rapidly developed which automatically identify and exploit the driving variables during the process of exploring the risk-reward trade-off of model complexity vs. model accuracy. The net effect is that we let the data tell us what accuracy is possible and appropriate and mitigate the risk of using overly complex and fragile models or incorporating spurious variables.

Because the developed models will minimize nuisence variables, are at appropriate levels of complexity and are human interpretable, they are intrinsically more trustworthy than models developed using most other data modeling techniques. Furthermore, because the symbolic regression processing explores a plethora of possible model structures, we can use ensembles of diverse models to develop trustable models and trust metrics — the ensemble will be constrained to agree if during use the system hasn't changed and it is not being asked to extrapolate into new regions of parameter space. If these two core assumptions are violated the models will be constrained to disagree — thereby providing awareness that the model prediction should be regarded with suspicion.

In this talk we will demonstrate the DataModeler package with a special emphasis on the development of symbolic regression models for human insight and understanding and ensembles for real-world deployment. Along the way we will feature interface improvements made possible by the migration to Mathematica 6 as well as discuss issues associated with model development, selection and management.

What we want …

<<DataModeler`

Show[
Import[ToFileName[EvaluationNotebookDirectory[],"Road Picture.jpg"]],PlotLabel->LabelForm["We WANT reliable and accurate predictions",FontColor->Darker@Orange,FontSize->24],ImageSize->600]

Presentation Notes

The goal for any model is to capture reality. If this is done, the model can be used to predict or optimize as well as provide insight into the targeted system be that physical, chemical, financial, biological or cultural.

The reality is …

The windshield has been painted over … AND we are making an assumption that the road ahead looks like the road behind!

Show[Import[ToFileName[EvaluationNotebookDirectory[],"RoadRearview.jpg"]],PlotLabel->LabelForm["We have ONLY the past",FontColor->Darker@Orange,FontSize->24],ImageSize->800]

Presentation Notes

Unfortunately, the caveat that "past performance does not guarantee future results" applies since the model is only as good as the assumptions and data used in its development. If the modeled system has changed or if the model is being used in previously uncharted areas, the model could be wrong — possibly wildly and dangerously so!

Unique aspects …

Focus — automatic variable selection

Insight — human-interpretable models

Trust — prediction reliability metrics

Today…

Very simple example — illustrate trust metric behavior & exploitation

See other publications for applications to real-world problems

Presentation Notes

Symbolic regression lets the data define the structure and complexity of the model rather than imposing an a priori assumption that the system consists of polynomials or cascades of sigmoid functions. Switching from a parameter optimization to a search for structure AND associated parameters greatly increases the difficulty of the modeling effort. However, our algorithms are more than 3-orders-of-magnitude faster than generic genetic programming implementations which has enabled their application to real-world problems having hundreds of variables and tens of thousands of data records — as well as smaller data sets.

Automatic variable selection lets us attack fat data arrays which possibly include correlated and coupled variables. One of our early success stories was the discovery that the ratio of two (highly correlated) temperature probes was an excellent predictor of a hard-to-measure-directly variable in a chemical reactor. Millions of dollars had been spent in the development of a direct sensor; however, the indirect inferential sensor based upon the easily measured and robust temperatures has been operational for over a decade now.

Symbolic regression models can be viewed as a form of high-dimensional visualization. Placed in front of a domain expert, these algebraic expressions can provide cues to underlying mechanisms as well as important variable relationships. As such, it is a discovery engine which augments rather than replaces the expert to guide future research and first-principles model development.

For a finite data set, there are an infinite number of models which will summarize the response behavior. We can use an ensemble of diverse "good" models to guide us as to whether the ensemble prediction is trustworthy since the diverse models will be constrained to agree where there is data and will tend to diverge when in extrapolating into unknown regions. Because diverse model structures are explored during the symbolic regression processing, it is easier to achieve the required diversity in ensemble models. Knowing  when NOT to trust the model can be very important!

The presentation time constraints mean that today's example will be quite simple and illustrate only a subset of the DataModeler capabilities. For additional examples, see the other publications of the author from prior Wolfram Technology Conferences as well as in the evolutionary computing technical literature. Additionally, the package help documentation contains a number of case studies showing the application to real-world data sets.

Motivations

Problems with model building …

Is the data accurate and noise-free?

Is the data complete?

Which variables are important? Which variables are spurious? Are there neglected variables?

Are variable interactions important? Are the variables coupled or correlated?

What is the proper model structure? What is the proper model complexity?

Is the model appropriate and reasonable?

Problems with model use …

Is the model valid?

Has the system changed behavior?

Am I in a new operating region?

Do I feel lucky?

Presentation Notes

As summarized in the above lists, there are issues with both model use as well as the subsequent use.

The problems start with the data since the goal of highly accurate, balanced and complete is rarely achieved in real-world modeling. The objective of balanced data is driven by the model assessment process since an over-representation in one region of parameter space can result in selecting models with high performance in that region at the expense of global accuracy over the entire operating region. Similarly, gaps in parameter space mean that models are freed from constraints in those regions and, therefore, predictions in those regions can be incorrect without an ability to detect that error. (As we shall see, the use of diverse ensembles allow us to define a trust metric to detect when the model output should not be trusted.)

The next issue with the data is selecting the variables to be used in the model building. If the actual model variables are not available (or cannot be constructed using the available variables) then the modeling effort is pointless. Unfortunately, for many situations we do not know the true driving variables — although we may have suspicions. Spurious correlations are possible which, especially for small data sets, may lead us to conclude that the number of chickens in Ethiopia is the best predictor of the S&P 500. In general, inclusion of spurious variables will make the modeling result less reliable. For this reason the DataModeler ability to accomodate correlated variables and automatically select driving variables is VERY significant.

One side effect of being able to naturally handle correlated and coupled variables is that variable combinations — aka, metavariables — can be discovered and identified as more natural building blocks. Discovery of such variable combinations can give insight into the system being modeled as well as improve the modeling efficiency.

Letting the data define the model structure is also very important. Of course, for typical data we have noise and other error sources so we do not want to chase those noise contributions; rather, we want the model to summarize the response behavior. Our intuition, therefore, tells us that we want to satisfy multiple objectives:

accuracy — obviously, we want the model to match the observed behavior and

simplicity — parsimonious expressions will generally be better behaved and more stable than ones which are much more complicated.

There are actually additional criteria we might want; however, if we can develop an accurate but simple model we will be more comfortable in using the model as well as gaining insight into the model characteristics. That insight into the constituent variables and model structure and complexity as well limit behavior helps us to select reasonable and appropriate models for deployment.

Capturing the value of a model often involves deploying it against new data. There are two basic risks in using a model in the real-world:

system changes — empirical model building makes an implicit assumption that the future behavior will resemble the past behavior used to develop the model. Physical systems often change over time either slowly (e.g., an injection nozzle wearing) or abruptly (e.g., a steep jump in the price of natural gas). If the change is large enough, then it can be financially or physically dangerous to use the now inappropriate model.

extrapolation — if the data used in the model development does not cover the full operating range or there are internal gaps in the data, the model predictions could be wrong when it is asked to operate in new regions. Unfortunately, if there are many variables in the model, detecting that operating region is new and novel is quite difficult.

While there is no such thing as a guarantee, using an ensemble of diverse, high-quality and simple models decreases the risk of using a model since these models will agree when they are in known operating regions (otherwise, they would not be good models) but will tend to disagree when in new regions or if the system has changed (otherwise, they would not be diverse). This is a VERY powerful and unique capability.

Why use models — especially empirical models?

No better option?

No choice — Despite all of the flaws and uncertainties, we often have no better alternative.

Modeling Applications

Show[Import[ToFileName[EvaluationNotebookDirectory[],"02 Industrial Applications.pdf"]],PlotLabel->LabelForm["The application changes the modeling strategy",FontColor->Darker@Orange,FontSize->20],ImageSize->750]

Today's Focus: Adaptive Data Gathering

The help documentation contains additional case studies applying DataModeler against industrial data sets for a variety of applications.

Presentation Notes

People LIKE models and can often see patterns where none exist. Thus, the ancients cast horoscopes and studied the entrails of chickens to gain a prediction advantage. People will also become emotionally invested in their choice of model as witnessed by the discord between the technical traders and the fundamentalists in the financial realm. Similarly, the sciences are populated with researchers who embrace models and forget the significance of the simplifying assumptions used in their development. (See The Future of Everything by David Orrell for some entertaining examples.)

Visual and simple models ARE very important; however, they need to be constrained by data and the models and the data iteratively refined as required since if the theory doesn't fit the facts, either the theory or the data (or both) are wrong. Symbolic regression offers a bridge between the theory and the data to enable research acceleration since driving variables, variable relationships and candidate model structures can be identified and used to give insight and guide first-principles model development.

Although symbolic regression is a nonlinear modeling technique, linear statistics has a long history of success and many accepted mathematical tools for model quality and prediction assessment. Identifying the important variables and variable transforms can convert a data set into a less nonlinear system which is tractable using conventional techniques. Such preprocessing also can help other nonlinear techniques such as neural networks and support vector machines to generate more accurate and robust models.

We are often unable to measure a desired variable directly or without a substantial processing delay. In these situations, we might use a model as an inferential sensor to estimate the desired variable in a timely fashion using easily available inputs.

In a similar fashion, we might have a complex system which we want to optimize or understand in which data points are precious and difficult to acquire. This could be a physical system (e.g., a circuit or product flow or a complicated chemical reaction) or even another model (e.g., a complex first-principles or finite-element-analysis model). If we build an emulator of such a system, we can gain insight into the system as well as enable coarse optimization. This surrogate for the targeted system can help speed the desired system optimization since it can be optimized far more quickly than the source system.

Nonlinear design-of-experiments (DOE) exploits the notion of an emulator to iteratively develop models as new data becomes available. Using these models, we can do two things:

optimize where we search for the desired response AND

reduce the model uncertainty by also collecting data where the developed models diverge.

Thus, given some initial data samples, we can efficiently collect data to both maximize our system performance and maximize our confidence in the developed models. Efficient data collection is important if experiments are expensive or if we have a large search space. (The inspiration for this concept was work that Steven Asprey had done for optimal design of experiments is selecting between chemical reaction kinetics models — although he did not evolve the model structures.)

The focus of this presentation will be a subset of the nonlinear DOE application as we adaptively add data points to build models with confidence over an operating range.

What is DataModeler?

The Big Picture

"photo from African Wildlife Federation: http://www.awf.org/section/gallery_detail?id=653";

Hyperlink[Tooltip[Labeled[
Show[Import[ToFileName[EvaluationNotebookDirectory[],"elephants.jpg"]],PlotLabel->LabelForm["Powerful, Fast, Flexible, Precise, Intelligent, …",FontColor->Darker@Gray,FontSize->24],ImageSize->800
],
LabelForm["The choice of descriptor depends upon your needs & perspective",FontColor->Darker@Gray,FontSize->24]
],"Photo from African Wildlife Federation — www.awf.org; \nPhoto by James Weis/eyesonafrica.net"],"http://www.awf.org/section/gallery_detail?id=653"]

The Technology View

State-of-the-art multi-objective genetic programming algorithms

Orders-of-magnitude faster than vanilla GP

Developed against real-world industrial applications and needs

The User View

Ease-of-use: modeling lifecycle tools

Automatically select driving variables, variable combinations and explore the accuracy-complexity trade-offs

Ability to model fat data arrays

Fast & efficient modeling

Robust & human-interpretable models

Diverse ensembles for trustable model use

The Corporate View

Efficient & robust modeling: reducing the need for gurus

Efficient custom system development: building upon Mathematica facilitates rapid development of targeted applications — either web-based or via Mathematica Player.

Presentation Notes

What is an elephant? In the classic story of blind men exploring an elephant, it is a rope, snake, tree trunk, fan, etc. Similarly, much like Mathematica, the definition of the DataModeler depends upon the needs and perspectives of the viewer.

It is tempting to go into excruciating detail on the technology aspects and describe exactly how and why our algorithms have produced over a three order-of-magnitude boost (relative to the generic GP algorithms typically used) for the symbolic regression which is the heart of DataModeler. It is also tempting to outline the pipeline of research ideas being explored for future inclusion. However, we have published extensively (especially at the Genetic Programming Theory & Application Workshops which annually gathers the leaders in the GP field as well as via GECCO and CEC tutorials) so, if you are interested, those references are available. The key message is that the algorithms are state-of-the-art, constantly being improved and have been testing in the crucible of industrial application. We firmly believe that toy application lead to a toy level of understanding; hence, the application to real-world data sets and industrial needs for timely analysis and documentation has been critical.

Since we also use DataModeler, we have a vested interest in making the model development as pain-free as possible. The target is to enhance rather than replace the user. Towards that end, in addition to the core model development capabilities, we also provide the infrastructure for model management, data exploration and model exploration.

From a corporate standpoint, these all fall into the nice-to-have category. The critial aspect, however, is the efficiency of conversion of data into units of dollars. In addition to allowing the user to be efficient, we can also exploit the Mathematica infrastructure — such as webMathematica and gridMathematica — to build focused automated and semi-automated systems which reduce the need for inimate knowledge of Mathematica.

Today's Example: Adaptive Design-of-Experiments

Goal: Adaptively collect data to understand & model a system

We will illustrate

Symbolic regression modeling

Assembly of diverse model ensembles for trustable modeling

Using ensemble divergence to guide future data collection

Adaptive DOE applications — whenever data collection is precious

Emulator (system surrogate) development

Combinatorial chemistry

Operational process optimization

Caveats

1-D example for ease of illustration (n-D is more realistic in real-life)

VERY limited CPU effort (real-life would skew the effort expended to the CPU rather than the human)

Will NOT illustrate (even though they are DataModeler capabilities)

Automatic variable selection

Model development for human insight

Model management capabilities

Data managment & exploration capabilities

Genetic engineering capabilities to create, select & refine models

Presentation Notes

We have addressed most of these points previously. Suffice to say that due to time restrictions we will not cover the full suite of capabilities and we will use a very simple demonstration suitable for that time constraint.

The Target System

Define a structure and explore possible parameter settings

Manipulate[ ] is really neat to interactively explore a function definition to generate something interesting

Manipulate[Plot[(a/x)+b(Sqrt[x])-c (x^2),{x,x0,xf}],
{{a,3},0,10},
{{b,5},0,10},
{{c,0.1},0,5},
{{x0,0.5},0,3},
{{xf,6},2,10}
]

Chosen response generating function

func1Var=Block[
{a,b,c,x},
Function@@{(a/x)+b(Sqrt[x])-c (x^2) /. {x->#1,a->3,b->5,c->0.1}}
];
func1Var[x]

As an aside, the ModelComplexity[ ] of the generating expression is....

ModelComplexity@CreateModelFromExpression@func1Var[x1]

Presentation Notes

The demonstration function is, obviously, quite simple despite being quite nonlinear. Given enough time, the symbolic regression would probably discover the exact solution; however, we will be restricting our modeling to very limited CPU expenditures rather than what would be more typical.

The starting data

Notice that we have two clumps of data with external and internal regions of missing data.

Dimensions[Block[
{numVars=1,
numPts=1,
sampleRange={{1,1.5},{1.5,2},{4,5},{5,6}}
},
data1=Join@@(RandomReal[#,{numPts,numVars}]&/@sampleRange)
]];
response1=func1Var@@@data1;

Show[Plot[func1Var[x],{x,0.5,7}],ListPlot[Transpose@{(Subscript[data1, [[All,1]]]),response1},PlotStyle->{Red,PointSize[0.02]}]
,PlotLabel->LabelForm@"True Function & Sampled Points"]

Presentation Notes

To start we will randomly synthesize two clusters of two data points. Note that we have a gap in the middle as well as missing coverage at the ends. The DataModeler LabelForm[ ] function helps to easily format results.

The initial model building — 30 seconds of CPU time

Here we devote 30 seconds of CPU time to developing models. Notice that they are quite good from the point of view of fitting the observed data. Perfection would be a very simple and very accurate model (i.e., lower-left on the plot). Intelligent defaults for functions coupled with pre-defined option sets simplify the efficient application of DataModeler to real-world problems.

ParetoFrontLogPlot[
modelSet1=SymbolicRegression[
data1,
response1,
IndependentEvolutions->3,
TimeConstraint->10
]
]

For this presentation, DataModeler functions and options will be colored for clarity.

Presentation Notes

The SymbolicRegression[ ] function does most of the core data modeling The default behavior of one IndependentEvolutions with a TimeConstraint of 300 seconds (five minutes) is generally suitable for exploratory modeling. However, for serious modeling of real-world data sets we will perform multiple IndependentEvolutions in an effort to discover diverse model structures and devote much more time to each of these modeling efforts.

The ParetoFrontLogPlot[ ] (as well as the ParetoFrontPlot[ ] function) lets us explore the trade-off between model accuracy (here defined as 1-(R^2)) and the ModelComplexity[ ]. For real-world problems there will generally be a knee to the ParetoFront[ ] which defines the set of optimal models — in the sense that they are the most accurate for a given level of complexity or, alternately, the simplest for a given level of accuracy. These models are denoted in red in the plot above. Note that if we mouse-over the model fitness locations, the underlying model expression will be displayed.

Examining the developed models

Notice that, although these "optimal" models agree when constrained by data, when not constrained they are able to — and tend to — diverge. As we shall soon see, we can exploit this characteristic!

Show[
Plot[Tooltip[func1Var[x1],"True Function"],{x1,0.5,7},PlotStyle->{Thickness[0.02], Dashing[Large],Black}],
Plot[Evaluate[Tooltip[#,#]&/@ModelPhenotype@ParetoFront@modelSet1],{x1,0.5,7},PlotStyle->({Thick,Hue[(1/3)+(2/3)(#-1)]}&/@Range[12])],
ListPlot[Transpose@{(Subscript[data1, [[All,1]]]),response1},PlotStyle->{Yellow,PointSize[0.03]}],
PlotLabel->LabelForm@"Pareto Front Models vs. Truth",PlotRange->Automatic]

Presentation Notes

Some of the active cells in this presentation have been closed to avoid visual clutter during execution. To see the underlying code, select the desired cell and select the Cell:Cell Properties:Open menu item.

In this graphic we are plotting the models lying on the ParetoFront[ ]. Mousing over the response curves displayes the underlying models.

Forming a robust ensemble

The RobustModels[ ] function uses Interval[ ] arithmetic to reject models with potential pathologies within a DataVariableRange. Here we select UncorrelatedModels[ ] from a BoxRegion to build a collection of diverse models. Note that the model selection is stochastic so each time we select, we will likely choose different models.

modelSet1a=RobustModels[modelSet1,
DataVariableRange->{0.5,7},
BoxRegion->{150,0.1},
SelectionStrategy->AllModels
];

modelEnsemble1=CreateModelEnsemble[
UncorrelatedModels[modelSet1a,data1,response1],
UncorrelatedModels[ParetoFront@modelSet1a,data1,response1],
UncorrelatedModels[ParetoFront@modelSet1a,data1,response1,SignificanceLevel->0.01]
];

Show[
ParetoFrontLogPlot[modelSet1a,PlotStyle->{{Blue,PointSize[0.01]},{Green,PointSize[0.03]}}],
ParetoFrontLogPlot[modelEnsemble1,PlotStyle->{{Red,PointSize[0.02]},{Red,PointSize[0.03]}}]
,PlotLabel->LabelForm["Selected Ensemble Models",FontColor->Red]]

Presentation Notes

In choosing models, we have two constraints: we want the models to be simple AND accurate. Towards that end, the BoxRegion option lets us easily specify limits on the model selection.

Our strategy here is to select UncorrelatedModels[ ] — as determined by the correlation of the model prediction residuals — from the overall candidate set as well as from just the ParetoFront[ ] models. Additionally, we include the "most typical" model from the ParetoFront[ ]. If we wanted a tabular view of the selected models, we could use the ModelSelectionReport[ ] function as shown below. Note that the structure of the selected models look quite different.

ModelSelectionReport@modelEnsemble1

Assessing the ensemble

The constituent model response behaviors

Below we show the selected model behavior relative to the data points used in the modeling as well as the true response behavior.

Show[
Plot[Tooltip[func1Var[x1],"True Function"],{x1,0.5,7},PlotStyle->{Thickness[0.02], Dashing[Large],Black}],
Plot[Evaluate[Tooltip[#,#]&/@ModelPhenotype@EnsembleModels@modelEnsemble1],{x1,0.5,7},PlotStyle->({Thick,Hue[(1/3)+(2/3)(#-1)]}&/@Range[12])],
ListPlot[Transpose@{(Subscript[data1, [[All,1]]]),response1},PlotStyle->{Yellow,PointSize[0.03]}],
PlotLabel->LabelForm@"Ensemble Models vs. Truth",PlotRange->Automatic]

The constituent model divergence

The DivergenceSurfacePlot[ ] lets us easily view the agreement or lack of agreement for model ensembles. Below we show the effect of different choices of EnsembleDivergenceFunction.

Show[
DivergenceSurfacePlot[modelEnsemble1,ResponseSurfaceParameters->{0.5,3,7},PlotStyle->{Thickness[0.015],Red}],
ListPlot[Flatten@{#,EvaluateEnsembleDivergence[modelEnsemble1,{#},EnsembleDivergenceFunction->((Max@#-Min@#)&)]}&/@data1,
PlotStyle->{Yellow,PointSize[0.03]}],

DivergenceSurfacePlot[modelEnsemble1,ResponseSurfaceParameters->{0.5,3,7},EnsembleDivergenceFunction->StandardDeviation,PlotStyle->{Thickness[0.015],Blue}],
ListPlot[Flatten@{#,EvaluateEnsembleDivergence[modelEnsemble1,{#},EnsembleDivergenceFunction->StandardDeviation]}&/@data1,
PlotStyle->{Green,PointSize[0.03]}],

AspectRatio->(1/GoldenRatio),
PlotLabel->LabelForm["Ensemble Divergence vs Sampled Data Points",FontSize->14,FontColor->Red]
]

The ensemble prediction

The ResponseSurfacePlot[ ] function allows us to visualize model and ensemble predictions. As we can see, the models fit the data but do not necessarily accomodate information which has not been supplied to the selection process. (For more model variables, it will plot all unique pairwise combinations of variables.)

Show[
Plot[Tooltip[func1Var[x1],"True Function"],{x1,0.5,7},PlotStyle->{Thickness[0.02], Dashing[Large],Black}],
ResponseSurfacePlot[modelEnsemble1,ResponseSurfaceParameters->{0.5,3,7},PlotStyle->{Thickness[0.015],Red}],
ListPlot[Transpose@{(Subscript[data1, [[All,1]]]),response1},PlotStyle->{Yellow,PointSize[0.03]}]
,PlotLabel->LabelForm["Ensemble Prediction vs True Function & Sampled Data",FontSize->14,FontColor->Red],PlotRange->All]

Presentation Notes

The DivergenceSurfacePlot[ ] will automatically display the divergence surface over all pairwise combinations of constituent variables over the range specified by the ResponseSurfaceParameters (with any nonvarying variables in each pairwise surface held constant at the middle values). The ResponseSurfacePlot[ ] has a similar behavior.

The key point is that we have selected models which agree when constrained by the data and diverge when not subject to observational constraints.

Localizing the Uncertainty

The location of most divergence

The ModelMaximum[ ] function uses Mathematica's built-in NMinimize[ ] function to search for the maximum of a function. In this case, we are not seeing to maximize the function; rather we are trying to identify the point of maximum model disagreement. To achieve this goal, we simply need to change the EnsembleEvaluationFunction to match our objective, i.e.,

ModelMaximum[modelEnsemble1,DataVariableRange->{0.5,7},EnsembleEvaluationFunction->StandardDeviation]

Given that we can identified the point of maximum divergence,

newSamplePoint=x1 /. Last@ ModelMaximum[modelEnsemble1,DataVariableRange->{0.5,7},EnsembleEvaluationFunction->StandardDeviation]

We can "run an experiment" and collect a new observation

newResponsePoint=func1Var[newSamplePoint]

Incorporate the new data into the data set

The new data point is shown below

data2=Flatten@{(Subscript[data1, [[All,1]]]),newSamplePoint};
response2=Flatten@{response1,newResponsePoint};

Show[
Plot[Tooltip[func1Var[x1],"True Function"],{x1,0.5,7},PlotStyle->{Thickness[0.02], Dashing[Large],Black}],
Plot[Evaluate[Tooltip[#,#]&/@ModelPhenotype@EnsembleModels@modelEnsemble1],{x1,0.5,7},PlotStyle->({Thick,Hue[(1/3)+(2/3)(#-1)]}&/@Range[12])],
ListPlot[Transpose@{data2,response2},PlotStyle->{Yellow,PointSize[0.03]}],
ListPlot[{{newSamplePoint,newResponsePoint}},PlotStyle->{Red,PointSize[0.05]}],
PlotLabel->LabelForm@"New Data Point vs. Ensemble Models vs. Truth",PlotRange->All]

Presentation Notes

Of course, the ModelMaximum[ ] can easily be used to determine the model maxima. If desired, we can allow the search to explore outside the DataVariableRange as well as impose constraints on the model and the variables.

Options@ModelExtrema

Doing another round of modeling

Update the legacy models

At this point, many of the previously developed models will not be able to accomodate the new information; however, some may still be valid.

ParetoFrontLogPlot[
modelSet2a=AlignModel[modelSet1a,data2,response2],
PlotLabel->LabelForm@"Updated Legacy Models"]

Evolving the ensemble models against the new data — 30 more seconds (60 total)

Here we have implemented the RobustModels option for SymbolicRegression[ ] to avoid spending effort pursuing models with pathologies. We will seed our development with the previously selected EnsembleModels.

ParetoFrontLogPlot[
modelSet2b=SelectModels[
SymbolicRegression[data2,response2,
RobustModels->True,
IndependentEvolutions->3,
InitialPopulation->EnsembleModels@modelEnsemble1,
TimeConstraint->10],
BoxRegion->{All,0.1},
SelectionStrategy->AllModels],
PlotLabel->LabelForm@"Models Built upon the Chosen Ensemble Base"]

Evolving new models from scratch — 30 more seconds (1.5 minute total)

The founders effect can color what solutions are discovered. Hence, we also run several evolutionary model building using random starting models (which is the default behavior).

ParetoFrontLogPlot[
modelSet2c=SelectModels[
SymbolicRegression[data2,response2,
RobustModels->True,
IndependentEvolutions->3,
TimeConstraint->10],
BoxRegion->{All,0.1},
SelectionStrategy->AllModels],PlotLabel->LabelForm@"Models evolved from random initial models"]

Merging the subpopulations

The default SelectionStrategy is actually to select the 50% of models closest to the ParetoFront; however, we have explicitly shown it here for illustration purposes.

ParetoFrontLogPlot[#,
PlotLabel->LabelForm[{"Merged Model Set =",Length@#,"Models"},Joined->True]
]&[modelSet2=RobustModels[MergeModelSets[modelSet2a,modelSet2b,modelSet2c],
DataVariableRange->{0.5,7},
BoxRegion->{150,0.1},
SelectionStrategy->(ParetoFrontSelect[#,SelectionSize->0.50]&)
]]

The ParetoFront models

There are tools to let us get summary statistics as well as report-quality overviews of developed models. For example, the ModelSelectionReport[ ] shown below which, by default, shows the models on the ParetoFront of the supplied model set.

ModelSelectionReport@modelSet2

Presentation Notes

AlignModels[ ] is an example of a genetic engineering tool in the sense that we can take a model and scale and translate it to get a least-squares fit of the model to the data. As we see, some of the functions survive the inclusion of new data quite well and others do not.

Defining a new ensemble

The new ensemble

We again build an ensemble using UncorrelatedModels[ ] from the overall population as well as the ParetoFront subpopulation.

modelEnsemble2=CreateModelEnsemble[
UncorrelatedModels[modelSet2,data2,response2],
UncorrelatedModels[ParetoFront@modelSet2,data2,response2],
UncorrelatedModels[ParetoFront@modelSet2,data2,response2,SignificanceLevel->0.01]
];

Show[
ParetoFrontLogPlot[modelSet2,PlotStyle->{{Blue,PointSize[0.01]},{Green,PointSize[0.03]}}],
ParetoFrontLogPlot[modelEnsemble2,PlotStyle->{{Red,PointSize[0.02]},{Red,PointSize[0.03]}}]
,PlotLabel->LabelForm["Selected Ensemble Models",FontColor->Red,FontSize->18]]

The new ensemble constituent predictions

The new data point has introduced an additional constraint on the constituent model behaviors.

Show[
Plot[Tooltip[func1Var[x1],"True Function"],{x1,0.5,7},PlotStyle->{Thickness[0.02], Dashing[Large],Black}],
Plot[Evaluate[Tooltip[#,#]&/@ModelPhenotype@EnsembleModels@modelEnsemble2],{x1,0.5,7},PlotStyle->({Thick,Hue[(1/3)+(2/3)(#-1)]}&/@Range[12])],
ListPlot[Transpose@{data2,response2},PlotStyle->{Yellow,PointSize[0.03]}],
PlotLabel->LabelForm@"New Data Point vs. Revised Models vs. Truth",PlotRange->Automatic]

The new ensemble prediction

Block[
{ensemble=modelEnsemble2,
data=data2,
response=response2},
Show[
Plot[Tooltip[func1Var[x1],"True Function"],{x1,0.5,7},PlotStyle->{Thickness[0.02], Dashing[Large],Black}],

ResponseSurfacePlot[ensemble,ResponseSurfaceParameters->{0.5,3,7},PlotStyle->{Thickness[0.015],Red}],

ListPlot[Transpose@{data,response},PlotStyle->{Yellow,PointSize[0.03]}]
,PlotLabel->LabelForm["Ensemble Prediction vs True Function & Sampled Data",FontSize->14,FontColor->Red],PlotRange->All]
]

The new ensemble divergence

We will want to collect more data at the points where the models diverge.

Show[
DivergenceSurfacePlot[modelEnsemble2,ResponseSurfaceParameters->{0.5,3,7},PlotStyle->{Thickness[0.015],Red},EnsembleDivergenceFunction->StandardDeviation],

ListPlot[Flatten@{#,EvaluateEnsembleDivergence[modelEnsemble2,{#},EnsembleDivergenceFunction->StandardDeviation]}&/@data2,
PlotStyle->{Yellow,PointSize[0.03]}],

AspectRatio->(1/GoldenRatio),
PlotLabel->LabelForm["Ensemble Divergence vs Sampled Data Points",FontSize->16,FontColor->Red],
PlotRange->All]

Presentation Notes

In this section we have built new models using the additional information from our new data. As we can see, this improves the quality of the model in that region.

A 3rd round of model building — 60 more seconds (2.5 minutes total)

Here we incorporate another data point and spend 10 seconds each on six IndependentEvolutions.

{data3,response3,modelEnsemble3,modelSet3}=Block[
{newSamplePoint,newResponsePoint,data,response,updatedModels,popModels,newModels,refinedModels,mergedModels,newEnsemble,
dataIn=data2,
responseIn=response2,
modelPop=modelSet2,
ensembleIn=modelEnsemble2,
timeLimit=10,
numEvol=3,
maxError=0.1,
maxComplexity=200
},

newSamplePoint=x1 /. Last@ ModelMaximum[ensembleIn,DataVariableRange->{0.5,7},EnsembleEvaluationFunction->StandardDeviation];
newResponsePoint=func1Var[newSamplePoint];

data=Flatten@{dataIn,newSamplePoint};
response=Flatten@{responseIn,newResponsePoint};

updatedModels=SelectModels[UpdateModelFitness[modelPop,data,response],
BoxRegion->{All,0.1},
SelectionStrategy->AllModels];
refinedModels=SelectModels[
SymbolicRegression[data,response,
RobustModels->True,
IndependentEvolutions->numEvol,
InitialPopulation->EnsembleModels@ensembleIn,
TimeConstraint->timeLimit],
BoxRegion->{All,0.1},
SelectionStrategy->AllModels];
newModels=SelectModels[
SymbolicRegression[data,response,
RobustModels->True,
IndependentEvolutions->numEvol,
TimeConstraint->timeLimit],
BoxRegion->{All,0.1},
SelectionStrategy->AllModels];

mergedModels = RobustModels[popModels=MergeModelSets[updatedModels,refinedModels,newModels],
DataVariableRange->{0.25,7},
BoxRegion->{maxComplexity,maxError}
];

newEnsemble=CreateModelEnsemble[
UncorrelatedModels[mergedModels,data,response],
UncorrelatedModels[ParetoFront@mergedModels,data,response],
UncorrelatedModels[ParetoFront@mergedModels,data,response,SignificanceLevel->0.01]
];

Print@GraphicsGrid[{{Show[
ParetoFrontLogPlot[popModels,PlotStyle->{{Blue,PointSize[0.01]},{Green,PointSize[0.03]}}],
ParetoFrontLogPlot[newEnsemble,PlotStyle->{{Red,PointSize[0.02]},{Red,PointSize[0.03]}}]
,PlotLabel->LabelForm["BoxRegion -> "<>ToString@{maxComplexity,maxError}]],

Show[
Plot[Tooltip[func1Var[x1],"True Function"],{x1,0.5,7},PlotStyle->{Thickness[0.02], Dashing[Large],Black}],
Plot[Evaluate[Tooltip[#,#]&/@ModelPhenotype@EnsembleModels@newEnsemble],{x1,0.5,7},PlotStyle->({Thick,Hue[(1/3)+(2/3)(#-1)]}&/@Range[12])],
ListPlot[Transpose@{data,response},PlotStyle->{Yellow,PointSize[0.03]}],
ListPlot[{{newSamplePoint,newResponsePoint}},PlotStyle->{Red,PointSize[0.05]}],
PlotLabel->LabelForm@"Data Points vs. Revised Models vs. Truth",PlotRange->Automatic]
},{
Show[
DivergenceSurfacePlot[newEnsemble,ResponseSurfaceParameters->{0.5,3,7},PlotStyle->{Thickness[0.015],Red},EnsembleDivergenceFunction->StandardDeviation],

ListPlot[Flatten@{#,EvaluateEnsembleDivergence[newEnsemble,{#},EnsembleDivergenceFunction->StandardDeviation]}&/@data,
PlotStyle->{Yellow,PointSize[0.03]}],

AspectRatio->(1/GoldenRatio),
PlotLabel->LabelForm@"Ensemble Divergence vs Data Points",
PlotRange->All],

Show[
Plot[Tooltip[func1Var[x1],"True Function"],{x1,0.5,7},PlotStyle->{Thickness[0.02], Dashing[Large],Black}],

ResponseSurfacePlot[newEnsemble,ResponseSurfaceParameters->{0.5,3,7},PlotStyle->{Thickness[0.015],Red}],

ListPlot[Transpose@{data,response},PlotStyle->{Yellow,PointSize[0.03]}]
,PlotLabel->LabelForm@"Prediction vs True Function & Data",PlotRange->All]
}},ImageSize->600,Background->White,Spacings->0,
PlotLabel->LabelForm["The Results from a Second Round of Adaptive Data Collection",FontSize->18,FontColor->Blue]];

{data,response,newEnsemble,mergedModels}
];

A 4th round of model building — 60 more seconds (3.5 minutes total)

Again we incorporate another data point and spend 10 seconds each on six IndependentEvolutions.

{data4,response4,modelEnsemble4,modelSet4}=Block[
{newSamplePoint,newResponsePoint,data,response,updatedModels,newModels,popModels,refinedModels,mergedModels,newEnsemble,
dataIn=data3,
responseIn=response3,
modelPop=modelSet3,
ensembleIn=modelEnsemble3,
timeLimit=10,
numEvol=3,
maxError=0.1,
maxComplexity=200
},

newSamplePoint=x1 /. Last@ ModelMaximum[ensembleIn,DataVariableRange->{0.5,7},EnsembleEvaluationFunction->StandardDeviation];
newResponsePoint=func1Var[newSamplePoint];

data=Flatten@{dataIn,newSamplePoint};
response=Flatten@{responseIn,newResponsePoint};

updatedModels=SelectModels[UpdateModelFitness[modelPop,data,response],
BoxRegion->{All,0.1},
SelectionStrategy->AllModels];
refinedModels=SelectModels[
SymbolicRegression[data,response,
RobustModels->True,
IndependentEvolutions->numEvol,
InitialPopulation->EnsembleModels@ensembleIn,
TimeConstraint->timeLimit],
BoxRegion->{All,0.1},
SelectionStrategy->AllModels];
newModels=SelectModels[
SymbolicRegression[data,response,
RobustModels->True,
IndependentEvolutions->numEvol,
TimeConstraint->timeLimit],
BoxRegion->{All,0.1},
SelectionStrategy->AllModels];

mergedModels = RobustModels[popModels=MergeModelSets[updatedModels,refinedModels,newModels],
DataVariableRange->{0.25,7},
BoxRegion->{maxComplexity,maxError}
];

newEnsemble=CreateModelEnsemble[
UncorrelatedModels[mergedModels,data,response],
UncorrelatedModels[ParetoFront@mergedModels,data,response],
UncorrelatedModels[ParetoFront@mergedModels,data,response,SignificanceLevel->0.01]
];

Print@GraphicsGrid[{{Show[
ParetoFrontLogPlot[popModels,PlotStyle->{{Blue,PointSize[0.01]},{Green,PointSize[0.03]}}],
ParetoFrontLogPlot[newEnsemble,PlotStyle->{{Red,PointSize[0.02]},{Red,PointSize[0.03]}}]
,PlotLabel->LabelForm["BoxRegion -> "<>ToString@{maxComplexity,maxError}]],

Show[
Plot[Tooltip[func1Var[x1],"True Function"],{x1,0.5,7},PlotStyle->{Thickness[0.02], Dashing[Large],Black}],
Plot[Evaluate[Tooltip[#,#]&/@ModelPhenotype@EnsembleModels@newEnsemble],{x1,0.5,7},PlotStyle->({Thick,Hue[(1/3)+(2/3)(#-1)]}&/@Range[12])],
ListPlot[Transpose@{data,response},PlotStyle->{Yellow,PointSize[0.03]}],
ListPlot[{{newSamplePoint,newResponsePoint}},PlotStyle->{Red,PointSize[0.05]}],
PlotLabel->LabelForm@"Data Points vs. Revised Models vs. Truth",PlotRange->Automatic]
},{
Show[
DivergenceSurfacePlot[newEnsemble,ResponseSurfaceParameters->{0.5,3,7},PlotStyle->{Thickness[0.015],Red},EnsembleDivergenceFunction->StandardDeviation],

ListPlot[Flatten@{#,EvaluateEnsembleDivergence[newEnsemble,{#},EnsembleDivergenceFunction->StandardDeviation]}&/@data,
PlotStyle->{Yellow,PointSize[0.03]}],

AspectRatio->(1/GoldenRatio),
PlotLabel->LabelForm@"Ensemble Divergence vs Data Points",
PlotRange->All],

Show[
Plot[Tooltip[func1Var[x1],"True Function"],{x1,0.5,7},PlotStyle->{Thickness[0.02], Dashing[Large],Black}],

ResponseSurfacePlot[newEnsemble,ResponseSurfaceParameters->{0.5,3,7},PlotStyle->{Thickness[0.015],Red}],

ListPlot[Transpose@{data,response},PlotStyle->{Yellow,PointSize[0.03]}]
,PlotLabel->LabelForm@"Prediction vs True Function & Data",PlotRange->All]
}},ImageSize->600,Background->White,Spacings->0,
PlotLabel->LabelForm["The Results from a Third Round of Adaptive Data Collection",FontSize->18,FontColor->Blue]];

{data,response,newEnsemble,mergedModels}];

A 5th round of model building — 60 more seconds (4.5 minutes total)

This round we will maintain the CPU effort expended on any individual evolution of model building — a whopping 10 seconds for each of the six IndependentEvolutions!

{data5,response5,modelEnsemble5,modelSet5}=Block[
{newSamplePoint,newResponsePoint,data,response,updatedModels,popModels,newModels,refinedModels,mergedModels,newEnsemble,
dataIn=data4,
responseIn=response4,
modelPop=modelSet4,
ensembleIn=modelEnsemble4,
timeLimit=10,
numEvol=3,
maxError=0.1,
maxComplexity=200
},

newSamplePoint=x1 /. Last@ ModelMaximum[ensembleIn,DataVariableRange->{0.5,7},EnsembleEvaluationFunction->StandardDeviation];
newResponsePoint=func1Var[newSamplePoint];

data=Flatten@{dataIn,newSamplePoint};
response=Flatten@{responseIn,newResponsePoint};

updatedModels=SelectModels[UpdateModelFitness[modelPop,data,response],
BoxRegion->{All,0.1},
SelectionStrategy->AllModels];
refinedModels=SelectModels[
SymbolicRegression[data,response,
RobustModels->True,
IndependentEvolutions->numEvol,
InitialPopulation->EnsembleModels@ensembleIn,
TimeConstraint->timeLimit],
BoxRegion->{All,0.1},
SelectionStrategy->AllModels];
newModels=SelectModels[
SymbolicRegression[data,response,
RobustModels->True,
IndependentEvolutions->numEvol,
TimeConstraint->timeLimit],
BoxRegion->{All,0.1},
SelectionStrategy->AllModels];

mergedModels = RobustModels[popModels=MergeModelSets[updatedModels,refinedModels,newModels],
DataVariableRange->{0.25,7},
BoxRegion->{maxComplexity,maxError}
];

newEnsemble=CreateModelEnsemble[
UncorrelatedModels[mergedModels,data,response],
UncorrelatedModels[ParetoFront@mergedModels,data,response],
UncorrelatedModels[ParetoFront@mergedModels,data,response,SignificanceLevel->0.01]
];

Print@GraphicsGrid[{{Show[
ParetoFrontLogPlot[popModels,PlotStyle->{{Blue,PointSize[0.01]},{Green,PointSize[0.03]}}],
ParetoFrontLogPlot[newEnsemble,PlotStyle->{{Red,PointSize[0.02]},{Red,PointSize[0.03]}}]
,PlotLabel->LabelForm["BoxRegion -> "<>ToString@{maxComplexity,maxError}]],

Show[
Plot[Tooltip[func1Var[x1],"True Function"],{x1,0.5,7},PlotStyle->{Thickness[0.02], Dashing[Large],Black}],
Plot[Evaluate[Tooltip[#,#]&/@ModelPhenotype@EnsembleModels@newEnsemble],{x1,0.5,7},PlotStyle->({Thick,Hue[(1/3)+(2/3)(#-1)]}&/@Range[12])],
ListPlot[Transpose@{data,response},PlotStyle->{Yellow,PointSize[0.03]}],
ListPlot[{{newSamplePoint,newResponsePoint}},PlotStyle->{Red,PointSize[0.05]}],
PlotLabel->LabelForm@"Data Points vs. Revised Models vs. Truth",PlotRange->Automatic]
},{
Show[
DivergenceSurfacePlot[newEnsemble,ResponseSurfaceParameters->{0.5,3,7},PlotStyle->{Thickness[0.015],Red},EnsembleDivergenceFunction->StandardDeviation],

ListPlot[Flatten@{#,EvaluateEnsembleDivergence[newEnsemble,{#},EnsembleDivergenceFunction->StandardDeviation]}&/@data,
PlotStyle->{Yellow,PointSize[0.03]}],

AspectRatio->(1/GoldenRatio),
PlotLabel->LabelForm@"Ensemble Divergence vs Data Points",
PlotRange->All],

Show[
Plot[Tooltip[func1Var[x1],"True Function"],{x1,0.5,7},PlotStyle->{Thickness[0.02], Dashing[Large],Black}],

ResponseSurfacePlot[newEnsemble,ResponseSurfaceParameters->{0.5,3,7},PlotStyle->{Thickness[0.015],Red}],

ListPlot[Transpose@{data,response},PlotStyle->{Yellow,PointSize[0.03]}]
,PlotLabel->LabelForm@"Prediction vs True Function & Data",PlotRange->All]
}},ImageSize->600,Background->White,Spacings->0,
PlotLabel->LabelForm["The Results from a Fourth Round of Adaptive Data Collection",FontSize->18,FontColor->Blue]];

{data,response,newEnsemble,mergedModels}
];

Adaptive Data Collection Summary

Show[GraphicsGrid@{{
Show[
DivergenceSurfacePlot[modelEnsemble5,ResponseSurfaceParameters->{0.25,3,8},PlotStyle->{Thickness[0.015],Red},EnsembleDivergenceFunction->StandardDeviation],

ListPlot[Flatten@{#,EvaluateEnsembleDivergence[modelEnsemble5,{#},EnsembleDivergenceFunction->StandardDeviation]}&/@data5,
PlotStyle->{Yellow,PointSize[0.03]}],

AspectRatio->(1/GoldenRatio),
PlotLabel->LabelForm@"Ensemble Divergence vs Data Points",
PlotRange->All],

Show[
Plot[Tooltip[func1Var[x1],"True Function"],{x1,0.25,8},PlotStyle->{Thickness[0.02], Dashing[Large],Black}],
Plot[Evaluate[Tooltip[#,#]&/@ModelPhenotype@EnsembleModels@modelEnsemble5],{x1,0.25,8},PlotStyle->({Thick,Hue[(1/3)+(2/3)(#-1)]}&/@Range[12])],
ListPlot[Transpose@{data5,response5},PlotStyle->{Yellow,PointSize[0.03]}],
ListPlot[{{newSamplePoint,newResponsePoint}},PlotStyle->{Red,PointSize[0.05]}],
PlotLabel->LabelForm@"Data Points vs. Revised Models vs. Truth",PlotRange->Automatic]
}},PlotLabel->LabelForm["Final Ensemble Under Extrapolation",FontSize->20],ImageSize->600]

Devoted  4.5 minutes of CPU time to adaptive data collection & model building

developed good quality models

showed the assembly of diverse models

demonstrated the exploitation of trustability in adaptive data collection.

In real life we, of course, would spend much more CPU time and be multivariate; however, the physical data collection would also be much more difficult and time-consuming

Relative to the Unique aspects …

Focus — automatic variable selection

Insight — human-interpretable models

Trust — prediction reliability metrics

Presentation Notes

In terms of the unique aspects, we have primarily demonstrated the trustable models with some component of human insight because of the ability to view the evolved expressions. The other publications and DataModeler documentation case studies go into much more detail in demonstrating all of these aspects.

For more information, contact info@evolved-analytics.com