MXLPlus

data Mining eXtensible Library

Ariel Sepúlveda

Pronto Analytics, Inc.

What is MXL Plus?

MXL Plus (MXL) is an application designed to simplify and improve the use of Mathematica for doing data analysis by 
a) providing functions with the capability of operating over labeled data sets to emulate and improve spreadsheet or database type of operations, 
b) providing new methods and graphical tools for data analysis, and 
c) providing functions for scheduling tasks.  
These characteristics are used to transform Mathematica into a real-time data analysis and reporting system or process control and feedback environment.

Simplified operations for labeled data sets
Data base languages and other data analysis software allow for operations with field names such as "sales"="price"*"quantity".  Considering this a desirable and intuitive way of referencing data the functions developed in this application are mainly designed to allow users to do data operations by making reference to their column or field name instead of its position in the target data set.  This means that the user will keep all known functionalities of Mathematica but in an environment specially designed for simplifying data analysis.  This way of expressing operations is easy to understand (# and & not used) and, more importantly, makes programs more readable and thus easier to debug or modify specially if a considerable amount of time has elapsed since the first time the program was created and the time it will be modified.  Another advantage is that the code generated with these functions is dynamic in the sense that  adding or deleting fields from the source data set analyzed does not affect the code generated with MXL functions because of the field-wise reference to data.  In summary, functions like MapMXL, SelectMXL, SplitMXL, TakeMXL, DropMXL, and PartMXL are modifications of functions to allow the use of field names in their attributes instead of numerical references to column or row positions.  More important, MXL functions seamlesly operate over rectangular data sets, or over multirectangular data sets.  For example, a data set representing sales can easily be partitioned by "account_number" and in the same operation field-name-referenced analyses performed for the partitioned data set are automatically performed for all account numbers.

Plus... new methods and graphical tools for data analysis and report generation
Several functions like PivotAnalysis, MarketBasketAnalysis, ControlChart, SPC, PrioritizationPlot, GoalPlot, ControlPanel, analyze data and make intuitive graphical representation of results.  Other functions like SortByColumnMXL, TakeFractionMXL, TransformVariablesMXL, DefineIndexMXL, SelectMembersMXL, SelectDistinctMXL, TakeRandomSample, TakeFractionMXL, OutlineMXL, TransformVariableMXL, and MutateMXL, have been designed to provide flexible data manipulation and simplified data analysis capabilities.   In particular   PivotAnalysis is a versatile function for analyzing and reporting from data sets with categorical fields.  PivotAnalysis filters data by setting conditions on its fields, and produces several types of plots for the resulting filtered data (BoxPlot, BarChart, ListPlot, PieChart, etc).  Similarly, all MXL functions are capable of automatically operating either over rectangular data sets or over multi-rectangular data sets defined by splitting data by the elements of one of its categorical fields.  For example, if a data set has been split by one categorical field such that each subset belongs to the each different instance of the elements of that categorical field, then the entire split data set can be analyzed as if it were a rectangular data set but now with the advantage of getting the corresponding results to each data subset.   Other new types of plots are also provided for representing relationshipts between variables.  Two of the most significant plots are those generated by the functions MarketBasketAnalysis and PrioritizationPlot.  

Plus... Scheduling functionality
MXL provides scheduling functions that transform the Mathematica environment from a static to a dynamic one by allowing users to schedule tasks to be performed at any predetermined times and frequencies.  For example, users can easily create a multitasking system that a) monitors a production process in real time by means of control charts and other analyses techniques, b) creates hourly reports summarizing key information from several data sources, c) creates other high level summary analysis and reports at desired frequencies (e.g. daily, weekly, monthly, quarterly). 

Plus... you are still inside Mathematica and thus, you still get all its power for
	-   programming, graphics, string manipulation, symbolic processing etc.
	-  handling mixed data sets of numbers, strings, symbols, graphics, etc.
	-  creating more readable, and at the same time, even more condensed code with MXL.
	-  efficiently processing large amounts of data at high speeds.
	
	          real-time data analysis  +  
				scheduled function evaluation  + 
			intuitive labeled data analysis  + 
MXL Plus  =				graphical representations of results  + 
					new data analysis functions   + 
				excellent processing speed  + 
		DatabaseLink, Import, Export  +...

Introduction to MXL: Basic Examples

This section will show some data analysis examples using MXL functions.  In particular, the functions SelectMXL, TakeMXL, SplitMXL, MapMXL, and PivotAnalysis will be used to introduce the MXL concept.

Data for examples

\(\(data = {{"Town", "Employee", "Product", "Quantity", "Price", "Date"}, {"Ponce", "Sergio", "InkJet", 21.`, 
            350.`, "3/7/06"}, {"Ponce", "Tomas", "InkJet", 
            171.`, 350.`, "10/14/06"}, {"San Juan", "Ariel", "InkJet", 138.`, 
            350.`, "1/15/06"}, {"San Juan", "Ariel", "InkJet", 240.`, 
            350.`, "5/31/06"}, {"San Juan", "Ariel", "InkJet", 84.`, 
            350.`, "12/21/06"}, {"San Juan", "Mario", "InkJet", 33.`, 
            350.`, "11/17/06"}, {"San Juan", "Mario", "InkJet", 282.`, 
            350.`, "12/4/06"}, {"San Juan", "Antonio", "InkJet", 150.`, 
            350.`, "1/23/05"}, {"San Juan", "Maria", "InkJet", 84.`, 
            350.`, "10/5/05"}, {"San Juan", "Silvia", "InkJet", 261.`, 
            350.`, "2/1/06"}, {"Arecibo", "Allison", "InkJet", 180.`, 
            350.`, "4/1/05"}, {"Arecibo", "Mario", "InkJet", 
            87.`, 350.`, "7/12/05"}, {"Arecibo", "Carlos", "InkJet", 180.`, 
            350.`, "6/8/05"}, {"Arecibo", "Carlos", "InkJet", 
            12.`, 350.`, "2/18/06"}, {"Arecibo", "Javier", "InkJet", 243.`, 
            350.`, "7/29/05"}, {"Ponce", "Sergio", "Desktop", 
            9.`, 600.`, "8/24/06"}, {"San Juan", "Antonio", "Desktop", 15.`, 
            600.`, "6/17/06"}, {"San Juan", "Silvia", "Desktop", 6.`, 
            600.`, "9/1/05"}, {"Ponce", "Sergio", "Laptop", 
            228.`, 1000.`, "9/27/06"}, {"San Juan", "Ariel", "Laptop", 81.`, 
            1000.`, "2/26/05"}, {"Arecibo", "Mario", "Laptop", 288.`, 
            1000.`, "4/27/06"}, {"Arecibo", "Carlos", "Laptop", 192.`, 
            1000.`, "10/22/05"}, {"Arecibo", "Javier", "Laptop", 45.`, 
            1000.`, "11/8/05"}, {"San Juan", "Mario", "LaserJet", 150.`, 
            200.`, "3/24/06"}, {"San Juan", "Antonio", "LaserJet", 288.`, 
            200.`, "11/25/05"}, {"San Juan", "Antonio", "LaserJet", 126.`, 
            200.`, "8/7/06"}, {"San Juan", "Maria", "LaserJet", 165.`, 
            200.`, "7/21/06"}, {"Arecibo", "Carlos", "LaserJet", 48.`, 
            200.`, "9/18/05"}, {"Arecibo", "Carlos", "LaserJet", 186.`, 
            200.`, "7/4/06"}, {"Arecibo", "Javier", "LaserJet", 222.`, 
            200.`, "12/29/05"}, {"Ponce", "Sergio", "Ink Cartridge", 1680.`, 
            35.`, "3/15/05"}, {"Ponce", "Tomas", "Ink Cartridge", 960.`, 
            35.`, "5/22/05"}, {"San Juan", "Pepe", "Ink Cartridge", 2250.`, 
            35.`, "4/18/05"}, {"San Juan", "Ariel", "Ink Cartridge", 1980.`, 
            35.`, "4/10/06"}, {"San Juan", "Ariel", "Ink Cartridge", 1590.`, 
            35.`, "5/14/06"}, {"San Juan", "Ariel", "Ink Cartridge", 210.`, 
            35.`, "9/10/06"}, {"San Juan", "Ariel", "Ink Cartridge", 420.`, 
            35.`, "10/31/06"}, {"San Juan", "Mario", "Ink Cartridge", 1080.`, 
            35.`, "2/9/05"}, {"San Juan", "Mario", "Ink Cartridge", 2700.`, 
            35.`, "5/5/05"}, {"San Juan", "Maria", "Ink Cartridge", 2700.`, 
            35.`, "6/25/05"}, {"San Juan", "Silvia", "Ink Cartridge", 2010.`, 
            35.`, "12/12/05"}, {"Arecibo", "Carlos", "Ink Cartridge", 2850.`, 
            35.`, "1/6/05"}, {"Arecibo", "Carlos", "Ink Cartridge", 1050.`, 35.`, "8/15/05"}};\)\)

Data Manipulation using MXL

Each of the rows in this labeled data set represents sales transactions associated an Employee in a given Town and Date.  In each transaction a given Quantity of a Product were sold at the registered Price.  TakeRandomSample is used to sample from data.

TakeRandomSample[data, 3, True] // TableForm

SelectMXL allows to select rows from data by establishing a condition stated using any of the data labels.  Note that the output retains the labels.

newdata = 
        SelectMXL[data, "Price" > 500];
Take[newdata, 3] // TableForm

Similarly, TakeMXL allows to take from data any desired number of rows (2) from the columns of interest ("Town","Date").   Same as with SelectMXL,  the output retains the labels.  Note that MXL functions always make reference to the columns or fields of data by specifying only its name.  Thus, specifying a field name in MXL functions is analogous to taking the corresponding column from the data set being evaluated.  This is similar to what most data bases use to make reference to data fields.

newdata = 
        TakeMXL[data, 
          2, {"Town", "Date"}];
newdata // TableForm

In the example below SplitMXL will partition data such that each resulting subset will have equal entries in the fields {"Town","Employee"}.

splitdata = SplitMXL[data, {"Town", "Employee"}];
splitdata[[3]] // TableForm

The distinct values of {"Town","Employee"} found in data can be extracted from splitdata by using DistinctID.

splitdata // DistinctID

Note that the labels are not present in each subset, but they are only presented as the first element of splitdata along with an identifier for the associated partition or SplitID.  This has a justification and in general, the result of SplitMXL will always be presented with this type of header unless otherwise specified by the user using options (more details in the documentation for SplitMXL).

Dimensions /@ splitdata

Here is a key element of MXL functions: the input data to MXL functions can either be a rectangular data set (like data), or a multirectangular data presented in the format used in the output of SplitMXL (like splitdata).  Note how SelectMXL can select from splitdata even though nothing has been specified about the structure of splitdata.

newdata = 
        SelectMXL[
          splitdata, "Price" ≥ 
            350];
newdata[[3]] // TableForm

Dimensions /@ newdata

One of the most useful MXL functions is MapMXL which allows for row-wise calculations and column-wise calculations in a rectangular or a multirectangular labeled data set.  To understand the idea behind this function think of a spreadsheet where you want to:
a) row-wise compute a new column as a function of other cells in other columns and any other cell references (e.g. Sales=Quantity*Price)
b) column-wise summarize the content of a column (e.g. TotalSales=Total[Sales])
c) use the results of a) and b) to make new calculations

The statement below is an example of a row-wise calculation: "Sales"="Price"*"Quantity".  Note that Sales was added as a new column at the end of data; thus, the name row-wise calculation implies that a new output is produced for each row of the data being analyzed.

There are three required input parameters in MapMXL, these are: data, RowFunctions and ColumnFunctions.  In this example ColumnFunctions has been left as an empty set thus specify that no ColumnFunctions are required in this case.

newdata = 
        MapMXL[data, {"Sales" = "Price"*"Quantity"}, {}];
Take[newdata, 3] // TableForm

In the example below a summary ColumnFunction is calculated as "TotalSales"=Total["Sales"].  Note that "TotalSales" depends on "Sales" which is not an existing field name in the original data set, but has to be calculated prior to "TotalSales".  Summary calculations are named Breaks in MXL documentation and are presented as rules at the end of the corresponding data set.  Even though in this case there is only one RowFunction and one ColumnFunction you can set as many functions as needed as long as there is a logic and non-circular relationship among them and all fields used in them are either contained in the data or generated in the current instance of MapMXL.  In other words, RowFunctions and ColumnFunctions are two lists of functions whose elements don't need to be in any particular order since MapMXL itself will determine the dependencies among them and will start by making those calculations which have no dependencies other than the existing field names.

\(\(newdata = 
        MapMXL[data, {"Sales" = "Price"*"Quantity"}, {"TotalSales" = Total["Sales"]}];\)\)
Take[newdata, 3] // TableForm

Take[newdata, -3] // TableForm

There are sevral ways designed to access the output data.  Two of them are FromMXL and OutlineMXL.  FromMXL searches for the desired variable while OutlineMXL presents the column-wise calculations as a list.

FromMXL[newdata, "TotalSales"]

OutlineMXL[newdata] // TableForm

Just as an example of function dependencies consider the case below where "PercentageOfTotalSales" is a function of "TotalSales" wich is being calculated as a ColumnFunction.  This example shows that you can use any of your newly defined variables (row-wise or column-wise) in any of your calculations, and also shows how the order of calculations was automatically chosen by MapMXL.

\(\(newdata = 
        MapMXL[data, {"Sales" = "Price"*"Quantity", "PercentageOfTotalSales" = "Sales"/"TotalSales"*100}, {"TotalSales" = Total["Sales"]}];\)\)
Take[newdata, 3] // TableForm

newdata

MapMXL has several options.  In the next example "SplitID"->{"Town"} generates a pre-analysis SplitMXL[data, "Town"] such that all row and column functions will be cauculated over the partitioned data resulting from SplitMXL.  Another option shown is "OutputColumns"->{"Employee","Date","Sales"} which is used to show only "Employee","Date", and "Sales" as the output fields.  For example, newdata[[3]] shows that Total[sales] in the town of Ponce is $2370, and that the corresponding number of transactions are 6.

MapMXL // Options

\(\(newdata = 
        MapMXL[data, {"Sales" = "Price"*"Quantity"}, {"TotalSales" = Total["Sales"], "TotalTransactions" = 
              Length["Sales"]}, "SplitID" -> {"Town"}, "OutputColumns" -> {"Town", "Employee", "Date", "Sales"}];\)\)
newdata[[3]]

These are the distinct categories found in the data based on the "SplitID" specified:

newdata // DistinctID

These catagories can be used to access the resulting data by specifying the required category using FromMXL.

FromMXL[newdata, "TotalSales", {"Arecibo"}]

FromMXL[
      newdata, {"Sales", "TotalSales"}, {"Arecibo"}]

In many cases analysts will only be interested in the summary calculations resulting from MapMXL.  From the previous example the summary statistics are shown below.  Another way to to get the summary only is using the option "OutputType"->"Summary".  This is shown in the next example for MapMXL.

OutlineMXL[newdata] // TableForm

Recall that all MXL functions can be used in either a rectangular or a multirectangular labeled data set.  So as an example, the command below operates over the resulting data (newdata) and returns the selected members in each of the subsets of newdata.

SelectMXL[newdata, 5 < "TotalTransactions" < 10]

The option "OutputType"->"Summary" for MapMXL is shown below along with "SelectCriterion"->"TotalTransactions">10 and "SortBy"->{{"TotalSales",-1}}.  "SelectCriterion" is a Rule pointing at a condition used to filter the otput of MapMXL.  The condition must use at least one field name for it to be valid.  SortBy is simply used to sort the output by {{field1, direction}, and then by {field2, direction}...} where direction is -1 for asscending and 1 for descending.

\(\(newdata = 
        MapMXL[data, {"Sales" = "Price"*"Quantity"}, {"TotalSales" = Total["Sales"], "TotalTransactions" = 
              Length["Sales"]}, "SplitID" -> {"Town"}, "OutputType" -> "Summary", "SortBy" -> {{"TotalSales", \(-1\)}}, "SelectCriterion" -> "TotalTransactions" > 
              10];\)\)
newdata // TableForm

Note that there is no limitation on the type of output produced by row functions or column functions specified in MapMXL.  As an example consider the following column function that outputs a bar chart for the sales at each town.

\(\(newdata = 
        MapMXL[data, {"Sales" = "Price"*"Quantity"}, {"TotalSales" = Total["Sales"], "SalesChart" = 
              BarChart["Sales", DisplayFunction -> Identity, 
                Frame -> True, 
                FrameLabel -> {"transaction number", "sales", "Sales @ " <> "Town"[\([1]\)], "total sales = $" <> 
                      ToString["TotalSales"/
                          1000000] <> " M"}]}, "SplitID" -> {"Town"}, "OutputType" -> "Summary"];\)\)
newdata // TableForm

FromMXL[newdata, "SalesChart"]

Show[FromMXL[newdata, "SalesChart"][[2, 1]], 
        DisplayFunction -> $DisplayFunction];

Even more interesting types of analyses can be performed like market basket analysis.  Thse types of plots will be discussed latter.

\(\(splitdata = 
        MapMXL[data, {"Sales" = "Price"*"Quantity"}, {"TotalSales" = 
              Total["Sales"], 
"CrossSellPlot" = 
MarketBasketAnalysis[{"Product"}, 
                DisplayFunction -> Identity, "DistinctOnly" -> 
                  False, 
PlotLabel -> \(("Cross Sells @ " <> "SplitID"[\([2]\)] <> " 
        ($" <> 
                        ToString["TotalSales"/1000000] <> " M)" // 
                      MyFont2)\)]
}, "SplitID" -> "Town", "OutputType" -> "Summary"];\)\)
splitdata // TableForm

Show[
        GraphicsArray[
          Rest[TakeMXL[splitdata, All, "CrossSellPlot"]] // Transpose], 
        DisplayFunction -> $DisplayFunction];

PivotAnalysis (an MXL application)

MXL functions can be used to create applications like the one discussed in this section: PivotAnalysis.  PivotAnalysis is designed to summarize data by using its categorical fields and to represent the results in graphical form.  Some examples are shown next.  Before showing the examples the column "Sales" is added to data and the resulting data set is named newdata.

newdata = 
        MapMXL[data, {"Sales" = "Price"*"Quantity"/
                  1000. }, {}];
newdata[[1]]

The required arguments of PivotAnalysis are the data to be analyzed (newdata) and the pivot column ("Sales").  Thus, the simplest form of PivotAnalysis is just a bar indicating the total for the variable analyzed or Pivot Column.

plt = PivotAnalysis[newdata, "Sales"];

The per-town sales can be calculated and shown by specifying partitions: "By"-> "Town".

plt = 
        PivotAnalysis[
          newdata, "Sales", "By" -> "Town", "PivotFunction\
" -> Total];

In this example the per-town sales are shown along with its per product distribution by specifying "ThenBy"->"Product".  "PivotFunction" is an optional parameter with default value of Total.  Thus, the option "PivotFunction"->Total is not required but has been included just to show that you could have used any other function like "PivotFunction"->Mean or "PivotFunction"->Length, etc.

\(\(plt = 
        PivotAnalysis[
          newdata, "Sales", "By" -> "Town", "ThenBy" -> "Product", "PivotFunction" -> Total];\)\)

The output of PivotAnalysis contains both, the graphical output as the first element, and the data used to create the graphic as the second element.

plt[[1]]

plt[[2]] // TableForm

PivotAnalysis has several options.

PivotAnalysis // Options

Different types of plots can be attained by using the option "PlotID".

\(\(plt = 
        PivotAnalysis[
          newdata, "Sales", "By" -> {"Town" -> {"Arecibo", "San Juan"}}, "ThenBy" -> "Product", "PlotID" -> "BoxWhiskerPlot"];\)\)

These are some examples using the option "PlotID".  Note that when PivotFunction is not specified "PivotFunction"->Total.
The 0.5 at the end of the ThenBy implies that only those employees whose sales account for at least 50% of total sales will be selected from the data.

\(\(plt = 
        PivotAnalysis[
          newdata, "Sales", "By" -> "Town", "ThenBy" -> \("Employee" -> \(All -> 
                0.50\)\), "PlotID" -> {"Bar", 
              PercentileBarChart}];\)\)

\(\(plt = 
        PivotAnalysis[
          newdata, "Sales", "By" -> "Product", "ThenBy" -> "Town", "PlotID" -> {"Bar", 
              StackedBarChart}];\)\)

This plot represents the per-town distribution of sales from month 1 to month 12.

nd = 
        MapMXL[newdata, {"MonthNo" -> 
              ToExpression[
                First[StringSplit["Date", "/"]]]}, {}];
\(\(plt = 
        PivotAnalysis[
          nd, "Sales", "By" -> "MonthNo", "ThenBy" -> "Town", "PlotID" -> "Line"];\)\)

This plot represents the per-product distribution of total sales for all salesmen in San Juan and Arecibo.

\(\(plt = 
        PivotAnalysis[
          newdata, "Sales", "By" -> {"Town" -> {"San Juan", "Arecibo"}, "Employee"}, "ThenBy" -> "Product", "PivotFunction" -> Length, AspectRatio -> 1];\)\)

plt[[2]] // TableForm

This is a Pareto plot for the distribution of total sales for all towns.  Note that the bars have been sorted.

\(\(plt = 
        PivotAnalysis[
          newdata, "Sales", "By" -> "Town", "PlotID" -> "Pareto"];\)\)

This plot represents the per-product distribution of sales for all salesmen for all the towns.  However, now product distributions are shown in decreasing-importance order for each salesman.  The most important product is the one that has the largest sales associated.

\(\(plt = 
        PivotAnalysis[
          newdata, "Sales", "By" -> {"Town", "Employee"}, "ThenBy" -> "Product", "PlotID" -> "Pareto", 
          AspectRatio -> 2/3];\)\)

newdata[[1]]

This plot represents the per-product distribution of sales for all salesmen for all the towns.  However, now only the most important two products are shown for each salesman.

\(\(plt = 
        PivotAnalysis[
          newdata, "Sales", "By" -> {"Town", "Employee"}, "ThenBy" -> \("Product" -> \(All -> 
                2\)\), "PlotID" -> "Pareto"];\)\)

This plot represents the per-product distribution of sales.  Included Towns account for at least 80% of total sales.  Inside each of these towns the salesmen shown are those responsible for at least 70% of the total sales in that town .  After the data is filtered by town and employee only the products that that account for least 60% of the sales are shown.  Note that "ThenBy"  is the last of the Pareto filters independently of its position.

\(\(plt = 
        PivotAnalysis[
          newdata, "Sales", "By" -> {"Town" -> \(All ->  .8\), "Employee" -> \(All ->  .7\)}, "ThenBy" -> \("Product" -> \(All ->  .60\)\), "PlotID" -> "Pareto"];\)\)

Similar to the previous case, in this example the least significant 10% of the poroducts are presented by specifying "Product" -> - 0.10.  Before that, only the most significant town between San Juan and Ponce and the least significant two employees are selected.

\(\(plt = 
        PivotAnalysis[
          newdata, "Sales", "By" -> {"Town" -> \({"San Juan", "Ponce"} -> 
                  1\), "Employee" -> \(All -> \(-2\)\)}, "ThenBy" -> \("Product" -> \(All -> \(- .1\)\)\), "PlotID" -> "Pareto"];\)\)

MarketBasketAnalysis

Market Basket Analysis is a data mining technique which objective is to determine which articles are prone to be bougth together by customers.   Internet marketting stores are interested in such technique to analyze their customer's behavior and use the resulting information for making attractive offers to customers.  Many other applications exists since the "web-representation" of the analysis helps understand relationships between variables in high dimensional problems.  Just to mention two more applications consider a process or system where you want to determine how often the system changes from one state to another, or visits to places by entities where you are interested in identifying the most common routes and the associated directions used by the entities.

The first example shown below resembles the purshasing behavior of customers.  Suppose that for each customer orders within a given time span were grouped to analyze the association within the articles {"prod 1", "prod 2"...) the client has bougth "together" or in the same market basket.  MarketBasketAnalysis calculates all the associations or pairs of proucts in each order and then the frequencies for each pair.  For example, the order with entries {"prod 1","prod 7","prod 3"} has these pairs associated: "prod 1"-"prod 7", "prod 1"-"prod 3", and "prod 7"-"prod 3".  After calculating the frequencies, the relationships are presented in a web.  In the example the strongest relationship is "prod 1"-"prod 3" and is represented with the thickest line in the web.  The numbers represent the frequencies associated to the pair of nodes linked by the arc.  The color of the numbers help understanding which two nodes they belong to since the color of the number is the same as its origin node.

\(\(byorderdata = {{"prod 1", "prod 7", "prod 3"}, {"prod 1", "prod 3", "prod 6", "prod 4"}, {"prod 7", "prod 3", "prod 2", "prod 1"}, {"prod 2", "prod 7", "prod 3", "prod 4", "prod 1"}, {"prod 1", "prod 3", "prod 4"}, {"prod 2", "prod 7", "prod 5"}};\)\)

MarketBasketAnalysis[byorderdata]

MarketBasketAnalysis has many options and the non-obvious are shortly presented next.  MaxNoThickArcs has default value of 7 and the larger the value of MaxNoThickArcs the larger the number of relationships displayed.  MaxNoThickArcs->k has the effect of ensuring that the largest amount of significant relationships or lines (not nodes) displayed on the web will not exceed k.  Selecting these k significant relationships implies the selection of a given number of significant nodes.  Once the significant nodes are chosen other non-significant relationships that connect these nodes are also included in the web if their associated proportion is not below MinProportionToPlot.  Thus, MinProportionToPlot is used to avoid showing non significant arcs.  Each arc's proportion is compared to MinProportionToPlot which is computed by dividing the arc's frequency by the total arc's frequencies calculated from the input data.

MarketBasketAnalysis // Options

MarketBasketAnalysis[
      byorderdata, "MinProportionToPlot" -> 
        0.0, "MaxNoThickArcs" -> 20]

Some options are associated to the way the analysis is performed.  DistinctOnly->True is used to make the analysis taking into account only the distinct products in each order.  "DistinctOnly"->True implies that a list like mb={a,b,a} will be transfomed into {a,b} before analyzing it because only the distinct ocurrences of events inside a list are of interest.  For example if mb represents a list of items bougth together in a store, then you can decide whether or not to analyze the order for distinct items only.  The cross sells in mb could be a-b with frequency 2 for "DistinctOnly"->False and the frequency will be 1 for "DistinctOnly"->True.

DataIsOrdered->True is used when the nature of the data requires for the arcs to be directional or that A-B is not the same as B-A and that these two events must be treated as different and counted individually.   Warning: Note that "DataIsOrdered" and "DistinctOnly" are options that will affect the analysis thus, changing when using "UsePreviousMBAnalysis"->True has no effect on the analysis.  In the example shown below the arc state 1 - state 7 has label 1+2 implying that the system moved twice from state 7 to state 1 and only once from state 1 to state 7.  The colors must be used to associate each numeric label with its origin node.

\(\(statesdata = {{"state 1", "state 3"}, {"state 7", "state 3"}, {"state 7", "state 1"}, {"state 3", "state 1"}, {"state 1", "state 7"}, {"state 7", "state 1"}, {"state 2", "state 7"}, {"state 1", "state 4"}};\)\)

MarketBasketAnalysis[
        statesdata, "DataIsOrdered" -> 
          True, "DistinctOnly" -> False];

In some cases you might want to use the results of the analysis on a larga data set to create several plots using new options  emphasizing on a selected set of products (or nodes).  To do this, once the first analysis has been performed you can set "UsePreviousMBAnalysis"->True and change the options as desired.  This will reduce the execution time since the analysis will not be performed again.  One potential application for  UsePreviousMBAnalysis is when you want to try different ways of displaying the results of the analysis by emphasizing on a given set of nodes.  En this case for example, the option EmphasizeTheseProducts->{"prod1","prod2",...} has the effect of eliminating from the web any node that does not connect to the products emphasized.  

Each arc has an associated weight which determines the arc thickness.  To calculate this weight consider that each arc has an associated total numeric label or frequency.  The arc with maximum frequency (mf) will have weight equal to one.  Other arcs will have weights proportional to their associated frequency or (arc's frequency)/mf.  Arcs with associated weight less than or equal to ChopValue will not be displayed on the web.

OutputType can take these values: "Plot", "Data", or "Data&Plot".  The data in the output is of the form {{{frequency1, arc1},...} for all the arcs resulting from the analysis without any filters, {{frequency1, arc1},...} only for those arcs presented in the plot}.

\(\(Show[
        GraphicsArray[{MarketBasketAnalysis[
              byorderdata, "MinProportionToPlot" -> 
                0.05, "MaxNoThickArcs" -> 
                10, "ChopValue" -> 0.4, 
              DisplayFunction -> Identity, 
              PlotLabel -> "ChopValue = .4", "EmphasizeTheseProducts" -> {"prod 2"}]
, 
MarketBasketAnalysis[
              byorderdata, "MinProportionToPlot" -> 
                0.05, "MaxNoThickArcs" -> 
                10, "ChopValue" -> 0, 
              DisplayFunction -> Identity, 
              PlotLabel -> "ChopValue = 0", "EmphasizeTheseProducts" -> {"prod 2"}]}, 
          DisplayFunction -> $DisplayFunction]];\)\)

GoalPlot

Nowadays organizations usually measure its effectiveness against a predefined set of metrics with their respective goals.  GoalPlot is designed to evaluate in time the relative achievement of each metric by comparing its initial value and its actual value to the predefined goal.  To simplify the way the information is portrayed in the plot all metrics are presented with a target of 100% implying that fully reaching each target means 100% compliance with the corresponding objective.  This plot is designed to be updated with the frequency that best fits the user needs such that the progress on each of the metrics is continuously available to the intended community.  Latter on in this section you will see how this plot can be joined with other plots to create a general overview of the behavior of metrics in time.

goaldata = {{"Costs", 2, 1.8, 1.8}, {"Productivity", 5, 4.65, 
            4.65}, {"Customer Satis", 100, 85, 
            85}, {"Incoming Quality", 30, 28, 
            28}, {"Outgoing Quality", 100, 90, 90}, {"Yield", 99, 97, 
            97}};
\(TableForm[goaldata, 
      TableHeadings -> {{}, {"Metric", "Goal", "Initial", "Actual"}}]\)

pl1 = GoalPlot[goaldata];

goaldata = {{"Costs", 2, 1.8, 1.9}, {"Productivity", 5, 5, 
            5}, {"Customer Satis", 100, 85, 80}, {"Incoming Quality", 
            30, 31, 33}, {"Outgoing Quality", 100, 90, 
            106}, {"Yield", 99, 99.5, 97}};
\(TableForm[goaldata, 
      TableHeadings -> {{}, {"Metric", "Goal", "Initial", "Actual"}}]\)

pl1 = GoalPlot[goaldata];

Some of the options of GoalPlot are exercised in the next example.

\(\(GoalPlot[
        goaldata, "PlotType" -> "Pareto", "TwoColors" -> {"Default", 
            PapayaWhip}, "FontStyle" -> {"FontFamily" -> "Helvetica", "FontSize" -> 11, "FontColor" -> 
              Blue}, "ShowData" -> True, "Min%" -> 
          75, "Max%" -> 200];\)\)

PrioritizationPlot

In many situations in practice decision makers are faced with a set of indicators from which a set of decisions must be made.  In theory, multivariate statistical analysis can be used to create a blend of all variables into single one and then only use that single combined variable to make decisions.  This is not widely accepted in many practical problems because of many reasons; one of them being the impossibility of reading from that compound variable what is happening in all the others.  In fact, what is usually done is to break back to the original variables to have a clear understanding of what is happening.  To avoid those problems PrioritizationPlot is a graphical representation of at most 5 variables into a two dimensional space.  There are infinite number of practical applications for this type of representation.  

In the following example PrioritizationPlot will be used to monitor the behavior of ten active accounts belonging to a certain company.  These accounts have been clustered into finance firms (Fin), Technology enterprises (Tec) or for the government (Gov).  The objective is to analyze how the accounts behaved in the last four quarters in terms of two main variables:  number of days since last purshase (recency) and the penetration opportunity on the account (To Penetrate).

data = {{{"Cmp1", "Q4", 20, 60, 20}, {"Cmp1", "Q3", 
              0, 19, 20}, {"Cmp1", "Q2", 23, 0, 
              20}, {"Cmp1", "Q1", 39, 0, 
              20}}, {{"Cmp2", "Q4", 31, 3, 
              20}, {"Cmp2", "Q3", 15, 18, 20}, {"Cmp2", "Q2", 
              24, 18, 20}, {"Cmp2", "Q1", 11, 17, 
              20}}, {{"Cmp3", "Q4", 7, 23, 
              20}, {"Cmp3", "Q3", 0, 51, 20}, {"Cmp3", "Q2", 
              84, 0, 20}, {"Cmp3", "Q1", 39, 16, 
              20}}, {{"Cmp4", "Q4", 8, 12, 
              20}, {"Cmp4", "Q3", 4, 0, 20}, {"Cmp4", "Q2", 
              39, 0, 20}, {"Cmp4", "Q1", 35, 0, 
              20}}, {{"Cmp5", "Q4", 18, 16, 
              20}, {"Cmp5", "Q3", 0, 0, 20}, {"Cmp5", "Q2", 
              0, 2, 20}, {"Cmp5", "Q1", 0, 0, 
              20}}, {{"Cmp6", "Q4", 58, 10, 
              20}, {"Cmp6", "Q3", 63, 175, 
              20}, {"Cmp6", "Q2", 62, 177, 
              20}, {"Cmp6", "Q1", 81, 173, 
              20}}, {{"Cmp7", "Q4", 14, 115, 
              15}, {"Cmp7", "Q3", 13, 54, 15}, {"Cmp7", "Q2", 
              22, 167, 15}, {"Cmp7", "Q1", 110, 124, 
              15}}, {{"Cmp8", "Q4", 2, 90, 
              15}, {"Cmp8", "Q3", 42, 36, 15}, {"Cmp8", "Q2", 
              49, 70, 15}, {"Cmp8", "Q1", 41, 100, 
              15}}, {{"Cmp9", "Q4", 30, 11, 
              25}, {"Cmp9", "Q3", 42, 22, 25}, {"Cmp9", "Q2", 
              55, 42, 25}, {"Cmp9", "Q1", 70, 77, 
              25}}, {{"Cmp10", "Q4", 4, 21, 
              20}, {"Cmp10", "Q3", 21, 0, 
              20}, {"Cmp10", "Q2", 15, 3, 
              20}, {"Cmp10", "Q1", 0, 36, 20}}};

The data above contains the key information to create a PrioritizationPlot.

data[[1]] // TableForm

From the Marketing Priorities plot below it can be seen that account 6 (Gov) shows a high recency (100 days) and $115M of penetration opportunity.  Account 5 has a lower penetration opportunity buy has a larger recency.  Thus, these two accounts are clear targets for action to improve the company's performance in the next quarter.  In general PrioritizationPlots should be used such that the upper right corner is the "action area" while the lower left corner is the target area for all circles in the plot.

\(\(pl1 = 
        Show[PrioritizationPlot[\({Last[#]} &\) /@ 
              data, "Marketing Priorities", "To Penetrate [M$]", "Recency [days]", {"Type of Company", "Fin", "Tec", "Gov"}, 15, True, 2, "Company"], 
          DisplayFunction -> $DisplayFunction];\)\)

If continuous effort is made to pursue the opportunities the effect of the company's effort can be monitored in a quarterly basis as shown below.  Note that most accounts are in better shape in Q4 than in Q1.  In this example the continuous improvement target should be to take all points to the axes origin (0, 0).

\(\(Show[
        PrioritizationPlot[
          data, "Marketing Priorities", "To Penetrate [M$]", "Recency [days]", {"Type of Company", "Fin", "Tec", "Gov"}, 15, True, 2, "Company"], 
        DisplayFunction -> $DisplayFunction];\)\)

ControlPanel

ControlPanel is a collage of GoalPlot, PrioritizationPlot and a MultipleListPlot to delve deeper on the information shown by GoalPlot in a particular time frame.  For example, consider that a company wants to monitor metrics such as Costs, Productivity, Customer Satisfaction, Incoming Quality, Outgoing Quality and Yield.  The list below (TimedMetrics) represents the values assosiated to these metrics in a quartely basis where the first element of the list represents the first quarter.  For each metric the goal, initial value and actual value are presented.

TimedMetrics = {{{  "Costs", 2, 1.8, 1.8}, {"Productivity", 5, 
                5, 5}, {"Cust Satisf", 4, 2.3, 2.3}, {"Incoming Q", 
                3.5, 2.5, 2.5}, {"Outgoing Q", 100, 90, 
                90}, {"Yield", 95, 90, 90}}, {{"Costs", 2, 1.8, 
                2.04}, {"Productivity", 5, 5, 4.7}, {"Cust Satisf", 
                4, 2.3, 3.11}, {"Incoming Q", 3.5, 2.5, 
                2.58}, {"Outgoing Q", 100, 90, 92.35}, {"Yield", 95, 
                90, 92.17}}, {{"Costs", 2, 1.8, 
                1.90}, {"Productivity", 5, 5, 4.354}, {"Cust Satisf", 
                4, 2.3, 3.213}, {"Incoming Q", 3.5, 2.5, 
                2.87}, {"Outgoing Q", 100, 90, 88.59}, {"Yield", 95, 
                90, 92.29}}, {{"Costs", 2, 1.8, 
                1.96}, {"Productivity", 5, 5, 4.55}, {"Cust Satisf", 
                4, 2.3, 3.22}, {"Incoming Q", 3.5, 2.5, 
                3.00}, {"Outgoing Q", 100, 90, 86.76}, {"Yield", 95, 
                90, 92.39}}, {{"Costs", 2, 1.8, 
                2.06}, {"Productivity", 5, 5, 4.47}, {"Cust Satisf", 
                4, 2.3, 3.17}, {"Incoming Q", 3.5, 2.5, 
                2.96}, {"Outgoing Q", 100, 90, 93.21}, {"Yield", 95, 
                90, 92.56}}, {{"Costs", 2, 1.8, 
                1.93}, {"Productivity", 5, 5, 4.22}, {"Cust Satisf", 
                4, 2.3, 3.51}, {"Incoming Q", 3.5, 2.5, 
                3.05}, {"Outgoing Q", 100, 90, 93.49}, {"Yield", 95, 
                90, 92.61}}};;

The ControlPanel below shows that even though Incoming Quality at ThisQuarter is almost 15% above its initial value it is not showing a significant growth rate in the last monts.  In the othre hand Productivity is not only below where it started but it is exhibiting a tendency to decrease in time.

\(plt = 
      ControlPanel[TimedMetrics, 
        6, "LowMedPrioritySets" -> {{"Costs", "Productivity", "Yield"}, {"Incoming Q", "Outgoing Q"}}]\)

\(ControlPanel[TimedMetrics, 
      6, "TimeSpans" -> {"ThisMonth", "LastMonth"}, "XLabel" -> "Month", "PeriodLength" -> 
        1, "LowMedPrioritySets" -> {{"Costs", "Productivity", "Yield"}, {"Incoming Q", "Outgoing Q"}}]\)

SPC

This section presents one example of the available control charts for statistical process control.  Charts for \(X\&_\), R, S, P, NP, C, nC, U, EWMA for averages, and multivariate (T^2) are available for individuals, fixed or variable sample sizes.  The function SPC analyzes a data set by using default \(X\&_\) chart.  out-of-control samples are removed from the data until a controlled process or state is reached.

data = 
        Table[RandomArray[
            NormalDistribution[100, 
              1], {3 + Random[DiscreteUniformDistribution[3]]}], {i, 1, 
            100}];
newdata = 
        Table[RandomArray[
            NormalDistribution[102.3, 
              3], {3 + Random[DiscreteUniformDistribution[3]]}], {i, 1, 
            10}];
data = Join[data, newdata];
Take[data, 5] // TableForm

cc = SPC[data]; // Timing

cc

Options[SPC]

SPC[
          Take[data, All], "SetupControl" -> 
            False, "ControlParameters" -> {"ControlMean" -> 
                100.0187, "ControlStandardDeviation" -> 1.0565}]; // 
      Timing

cc = SPC[data, "ChartType" -> {"XBar", "R"}]; // 
      Timing

Scheduled Function Evaluation

Several functions have been designed for scheduling any kind of Mathematica functions at the user's desired frequency.  Frequencies can be established in time spans like every 3 seconds, every two hrs, daily (at 6:00), the last day of the month, etc.

An important characteristic of these scheduling functions is that the schedule is stored at a directory specified by the user such that if for any reason the Mathematica session needs to be restarted when there are jobs in queue, when GoRealTime[] is executed again, the system will recognize all prescheduled jobs and will start running where it had stopped.  Another key characteristic is that you can define jobs that load jobs into the JobsQueue.  For example, if you need to perform 1000 analyses and produce the same number of reports every day at 6:00 am it might me wiser to load each of the 1000 analyses independently in the JobsQueue in case some unexpected system error occurs that locks your Mathematica session.  In this case when you restart your session your program will keep executing where it stopped and will not redo all the already performed analyses.  Note that by system error I mean something like non-responding database or any other thing that you cannot control.  If your Mathematica code is of high quality you should have no problems running 7x24.

DeleteJob /@ JobsDefined[];

    
DefineJob["Job1"] := (Print["Executing job 1 @ " <> 
              ToString[Date[]]];)
\(\(\(ScheduleJob["Job1", "Frequency" -> { .1, "Minutes"}, "Priority" -> 1];\)\(
\)
    \)\)
DefineJob["Job2"] := (Print["Executing job 2 @ " <> 
              ToString[Date[]]];)
\(\(\(ScheduleJob["Job2", "Frequency" -> { .25, "Minutes"}, "Priority" -> 
          2, "StartDate" -> {2006, 1, 1, 0, 0, 
            0}];\)\(
\)
    \)\)
DefineJob["Job3"] := (Print["Executing job 3 @ " <> 
              ToString[Date[]]];)
\(\(\(ScheduleJob["Job3", "Frequency" -> {1, "Days"}, "Priority" -> 
          2, "StartDate" -> {2006, 1, 1, 6, 30, 
            0}];\)\(
\)
    \)\)
DefineJob["Job4"] := 
        Print["Executing job 4 @ " <> 
            ToString[Date[]]];
\(\(ScheduleJob["Job4", "Frequency" -> {1, "Months"}, "Priority" -> 
          3, "StartDate" -> {2006, 1, 1, 6, 0, 0}];\)\)

GoRealTime[]

The functions JobsDefined, DeleteJob, GetJobsQueue, and DeleteJobsQueue, can be used to manage your scheduling activities.

?DeleteJob

JobsDefined[]