Data Science in the Wolfram Cloud

Dillon Tracy, Senior Kernel Developer at Wolfram Research

Join the Conversation #WolframTechConf

MouseOver

MouseOver

MouseOver

MouseOver

External Service: Instagram

A template notebook containing code for analyzing data from the Instagram service:

ActionMenu[Style["Open Instagram Template",{Blue,Bold}],{
"Desktop":>NotebookOpen[FileNameJoin[{NotebookDirectory[],"templates","Instagram-ImageIdentify.nb"}]],
"Cloud":>SystemOpen[toDevLink[CloudObject["WTC-2015/templates/Instagram-ImageIdentify.nb"]]]
}]

Visit the Image Identification Project to explore further.

Deployment

connectTo["Public"]

iiObj=CloudObject["WTC-2015/Yellowstone"];

Quiet[RemoveScheduledTask[iiObj]];

Deploy the template as an automated report:

CloudDeploy[
DocumentGenerator[
File@FileNameJoin[{NotebookDirectory[],"templates","Instagram-ImageIdentify.nb"}],
None,
NotificationFunction->All,
GeneratorOutputType->"CloudCDF",
DeliveryFunction->{{$WolframID}->"PDF",{"wrievents@wolfram.com"}->Automatic}
],iiObj
]

Trigger the generator to run immediately:

RunScheduledTask@@{iiObj}

log=DocumentGeneratorInformation[iiObj,"Log"]

Import[log,"Text"]//StringSplit[#,"\n"]&//Take[#,-Min[5,Length[#]]]&//Column

Obtain current output:

With[{out=DocumentGeneratorInformation[iiObj,"CurrentOutput"]},
Column[{out,Hyperlink@toDevLink[out]}]]

Precomputed

A report instance from a previous run:

With[{out=DocumentGeneratorInformation[CloudObject["WTC-2015/Yellowstone-pre"],"CurrentOutput"]},
Column[{out,Hyperlink@toDevLink[out]}]]

External Service: Twitter

A template notebook containing code for analyzing data from the Twitter service:

ActionMenu[Style["Open Twitter Template",{Orange,Bold}],{
"Desktop":>NotebookOpen[FileNameJoin[{NotebookDirectory[],"templates","Twitter.nb"}]],
"Cloud":>SystemOpen[toDevLink[CloudObject["WTC-2015/templates/Twitter.nb"]]]
}]

Deployment

twitObj=CloudObject["WTC-2015/Tweets"];

Quiet[RemoveScheduledTask[twitObj]];

Deploy the template as an automated report:

CloudDeploy[
DocumentGenerator[
File@FileNameJoin[{NotebookDirectory[],"templates","Twitter.nb"}],
None,
NotificationFunction->All,
GeneratorOutputType->"CloudCDF",
DeliveryFunction->{{$WolframID}->"PDF",{"wrievents@wolfram.com"}->Automatic}
],
twitObj
]

Trigger the generator to run immediately:

RunScheduledTask@@{twitObj}

log=DocumentGeneratorInformation[twitObj,"Log"]

Import[log,"Text"]//StringSplit[#,"\n"]&//Take[#,-Min[5,Length[#]]]&//Column

Obtain current output:

With[{out=DocumentGeneratorInformation[twitObj,"CurrentOutput"]},
Column[{out,Hyperlink@toDevLink[out]}]]

Precomputed

A report instance from a previous run:

With[{out=DocumentGeneratorInformation[CloudObject["WTC-2015/Tweets-pre"],"CurrentOutput"]},
Column[{out,Hyperlink@toDevLink[out]}]]

Online Spreadsheet

A template notebook containing code for analyzing house utilities data stored in a spreadsheet:

ActionMenu[Style["Open Utilities Template",{Brown,Bold}],{
"Desktop":>NotebookOpen[FileNameJoin[{NotebookDirectory[],"templates","Utilities.nb"}]],
"Cloud":>SystemOpen[toDevLink[CloudObject["WTC-2015/templates/Utilities.nb"]]]
}]

Deployment

utilObj=CloudObject["WTC-2015/Utilities"];

Quiet[RemoveScheduledTask[utilObj]];

Deploy the template as an automated report:

CloudDeploy[
DocumentGenerator[
File@FileNameJoin[{NotebookDirectory[],"templates","Utilities.nb"}],
None,
NotificationFunction->All,
GeneratorOutputType->"CloudCDF",
DeliveryFunction->{{$WolframID}->"PDF",{"wrievents@wolfram.com"}->Automatic}
],
utilObj
]

Trigger the generator to run immediately:

RunScheduledTask@@{utilObj}

log=DocumentGeneratorInformation[utilObj,"Log"]

Import[log,"Text"]//StringSplit[#,"\n"]&//Take[#,-Min[5,Length[#]]]&//Column

Obtain current output:

With[{out=DocumentGeneratorInformation[utilObj,"CurrentOutput"]},
Column[{out,Hyperlink@toDevLink[out]}]]

Precomputed

A report instance from a previous run:

With[{out=DocumentGeneratorInformation[CloudObject["WTC-2015/Utilities-pre"],"CurrentOutput"]},
Column[{out,Hyperlink@toDevLink[out]}]]

Wolfram Data Drop

A template notebook containing code for analyzing data from a sensor in our sixth floor breakroom fridge:

ActionMenu[Style["Open Fridge Template",{Purple,Bold}],{
"Desktop":>NotebookOpen[FileNameJoin[{NotebookDirectory[],"templates","Fridge.nb"}]],
"Cloud":>SystemOpen[StringReplace[First@CloudObject["WTC-2015/templates/Fridge.nb"],"://www."->"://dev."]]
}]

Deployment

fridgeObj=CloudObject["WTC-2015/Fridge"];

Quiet[RemoveScheduledTask[fridgeObj]];

Deploy the template as an automated report:

CloudDeploy[
DocumentGenerator[
File@FileNameJoin[{NotebookDirectory[],"templates","Fridge.nb"}],
None,
NotificationFunction->All,
GeneratorOutputType->"CloudCDF",
DeliveryFunction->{{$WolframID}->"PDF",{"wrievents@wolfram.com"}->Automatic}
],
fridgeObj
]

Trigger the generator to run immediately:

RunScheduledTask@@{fridgeObj}

log=DocumentGeneratorInformation[fridgeObj,"Log"]

Import[log,"Text"]//StringSplit[#,"\n"]&//Take[#,-Min[5,Length[#]]]&//Column

Obtain current output:

With[{out=DocumentGeneratorInformation[fridgeObj,"CurrentOutput"]},
Column[{out,Hyperlink@toDevLink[out]}]]

Precomputed

A report instance from a previous run:

With[{out=DocumentGeneratorInformation[CloudObject["WTC-2015/Fridge-pre"],"CurrentOutput"]},
Column[{out,Hyperlink@toDevLink[out]}]]

Public API

A template notebook containing code for analyzing data from a City of Chicago feed:

ActionMenu[Style["Open Chicago Crime Template",{Red,Bold}],{
"Desktop":>NotebookOpen[FileNameJoin[{NotebookDirectory[],"templates","Crime.nb"}]],
"Cloud":>SystemOpen[toDevLink[CloudObject["WTC-2015/templates/Crime.nb"]]]
}]

Deployment

crimeObj=CloudObject["WTC-2015/Crime"];

Quiet[RemoveScheduledTask[crimeObj]];

Deploy the template as an automated report:

CloudDeploy[
DocumentGenerator[
File@FileNameJoin[{NotebookDirectory[],"templates","Crime.nb"}],
None,
NotificationFunction->All,
GeneratorOutputType->"CloudCDF",
DeliveryFunction->{{$WolframID}->"PDF",{"wrievents@wolfram.com"}->Automatic}
],
crimeObj
]

Trigger the generator to run immediately:

RunScheduledTask@@{crimeObj}

log=DocumentGeneratorInformation[crimeObj,"Log"]

Import[log,"Text"]//StringSplit[#,"\n"]&//Take[#,-Min[5,Length[#]]]&//Column

Obtain current output:

With[{out=DocumentGeneratorInformation[crimeObj,"CurrentOutput"]},
Column[{out,Hyperlink@toDevLink[out]}]]

Precomputed

A report instance from a previous run:

With[{out=DocumentGeneratorInformation[CloudObject["WTC-2015/Crime-pre"],"CurrentOutput"]},
Column[{out,Hyperlink@toDevLink[out]}]]

Reporting Application: Debt Profiles

Automate debt profile reports on bond series. The application requirements are:

Source data is CSV

Output is presentation-quality PDF

Process must be cloud-resident

Must be able to drive the entire process from code

ActionMenu[Style["View Bloomberg Data",{Darker@Green,Bold}],{
"Refunded Cusip":>SystemOpen[inputPath1],
"Credits and Fields":>SystemOpen[inputPath2]
}]

Create debt profiles from raw data (desktop)

debtProfileDataset=debtProfileOneFolder`createDebtProfile[inputPath1,inputPath2]

{titles, debtProfileByGroups, columnLabels}=debtProfileOneFolder`debtProfileToComponents[debtProfileDataset];
Length/@{titles,debtProfileByGroups}

Generate a document from the template (desktop)

ActionMenu[Style["Open Debt Profile Template",{Darker@Green,Bold}],{
"Desktop":>NotebookOpen[debtProfileTemplate]
}]

Drive this template with a list of series:

dpObj=GenerateDocument[debtProfileTemplate,<|
"headers"->columnLabels,
"groups"->MapThread[<|
"identifier"->Last@StringSplit[#1[[2]],WhitespaceCharacter..],
"label"->#1,
"profile"->#2
|>&,{titles,debtProfileByGroups}]
|>]

Export this document as a PDF:

Export[outputPath,dpObj]

SystemOpen[outputPath]

Reporting Application: Cloud

Deploy to cloud as DocumentGenerator for scheduling, distribution etc.:

Deployment

dpCloudObj=CloudObject["WTC-2015/debt-profiles"];

Quiet[RemoveScheduledTask[dpCloudObj];DeleteFile[dpCloudObj]];FileExistsQ[dpCloudObj]

CloudDeploy[
DocumentGenerator[
File[debtProfileTemplate],
driverFn[input1Cloud,input2Cloud],(* generate the data used to fill the template *)
None,(* no schedule, on demand only *)
NotificationFunction->All,
DeliveryFunction->{{$WolframID,"wrievents@wolfram.com"}->"PDF"},
GeneratorOutputType->"CloudCDF",
EpilogFunction->myEpilog(* set printing options on the output *)
],
dpCloudObj
]

Tickle it to run synchronously and out-of-sequence.  There will be no delivery or notification.  The output link will be a document:

EvaluateScheduledTask@@{dpCloudObj}

Tickle it to run asynchronously and officially.  The return value is the document generator object (not a document):

RunScheduledTask@@{dpCloudObj}

Examine the log from cloud or desktop (also in notification email):

log=DocumentGeneratorInformation[dpCloudObj,"Log"]

Import[log,"Text"]//StringSplit[#,"\n"]&//Take[#,-Min[5,Length[#]]]&//Column

Obtain current output (as linked in delivery email):

With[{out=DocumentGeneratorInformation[dpCloudObj,"CurrentOutput"]},
Column[{out,Hyperlink@toDevLink[out]}]]

Precomputed

SystemOpen[FileNameJoin[{NotebookDirectory[],"Fine","debt-profiles-cloud.pdf"}]]

Cloud-enabled driver and epilog functions

Input: data files, output: pure function that generates a structure for filling the template.

driverFn[input1_,input2_]:=Module[
{driver,titles, debtProfileByGroups, columnLabels,newDataset},
	
Get["debtProfileOneFolder`"];

newDataset=debtProfileOneFolder`createDebtProfile[input1,input2];

{titles, debtProfileByGroups, columnLabels}=debtProfileOneFolder`debtProfileToComponents[newDataset];

driver=<|
"headers"->columnLabels,
"groups"->MapThread[<|
"identifier"->Last@StringSplit[#1[[2]],WhitespaceCharacter..],
"label"->#1,
"profile"->#2
|>&,{titles,debtProfileByGroups}]
|>;
driver
]&;

driverFn[input1Cloud,input2Cloud][]["groups"]//Length

Verify this works in the cloud:

CloudEvaluate[
Length@driverFn[input1Cloud,input2Cloud][]["groups"],Hold
]

The epilog will set the options on the generated document:

myEpilog=Function[{fn},
Module[{nbObj,h1,h2,f1,f2,date},
UsingFrontEnd[
nbObj=NotebookOpen[fn];

h1="See Disclaimer - Back Page";
h2="Sample Issuer\nRefunding Analysis";
f1="Prepared by FTN Financial Capital Markets";f2="Data Source: Bloomberg";
date=DateString[{"Month","/","Day","/","Year"}];

SetOptions[nbObj,
PrintingOptions-> {"FacingPages" -> False},
PageHeaders ->{
{h1, h2, Cell[TextData[{"Page: ",CounterBox["Page"]}]]},{h1, h2, Cell[TextData[{"Page: ",CounterBox["Page"]}]]}
},
PageFooters ->{
{f1, f2, Cell[TextData[{date}]]},
{f1, f2, Cell[TextData[{date}]]}
},
PageFooterLines-> {True, True};
PrintingOptions-> {"PaperOrientation" -> "Landscape"},		PrintingOptions-> {"PrintingMargins"-> {{5, 5}, {72, 72}}},		PrintingOptions-> {"PageHeaderMargins"-> {Automatic, Automatic}},		PrintingOptions-> {"PageFooterMargins"-> {15,15}},		PrintingOptions-> {"FirstPageHeader" -> True},		PrintingOptions-> {"PageSize" -> {792, 612}},		(*PrintingOptions-> {"PaperSize" -> {1008, 612}},*)		PrintingOptions-> {"FirstPageFooter" -> True},		PrintingOptions-> {"PrintMultipleHorizontalPages" -> False}
];

NotebookSave[nbObj]
]
]
];

Database Application: Linear Optimization

Perform a linear optimization subject to some constraints. 

Minimize x+2y, subject to constraints x+y≥26 and -5x+y=5, and bounds x≥3, y≥4:

sol=LinearProgramming[
{1,2},
{{-5,1},{1,1}},{{7,0},{26,1}},
{{3,∞},{4,∞}}
]

x+2 y/. Thread[{x,y}->sol]//N

The application requirements are:

The constraints must be stored in a database.

Application users must be able to add constraints from a web client.

Application users must be able to request an optimization solution based on the current contents of the database.

Bonus:  visualize the solution so we can see the consequences of suboptimal solutions.

Building this will introduce some the cloud concepts of APIFunction and FormFunction.

Database Application: Visualizing the Solution

Visualize the solution:

vis=With[{k0=x+2 y/. Thread[{x,y}->sol]},Manipulate[Show[RegionPlot[{x+y≥26,x≥3,y≥4},{x,0,7},{y,15,25},PlotLegends->"Expressions"],ContourPlot[{x+2 y==k,-5 x+y==7},{x,0,7},{y,15,25},PlotLegends->"Expressions",ContourStyle->44],Graphics[{PointSize[Large],Red,Point[sol]}]],{{k,k0},40,60}]]

Make this visualization accessible in the cloud:

manipObj=CloudDeploy[vis, "WTC-2015/optim/vis",Permissions->"Public"];

Column[{manipObj,Hyperlink@toDevLink@manipObj}]

Database Application: Private Clouds

Features and interface similar to Public Cloud

Enterprise product, hosted on your intranet

Private to you, so safe to connect to secure data sources

Database Application: Storing Constraints

Locally

Connect to a (local) SQLite database:

conn=OpenSQLConnection[JDBC["SQLite",FileNameJoin[{$UserBaseDirectory,"DatabaseResources","optim.db"}]]]

Create a constraints table:

Quiet[SQLDropTable[conn,"constraints"]];
SQLCreateTable[conn,SQLTable["constraints"],{
SQLColumn["id","DataTypeName"->"INTEGER","PrimaryKey"->True],
SQLColumn["a","DataTypeName"->"FLOAT"],
SQLColumn["b","DataTypeName"->"FLOAT"],
SQLColumn["c","DataTypeName"->"FLOAT"],
SQLColumn["inequality","DataTypeName"->"INTEGER"]
}]

Seed the table:

seed=With[{ab={{-5, 1}, {1, 1}, {1, 0}, {0, 1}},ci={{7, 0}, {26, 1}, {3, 1}, {4, 1}}},
Join[ab,ci,2]
]

SQLInsert[conn,"constraints",{"a","b","c","inequality"},seed]

CloseSQLConnection[conn];

Read the database contents:

readConstraints[]

visualizeFromDatabase[]

In the (Private) Cloud

Prepare a cloud-resident version of the same database:

connectTo["Private"]

CloudEvaluate[resetDatabase[]];

CloudEvaluate[readConstraints[],Hold]

CloudEvaluate[visualizeFromDatabase[]]

Database Application: FormFunction

Locally

Construct a form whose contents will be used to update the database:

form=FormFunction[{
{"a","Coeff a"}-><|"Interpreter"->"Number","Hint"->-0.2|>,
{"b","Coeff b"}-><|"Interpreter"->"Number","Hint"->1|>,
{"c","Const c"}-><|"Interpreter"->"Number","Hint"->22.7|>,
"inequality"-><|"Interpreter"->{"less than or equal to"->-1,"equal to"->0,"greater than or equal to"->1},"Hint"->1|>
},
addConstraint[#]&,
AppearanceRules-><|"Title" -> "Add a constraint", "Description" -> "Input coefficients, constants, and conditions for a new optimization constraint"|>
];

Form object can be used interactively or programmatically:

form[]

form[<|"a"->-.2,"b"->1,"c"->22.7,"inequality"->1|>]

visualizeFromDatabase[]

In the Cloud

CloudDeploy[form]

Visualize the solution with the new constraint:

CloudEvaluate[visualizeFromDatabase[]]

Database Application: APIFunction

Locally

Construct a form whose contents will be used to update the database:

api=APIFunction[{"a" -> "Number", "b" -> "Number","c"->"Number","inequality"->Restricted["Integer", {-1,1}]->1},addConstraint[#]&];

The API can be used programmatically:

api[<|"a"->-0.3,"b"->1.2,"c"->28.4,"inequality"->1|>]

visualizeFromDatabase[]

In the Cloud

Deploy the api for use by web clients:

webApi=CloudDeploy[api]

Construct a web request to this api:

Hyperlink[First[webApi]<>"?a=-0.3&b=1.2&c=28.4&inequality=1"]

widget=CloudEvaluate[visualizeFromDatabase[]]

Database Application: Deployment

A template that reports the optimal solution based on the constraints on any given day:

ActionMenu[Style["Open Optimization Template",{Darker@Green,Bold}],{
"Desktop":>NotebookOpen[FileNameJoin[{NotebookDirectory[],"templates","Optimization.nb"}]],
"Cloud":>SystemOpen[toDevLink[CloudObject["WTC-2015/templates/Optimization.nb"]]]
}]

Generate HTML for embedding the interactive visualization in a web page:

cloudWidget=CloudDeploy[widget];

EmbedCode[cloudWidget]

Export a minimal web page containing the visualization:

webPage=StringTemplate["
<!DOCTYPE html PUBLIC \"-//W3C//DTD HTML 4.01//EN\"
    \"http://www.w3.org/TR/html4/strict.dtd\">
<html lang=\"en\">
  <head>
    <meta http-equiv=\"content-type\" content=\"text/html; charset=utf-8\">
    <title>title</title>
    <link rel=\"stylesheet\" type=\"text/css\" href=\"style.css\">
    <script type=\"text/javascript\" src=\"script.js\"></script>
  </head>
  <body>
  <h2>Web page embedding deployed content</h2>
`1`
  </body>
</html>"]["<iframe src=\"https://www.qa-pc.wolfram.com/objects/5dad30cc-73e8-453c-8e72-7754ea658c5a?_embed=iframe\" width=\"600\" height=\"800\" />"];

Export[FileNameJoin[{NotebookDirectory[],"optim.html"}],webPage,"String"]

SystemOpen[%]

Generate e.g. the C# for invoking the instant API function:

EmbedCode[api,"C#"]

Summary

Data in -> computation -> computable cloud document out

Cloud data science solves the dissemination problem

Pure cloud and hybrid desktop workflows supported

“Computation” includes knowledge-based analytics and semantic interpretation

Private clouds enable cloud workflow for data sources you don’t want to expose

Wolfram Data Science Platform will offer features for streamlining these workflows

Everything presented here is doable with the current cloud

dillont@wolfram.com

Init

Quit

1;

Cloud

$cloud="Test";

$cloud="Local";

$cloud="Public";

(*PacletDirectoryAdd["/Users/dillont/eclipse/workspace/CloudObject/"]*)

FindFile["CloudObject`"]

SetDirectory[NotebookDirectory[]]

<<CloudObject`;

$CloudBase="http://datadrop.devel.wolframcloud.com/";

$CloudBase="http://www.devel.wolframcloud.com/";

$CloudBase="http://localhost/";

$CloudBase="http://www.test.wolframcloud.com/";

(*CloudConnect[]*)

connectTo["Test"]:=(
$CloudBase="http://www.test.wolframcloud.com";
CloudConnect[]);

connectTo["Devel"]:=(
$CloudBase="http://www.devel.wolframcloud.com";
CloudConnect[]);

connectTo["Local"]:=(
$CloudBase="http://localhost/";
CloudConnect[]);

connectTo["Public"]:=($CloudBase="https://www.wolframcloud.com";
CloudConnect[]);

connectTo["Private"]:=($CloudBase="https://www.qa-pc.wolfram.com";
CloudObject`Private`makeAuthURL["www.qa-pc.wolfram.com"]:="https://www.qa-pc.wolfram.com/oauth/access_token";
CloudObject`Private`$LocalhostAuthURL="https://www.qa-pc.wolfram.com/oauth/access_token";
CloudObject`Private`makeSignatureURL["www.qa-pc.wolfram.com"]:="https://qa-pc.wolfram.com";
CloudConnect[]
);

connectTo["test1"]:=($CloudBase="https://www.test1.wolframenterprise.com";
CloudObject`Private`makeAuthURL["www.test1.wolframenterprise.com"]:="https://www.test1.wolframenterprise.com/oauth/access_token";
CloudObject`Private`$LocalhostAuthURL="https://www.test1.wolframenterprise.com/oauth/access_token";
CloudObject`Private`makeSignatureURL["www.test1.wolframenterprise.com"]:="https://test1.wolframenterprise.com";
CloudConnect[]
);

connectTo[$cloud]

CloudEvaluate[1+1]

mn[]:=$MachineName
CloudEvaluate[{mn[],$MachineName,DownValues[mn]}]

toDevLink[co_CloudObject]:=StringReplace[First@co,"://www."->"://dev."]

Optimization Example Helpers

Needs["DatabaseLink`"]

sol=LinearProgramming[{1,2},{{-5,1},{1,1}},{{7,0},{26,1}},{{3,∞},{4,∞}}]

resetDatabase[]:=Module[
{conn,seed},
conn=DatabaseLink`OpenSQLConnection[DatabaseLink`JDBC["SQLite",FileNameJoin[{$UserBaseDirectory,"DatabaseResources","optim.db"}]]];
Quiet[DatabaseLink`SQLDropTable[conn,"constraints"]];
DatabaseLink`SQLCreateTable[conn,DatabaseLink`SQLTable["constraints"],{
DatabaseLink`SQLColumn["id","DataTypeName"->"INTEGER","PrimaryKey"->True],
DatabaseLink`SQLColumn["a","DataTypeName"->"FLOAT"],
DatabaseLink`SQLColumn["b","DataTypeName"->"FLOAT"],
DatabaseLink`SQLColumn["c","DataTypeName"->"FLOAT"],
DatabaseLink`SQLColumn["inequality","DataTypeName"->"INTEGER"]
}];
seed=With[{ab={{-5, 1}, {1, 1}, {1, 0}, {0, 1}},ci={{7, 0}, {26, 1}, {3, 1}, {4, 1}}},
Join[ab,ci,2]
];
DatabaseLink`SQLInsert[conn,"constraints",{"a","b","c","inequality"},seed];
DatabaseLink`CloseSQLConnection[conn]
]

wipeDatabase[]:=Module[
{conn,seed},
conn=DatabaseLink`OpenSQLConnection[DatabaseLink`JDBC["SQLite",FileNameJoin[{$UserBaseDirectory,"DatabaseResources","optim.db"}]]];
Quiet[DatabaseLink`SQLDropTable[conn,"constraints"]];
DatabaseLink`SQLCreateTable[conn,DatabaseLink`SQLTable["constraints"],{
DatabaseLink`SQLColumn["id","DataTypeName"->"INTEGER","PrimaryKey"->True],
DatabaseLink`SQLColumn["a","DataTypeName"->"FLOAT"],
DatabaseLink`SQLColumn["b","DataTypeName"->"FLOAT"],
DatabaseLink`SQLColumn["c","DataTypeName"->"FLOAT"],
DatabaseLink`SQLColumn["inequality","DataTypeName"->"INTEGER"]
}];
DatabaseLink`CloseSQLConnection[conn]
]

resetCloudDatabase[]:=Module[{obj=CloudSave[resetDatabase],res},
res=With[{o=obj},CloudEvaluate[CloudGet[o];resetDatabase[]]];
DeleteFile[obj];
res
]

readConstraints[]:=Module[{conn,raw,res},Needs["DatabaseLink`"];conn=DatabaseLink`OpenSQLConnection[DatabaseLink`JDBC["SQLite",FileNameJoin[{$UserBaseDirectory,"DatabaseResources","optim.db"}]]];raw=DatabaseLink`SQLSelect[conn,"constraints"];res={raw[[All,2;;All]][[All,1;;-3]],raw[[All,2;;All]][[All,-2;;All]]};DatabaseLink`CloseSQLConnection[conn];res]

addConstraint[row_]:=Module[{conn,raw,res},Needs["DatabaseLink`"];conn=DatabaseLink`OpenSQLConnection[DatabaseLink`JDBC["SQLite",FileNameJoin[{$UserBaseDirectory,"DatabaseResources","optim.db"}]]];res=DatabaseLink`SQLInsert[conn,"constraints",{"a","b","c","inequality"},{row["a"],row["b"],row["c"],row["inequality"]}];DatabaseLink`CloseSQLConnection[conn];
res
]

optimizeFromDatabase[___]:=Module[{c=readConstraints[],res},res=LinearProgramming[{1,2},c[[1]],c[[2]]];res]

inequalityToHead[i_Integer]:=Which[i==1,GreaterEqual,i==-1,LessEqual,True,Equal]

visualizeFromDatabase[]:=Module[
{c=readConstraints[],sol,regionPos,contourPos,regions,contours},
sol=LinearProgramming[{1,2},c[[1]],c[[2]]];regionPos=Position[c[[2]],{_,in_}/;in≠0];contourPos=Position[c[[2]],{_,in_}/;in==0];regions=MapThread[inequalityToHead[#2[[-1]]][#1[[1]] x+#1[[2]] y,#2[[1]]]&,(Extract[#1,regionPos]&)/@c];contours=MapThread[inequalityToHead[#2[[-1]]][#1[[1]] x+#1[[2]] y,#2[[1]]]&,(Extract[#1,contourPos]&)/@c];With[{k0=x+2 y/. Thread[{x,y}->sol],reg=regions,con=contours,s=sol},
Manipulate[
Show[RegionPlot[reg,{x,0,5},{y,20,25},PlotLegends->"Expressions"],With[{cp=Prepend[con,x+2 y==k]},ContourPlot[cp,{x,0,5},{y,20,25},PlotLegends->"Expressions",ContourStyle->44]],Graphics[{PointSize[Large],Red,Point[s]}]],
{{k,k0},40,60}
]
]
]

Fine Helpers

Setup

inputPath1=FileNameJoin[{NotebookDirectory[],"Fine","TMFGO_CompleteBondsCreditsAndFields.csv"}];
inputPath2=FileNameJoin[{NotebookDirectory[],"Fine","TxdotTmfGoUniqueRefundedCusipRevised.csv"}];
outputPath=FileNameJoin[{NotebookDirectory[],"Fine","debt-profiles.pdf"}];
input1Cloud=CloudObject["WTC-2015/debt-profiles-data/"<>FileNameTake[inputPath1,-1]];
input2Cloud=CloudObject["WTC-2015/debt-profiles-data/"<>FileNameTake[inputPath2,-1]];
debtProfileTemplate=FileNameJoin[{NotebookDirectory[],"Fine","debtProfileTemplate.nb"}];

Copy data resources to cloud (one-time)

CopyFile[inputPath1,input1Cloud]

CopyFile[inputPath2,input2Cloud]

Get["debtProfileOneFolder`"]

Copy package somewhere the template will be able to find it

Desktop:

DeleteFile[FileNameJoin[{$UserBaseDirectory,"Applications","debtProfileOneFolder.m"}]];
CopyFile[
FileNameJoin[{NotebookDirectory[],"Fine","debtProfileOneFolder.m"}],
FileNameJoin[{$UserBaseDirectory,"Applications","debtProfileOneFolder.m"}]
]

Cloud:

DeleteFile[CloudObject[{"Base","Applications","debtProfileOneFolder.m"}]];
CopyFile[
FileNameJoin[{NotebookDirectory[],"Fine","debtProfileOneFolder.m"}],
CloudObject[FileNameJoin[{"Base","Applications","debtProfileOneFolder.m"}]]
]

{FindFile["debtProfileOneFolder`"],CloudEvaluate[FindFile["debtProfileOneFolder`"]]}

Set options on template nb (desktop)

tObj=NotebookOpen[debtProfileTemplate];

h2="Sample Issuer\nRefunding Analysis";
reportDate=DateString[];

h1="See Disclaimer - Back Page";
f1="WTC 2015";
f2="Data Source: Bloomberg";

SetOptions[tObj,
PrintingOptions-> {"FacingPages" -> False},
PageHeaders ->{
{h1, h2, Cell[TextData[{"Page: ",CounterBox["Page"]}]]},
{h1, h2, Cell[TextData[{"Page: ",CounterBox["Page"]}]]}
},
PageFooters ->{
{f1, f2, Cell[TextData[{reportDate}]]},
{f1, f2, Cell[TextData[{reportDate}]]}
},
PageFooterLines-> {True, True};
PrintingOptions-> {"PaperOrientation" -> "Landscape"},		PrintingOptions-> {"PrintingMargins"-> {{5, 5}, {72, 72}}},		PrintingOptions-> {"PageHeaderMargins"-> {Automatic, Automatic}},		PrintingOptions-> {"PageFooterMargins"-> {15,15}},		PrintingOptions-> {"FirstPageHeader" -> True},		PrintingOptions-> {"PageSize" -> {792, 612}},		(*PrintingOptions-> {"PaperSize" -> {1008, 612}},*)		PrintingOptions-> {"FirstPageFooter" -> True},		PrintingOptions-> {"PrintMultipleHorizontalPages" -> False}
]

NotebookSave[tObj]

Paclet

PacletSites[]

PacletSiteRemove["http://paclet-int.wolfram.com:8080/PacletServerInternal"]

PacletSiteRemove["http://pacletserver.wolfram.com"]

PacletSiteAdd["http://paclet-int.wolfram.com:8080/PacletServerInternal"]

PacletSiteAdd["http://pacletserver.wolfram.com"]

PacletSiteUpdate/@PacletSites[]

PacletUpdate["CloudObject"]

PacletInformation["CloudObject"]

PacletFindRemote["CloudObject"]

PacletUninstall["CloudObject"]

Special Cells

Picture Cell

PicturePrint[expr_,opts:OptionsPattern[Cell]]:=CellPrint[TextCell[expr,"Picture",opts,ShowStringCharacters->False]]

Overview Mouseover

OverviewMouseover[header_,subs_]:=CellPrint@Cell[BoxData[ToBoxes@Mouseover[Framed[Style[header,"OverviewSection"],FrameMargins->{{15,15},{5,5}},BoxFrame->2,FrameStyle->None,RoundingRadius->5],

Evaluate@If[subs=="",Framed[Style[header,"OverviewSection",White],FrameMargins->{{15,15},{5,5}},BoxFrame->2,FrameStyle->None,RoundingRadius->5,Background->m8red[1]],Row[{Framed[Style[header,"OverviewSection",White],FrameMargins->{{15,15},{5,5}},FrameStyle->None,RoundingRadius->5,Background->m8red[1]]," » ",Framed[Style[subs,"OverviewSection"],FrameMargins->{{15,15},{5,5}},FrameStyle->m8red[1],BoxFrame->2,RoundingRadius->5,Background->White]}]]]],"OverviewSection"]

(*MapThread[OverviewMouseover,Transpose[{{"Chapter One","Chapter One topics, …"},{"Chapter Two","Chapter Two topics, …"},{"Chapter Three","Chapter Three topics, …"},{"Chapter Four","Chapter Four topics, …"}}]]*)

Large Overview Mouseover

LargeOverviewMouseover[header_,subs_]:=CellPrint@Cell[BoxData[ToBoxes@Mouseover[Framed[Style[header,"LargeOverviewSection",FontFamily->"Helvetica"],FrameMargins->{{15,15},{5,5}},FrameStyle->None,RoundingRadius->5],

Evaluate@If[subs=="",Framed[Style[header,"LargeOverviewSection",White],FrameMargins->{{15,15},{5,5}},FrameStyle->None,RoundingRadius->5,Background->Orange],Row[{Framed[Style[header,"LargeOverviewSection",{FontFamily->"Helvetica",White}],FrameMargins->{{15,15},{5,5}},FrameStyle->None,RoundingRadius->5,Background->Orange]," » ",Framed[Style[subs,"LargeOverviewSection",FontFamily->"Helvetica"],FrameMargins->{{15,15},{5,5}},FrameStyle->Orange,RoundingRadius->5,Background->White]}]]]],"LargeOverviewSection"]

DG removal

Template upload

CopyFile[
FileNameJoin[{NotebookDirectory[],"templates","Instagram-ImageIdentify.nb"}],
CloudObject["WTC-2015/templates/Instagram-ImageIdentify.nb"]
]

CopyFile[
FileNameJoin[{NotebookDirectory[],"templates","Twitter.nb"}],
CloudObject["WTC-2015/templates/Twitter.nb"]
]

CopyFile[
FileNameJoin[{NotebookDirectory[],"templates","Fridge.nb"}],
CloudObject["WTC-2015/templates/Fridge.nb"]
]

CopyFile[
FileNameJoin[{NotebookDirectory[],"templates","Crime.nb"}],
CloudObject["WTC-2015/templates/Crime.nb"]
]

CopyFile[
FileNameJoin[{NotebookDirectory[],"templates","Utilities.nb"}],
CloudObject["WTC-2015/templates/Utilities.nb"]
]

s/b private:

connectTo["Private"]

CopyFile[
FileNameJoin[{NotebookDirectory[],"templates","Optimization.nb"}],
CloudObject["WTC-2015/templates/Optimization.nb"]
]

connectTo["Public"]

Suspend WTD jobs

connectTo["Local"]

StopScheduledTask[CloudObject["WTD-2015/Yellowstone"]]
StopScheduledTask[CloudObject["WTD-2015/Yellowstone-pre"]]

StopScheduledTask[CloudObject["WTD-2015/Crime"]]
StopScheduledTask[CloudObject["WTD-2015/Crime-pre"]]

StopScheduledTask[CloudObject["WTD-2015/Tweets"]]
StopScheduledTask[CloudObject["WTD-2015/Tweets-pre"]]

StopScheduledTask[CloudObject["WTD-2015/twit"]]
StopScheduledTask[CloudObject["WTD-2015/twit-pre"]]

StopScheduledTask[CloudObject["WTD-2015/Fridge"]]
StopScheduledTask[CloudObject["WTD-2015/Fridge-pre"]]

StopScheduledTask[CloudObject["WTD-2015/Utilities"]]
StopScheduledTask[CloudObject["WTD-2015/Utilities-pre"]]

Scratch

connectTo["Private"]

resetDatabase[]

CloudEvaluate[
Needs["DatabaseLink`"];
conn=DatabaseLink`OpenSQLConnection[DatabaseLink`JDBC["SQLite",FileNameJoin[{$UserBaseDirectory,"DatabaseResources","optim.db"}]]];
SQLConnectionUsableQ[conn]
]

CloudEvaluate[
DatabaseLink`CloseSQLConnection[SQLConnections[]]
]

CloudEvaluate[resetDatabase[]]

resetCloudDatabase[]

CloudEvaluate[
Needs["DatabaseLink`"];
DatabaseLink`OpenSQLConnection[DatabaseLink`JDBC["SQLite",FileNameJoin[{"optim.db"}]]],
Hold
]

CloudEvaluate[resetDatabase[]]

readConstraints[]

CloudEvaluate[addConstraint[<|"a"->-.2,"b"->1,"c"->22.7,"inequality"->1|>]]

CloudEvaluate[readConstraints[]]

MapThread[LargeOverviewMouseover,Transpose[{
{"Import","Files, Public APIs, External Services, Databases ..."},
{"Analyze","Time Series, GeoGraphics, Machine Learning ..."},
{"Publish","Cloud Deployment, Templating, Document Generators ..."},
{"Examples","Instagram, Twitter, Utilities, Crime, Refrigerators ..."}
}]
];

CloudDeploy[
DocumentGenerator[
File@FileNameJoin[{NotebookDirectory[],"templates","Instagram-ImageIdentify.nb"}],
None,
NotificationFunction->All,
GeneratorOutputType->"CloudCDF",
DeliveryFunction->{{$WolframID}->"PDF",{"wrievents@wolfram.com"}->Automatic}
],CloudObject["testDir/ii-test"]
]

RunScheduledTask@@{CloudObject["testDir/ii-test"]}

DocumentGeneratorInformation[iiObj,"Log"]