Commanding the Wolfram Cloud

Todd Gayley
Wolfram

Amazon Echo

({{, Personal assistant,similar to Apple's Siri,but always on
      (shopping list, TODOs, alarms, etc.)

Good Bluetooth speaker

Links to your Amazon Prime account (music, playlists),
and other music services like Pandora

Connectivity to "Smart Home" devices like the Nest thermostat

Come with a Remote Control that you can use when in another room

Companion app for iOS/Android lets you configure the Echo,and see 
simple text output

Currently $179
}})

Amazon Echo

({{, But the most important thing...

Completely open developer model. You can create your
own "skills" as simple REST-style web services}})□

Wolfram Cloud Technologies in This Talk

InstantAPIs 
	CloudDeploy[APIFunction[...]]

Wolfram Development Platform
        Web interface to files, notebooks, APIs, etc.

Wolfram|Alpha API

Publish/Subscribe

Deployed Notebooks

Report Generation (Scheduled Tasks)

Cloud Mobile App

Creating Your Own Alexa Skill

Creating Your Own Alexa Skill

Creating Your Own Alexa Skill

Creating Your Own Alexa Skill

Creating Your Own Alexa Skill

Creating Your Own Alexa Skill

Defining the voice interface

Intent schema:

{
  "intents": [
    {
      "intent": "GetHoroscope",
      "slots": [
        {
          "name": "Sign",
          "type": "LIST_OF_SIGNS"
        },
        {
          "name": "Date",
          "type": "AMAZON.DATE"
        }
      ]
    },
    {
      "intent": "GetLuckyNumbers",
      "slots": []
    }
  ]
}

Sample utterances:

GetHoroscope what is the horoscope for {Sign}
GetHoroscope what will the horoscope for {Sign} be on {Date}
GetHoroscope get me my horoscope
GetHoroscope {Sign}

GetLuckyNumbers what are my lucky numbers
GetLuckyNumbers tell me my lucky numbers

Creating Your Own Alexa Skill

Format of JSON request
(what Amazon sends you)

{
  "version": "1.0",
  "session": {
    "new": false,
    "sessionId": "amzn1.echo-api.session.0000000-0000-0000-0000-00000000000",
    "application": {
      "applicationId": "amzn1.echo-sdk-ams.app.000000-d0ed-0000-ad00-000000d00ebe"
    },
    "user": {
      "userId": "amzn1.account.AM3B00000000000000000000000"
    }
  },
  "request": {
    "type": "IntentRequest",
    "requestId": " amzn1.echo-api.request.0000000-0000-0000-0000-00000000000",
    "timestamp": "2015-05-13T12:34:56Z",
    "intent": {
      "name": "GetHoroscope",
      "slots": {
        "Sign": {
          "name": "Sign",
          "value": "virgo"
        }
      }
    }
  }
}

ImportString["{
  \"request\": {
    \"type\": \"IntentRequest\",
    \"intent\": {
      \"name\": \"GetHoroscope\",
      \"slots\": {
        \"Sign\": {
          \"name\": \"Sign\",
          \"value\": \"virgo\"
        }
      }
    }
  }
}", "JSON"]

Creating Your Own Alexa Skill

Format of JSON response
(what you need to send back to Amazon)

{
  "version": "1.0",
  "sessionAttributes": {
    "supportedHoriscopePeriods": {
      "daily": true,
      "weekly": false,
      "monthly": false
    }
  },
  "response": {
    "outputSpeech": {
      "type": "PlainText",
      "text": "Today will provide you a new learning opportunity.  Stick with it and the possibilities will be endless. Can I help you with anything else?"
    },
    "card": {
      "type": "Simple",
      "title": "Horoscope",
      "content": "Today will provide you a new learning opportunity.  Stick with it and the possibilities will be endless."
    },
    "reprompt": {
      "outputSpeech": {
        "type": "PlainText",
        "text": "Can I help you with anything else?"
      }
    },
    "shouldEndSession": false
  }
}

InstantAPIs

The Wolfram Cloud makes it easy to deploy Wolfram Language code in such a way that it can be accessed as a standard REST-style API on the web.

InstantAPIs

The Wolfram Cloud makes it easy to deploy Wolfram Language code in such a way that it can be accessed as a standard REST-style API on the web.

Example: A function that adds two integers:

CloudDeploy[APIFunction[{"i"->"Integer", "j"->"Integer"}, #i + #j&], Permissions->"Public"]

InstantAPIs

The Wolfram Cloud makes it easy to deploy Wolfram Language code in such a way that it can be accessed as a standard REST-style API on the web.

Example: A function that adds two integers:

CloudDeploy[APIFunction[{"i"->"Integer", "j"->"Integer"}, #i + #j&], Permissions->"Public"]

One complication: The Web API call that Amazon sends does not have named arguments like a typical call, but rather just a block of JSON in the body of the HTTP request.

Solution:

APIFunction[{},
...
json = HTTPRequestData["Body"];
...
]

Create an Alexa Skill that Echoes User Speech

Intent schema and sample utterances:

{
  "intents": [
    {
      "intent": "EchoIntent",
      "slots": [
      	{
      		"name" : "Input",
      		"type" : "LITERAL"
      	}
      ]
    }
  ]
}


EchoIntent  {what|Input}
EchoIntent  {what is|Input}
EchoIntent  {what is the|Input}
EchoIntent  {what is the longest|Input}
EchoIntent  {what is the longest input|Input}
EchoIntent  {what is the longest input that|Input}
EchoIntent  {what is the longest input that this|Input}
EchoIntent  {what is the longest input that this service|Input}
EchoIntent  {what is the longest input that this service can|Input}
EchoIntent  {what is the longest input that this service can handle|Input}
EchoIntent  {what is the longest input that this service can reasonably handle|Input}

Create an Alexa Skill that Echoes User Speech

Deploying the APIFunction

Needs["Alexa`"]

This CloudObject URL was obtained previously by simply evaluating CloudObject[]:

echoInputEndpoint=CloudObject[MouseOver];

Here is the function that will handle the EchoIntent request:

echo[sess_, intentRequest_]:=AlexaResponse["outputSpeech"->GetSlot[intentRequest,"Input"]]

This deploys the appropriate APIFunction into the Wolfram Cloud:

DeployAlexaSkill[{"EchoIntent"->echo}, echoInputEndpoint]

The License Plate Game

Take a 3-letter sequence (from a license plate, for example) and try to come up with a word that has those three letters, in that order. 
The letters do not need to be consecutive in the word.
No proper names or hyphenated words.

For each sequence, there are 3 points to be had:
- one for the first word
- one for the shortest word
- one for a consecutive word

The implementation

{
  "intents": [
    {
      "intent": "NewRoundIntent",
      "slots": [ ]
    },
    {
      "intent": "AnswerIntent",
      "slots": [ ]
    },
    {
      "intent": "RepeatLettersIntent",
      "slots": [ ]
    }
  ]
}

NewRoundIntent  new round
NewRoundIntent  start new round
NewRoundIntent  next round
NewRoundIntent  start next round
NewRoundIntent  another round

AnswerIntent  say answer
AnswerIntent  say answers
AnswerIntent  answer
AnswerIntent  answers
AnswerIntent  get answer
AnswerIntent  get answers
AnswerIntent  what is the answer
AnswerIntent  give the answer
AnswerIntent  give me the answer
AnswerIntent  give us the answer
AnswerIntent  tell me the answer
AnswerIntent  tell us the answer
AnswerIntent  repeat answer
AnswerIntent  repeat the answer
AnswerIntent  repeat that answer
AnswerIntent  say answer again
AnswerIntent  say the answer again
AnswerIntent  say that answer again

RepeatLettersIntent  repeat letters
RepeatLettersIntent  repeat the letters
RepeatLettersIntent  repeat those letters
RepeatLettersIntent  say that again
RepeatLettersIntent  say those again

Needs["Alexa`"]
Needs["LicensePlateGame`"]

lpgEndpoint = "https://www.wolframcloud.com/objects/24177a43-38bd-4850-9cc5-33e3ae64a6b4";

newRound[sess_, intentRequest_]:=
Module[{userId, letters, letterStr},
userId=sess["user"]["userId"];
letters = LicensePlateGame`GenerateLetters[3];
letterStr = Riffle[ToUpperCase[letters], Table["! ",{Length[letters]}]];
LicensePlateGame`StoreUserRecord[userId, letters];
AlexaResponse["outputSpeech"->"The letters are, " <> letterStr]
]

repeatLetters[sess_, intentRequest_]:=
Module[{userId, letters, letterStr},
userId=sess["user"]["userId"];
letters = LicensePlateGame`GetUserRecord[userId];
letterStr = Riffle[ToUpperCase[letters], Table["! ",{Length[letters]}]];
AlexaResponse["outputSpeech"->"The letters are, " <> letterStr]
]

giveAnswer[sess_, intentRequest_]:=
Module[{userId, letters, words, shortest, consecutive, shortAndConsecutive},
userId=sess["user"]["userId"];
letters = LicensePlateGame`GetUserRecord[userId];
If[Head[letters]===Missing,
AlexaResponse["outputSpeech"->"I cannot find the record of your game"],
(* else *)
words = LicensePlateGame`GetWords[letters];
shortest = LicensePlateGame`FindShortest[words];
consecutive = LicensePlateGame`FindConsecutive[letters, words];
If[MissingQ[consecutive],
	AlexaResponse["outputSpeech"->"The shortest word is: " <>First[shortest] <> ". There are no consecutive words."],
(* else *)
	shortAndConsecutive=Intersection[shortest, consecutive];
	If[Length[shortAndConsecutive]>0,
	AlexaResponse["outputSpeech"->"The shortest word, which is also consecutive, is: " <>First[shortAndConsecutive]],
	(* else *)
	AlexaResponse["outputSpeech"->"The shortest word is: " <>First[shortest] <> ". A consecutive word is:" <> First[consecutive]]
]
]
]
]


DeployAlexaSkill[{"Launch" -> newRound, "NewRoundIntent"->newRound, "AnswerIntent"->giveAnswer, "RepeatLettersIntent"->repeatLetters}, lpgEndpoint];

Wolfram|Alpha Spoken Answers
Why should Siri have all the fun?

Wolfram|Alpha is capable of providing spoken-result answers to many types of queries.

Wolfram|Alpha Spoken Answers
Why should Siri have all the fun?

Wolfram|Alpha is capable of providing spoken-result answers to many types of queries.

This feature is available via the Wolfram|Alpha API (although it is an extra-cost feature).

Wolfram|Alpha Spoken Answers
Why should Siri have all the fun?

Wolfram|Alpha is capable of providing spoken-result answers to many types of queries.

This feature is available via the Wolfram|Alpha API (although it is an extra-cost feature).

URLExecute["http://api.wolframalpha.com/v2/query", {"podindex" ->"1", "appid"->$appid,"spokenresult"->True, "input" ->"2+2"}, "XML"]

Output:

XMLObject["Document"][{XMLObject["Declaration"]["Version"->"1.0","Encoding"->"UTF-8"]},XMLElement["queryresult",{"success"->"true","error"->"false","numpods"->"1","datatypes"->"Math","timedout"->"","timedoutpods"->"","timing"->"0.613","parsetiming"->"0.113","parsetimedout"->"false","recalculate"->"","id"->"MSPa2052083geh3f1ib3e22000018b3e8h61772cd0h","host"->"http://www4c.wolframalpha.com","server"->"36","version"->"2.6"},{XMLElement["pod",{"title"->"Input","scanner"->"Identity","id"->"Input","position"->"100","error"->"false","numsubpods"->"1"},{XMLElement["subpod",{"title"->""},{XMLElement["plaintext",{},{"2+2"}],XMLElement["img",{"src"->"http://www4c.wolframalpha.com/Calculate/MSP/MSP2072083geh3f1ib3e22000016ib5875ed3bif5a?MSPStoreType=image/gif&s=36","alt"->"2+2","title"->"2+2","width"->"32","height"->"18"},{}]}]}],XMLElement["spokenresult",{"generictemplate"->"TheResultIs","resulttype"->"Number","inputtype"->"SimpleMath","sampletext"->"The answer is 4"},{XMLElement["result",{"value"->"4"},{}]}]}],{}]

Wolfram|Alpha Spoken Answers
Why should Siri have all the fun?

The implementation

{
  "intents": [
    {
      "intent": "QueryIntent",
      "slots": [
        {
          "name": "Input",
          "type": "LITERAL"
        }
      ]
    }
  ]
}

QueryIntent  {one|Input}
QueryIntent  {one two|Input}
QueryIntent  {one two three|Input}
QueryIntent  {one two three four|Input}
QueryIntent  {one two three four five|Input}
QueryIntent  {one two three four five six|Input}
QueryIntent  {one two three four five six seven|Input}
QueryIntent  {one two three four five six seven eight|Input}
QueryIntent  {one two three four five six seven eight nine|Input}
QueryIntent  {one two three four five six seven eight nine ten|Input}
QueryIntent  {one two three four five six seven eight nine ten eleven|Input}
QueryIntent  {one two three four five six seven eight nine ten eleven twelve|Input}
QueryIntent  {one two three four five six seven eight nine ten eleven twelve thirteen|Input}
QueryIntent  {one two three four five six seven eight nine ten eleven twelve thirteen fourteen|Input}

Needs["Alexa`"]

doQuery[sess_, intentRequest_] := 
Module[{xml, spokenResultElement,spokenText},
xml=URLExecute["http://api.wolframalpha.com/v2/query", {"podindex" ->"1", "appid"->$appid,"spokenresult"->True, "input" ->GetSlot[intentRequest,"Input"]}, "XML"];
spokenResultElement=FirstCase[xml, XMLElement["spokenresult", __], Null,Infinity];
If[Head[spokenResultElement]===XMLElement,
spokenText = FirstCase[spokenResultElement, ("sampletext"->text_) :> text, Null, Infinity]
];
If[!StringQ[spokenText],
spokenText="Sorry, Wolfram Alpha could not determine a spoken answer"
];
AlexaResponse["outputSpeech"->spokenText]
]

DeployAlexaSkill[{"QueryIntent"->doQuery}, CloudObject["https://www.wolframcloud.com/objects/74c46dd4-bd9c-4e32-8d5b-a174d05a9e57"]]

Wolfram|Alpha Visual Output

I want to use voice input for queries, but get results in visual form

This requires the Cloud to “push” things to my computer

Wolfram|Alpha Visual Output

I want to use voice input for queries, but get results in visual form

This requires the Cloud to “push” things to my computer

Publish/Subscribe (“Channels”)

This package is not yet part of the standard Mathematica distribution:

Needs["HTTPListen`"]

ch=CreateChannel["Alexa/wtc2015",ChannelReceiver->rcv]

ChannelListen["Alexa/wtc2015"]

rcv[args___] := Print["rcv: ", {args}]

ChannelSend[ch, 42]

Clear[rcv]

rcv[msg_Association]:=
Module[{nb},
nb=CreateNotebook[WindowSize->{Full,Full}];
NotebookWrite[nb, "WolframAlpha[\"" <> msg["RawData"] <> "\"]", All];
SelectionEvaluate[nb]
]

Wolfram|Alpha Visual Output

(The hackery with Symbol[] here is needed because the Wolfram Cloud is currently running version 10.2 of the Wolfram Engine, not 10.3.)

sendToChannel[expr_] :=
Symbol["System`ChannelSend"][Symbol["System`ChannelObject"]["Alexa/wtc2015"],expr]

doQuery[sess_, intentRequest_] := 
Module[{xml, spokenResultElement,spokenText},
xml=URLExecute["http://api.wolframalpha.com/v2/query", {"podindex" ->"1", "appid"->$appid,"spokenresult"->True, "input" ->GetSlot[intentRequest,"Input"]}, "XML"];
spokenResultElement=FirstCase[xml, XMLElement["spokenresult", __], Null,Infinity];
If[Head[spokenResultElement]===XMLElement,
spokenText = FirstCase[spokenResultElement, ("sampletext"->text_) :> text, Null, Infinity]
];
If[!StringQ[spokenText],
spokenText="Sorry, Wolfram Alpha could not determine a spoken answer"
];
Get["HTTPListen`"];
sendToChannel[GetSlot[intentRequest,"Input"]];
AlexaResponse["outputSpeech"->spokenText]
]

DeployAlexaSkill[{"QueryIntent"->doQuery},
CloudObject["https://www.wolframcloud.com/objects/74c46dd4-bd9c-4e32-8d5b-a174d05a9e57"]]

TimeTracker App

Goal: Use Alexa in my office to help me keep track of time spent on various work projects

TimeTracker App

Voice Interface

“Tell TimeTracker to start project XYZ”

“Tell TimeTracker I’m switching to project XYZ”

“Tell TimeTracker I’m taking a break”

“Ask TimeTracker to list my projects”

“Ask TimeTracker how much time I have spent on project XYZ in the past week”

TimeTracker App

Visual Output on the Web

TimeTracker App

Visual Output on the Web

Report Generation (Scheduled Tasks)

At the end of every operation that changes the TimeTracker data stored in the cloud (like starting or stopping work on a project), the app schedules a task to run that generates one or more report pages. This also could be done on an hourly or daily basis, if desired.

This report generation might be time-consuming, so we do it as a scheduled task so that it will not cause a delay in the voice interface, or when the reports are viewed.

Conclusion

The Alexa package I developed for this talk will soon be available for public download. Run the following command to get it. If this complains about no such paclet being available, then it isn’t ready yet. Documentation will be included.

PacletUpdate["Alexa"]

This presentation notebook will be available from the conference website.