Time Series Processing

Gosia Konwerska
Wolfram Research, Inc.

Join the Conversation #WolframTechConf

TimeSeries, EventSeries, and TemporalData

TimeSeries - designed to store single path continuous-valued time series

TimeSeries as a default interpolates the values between given times stamps.

EventSeries - designed to store single path discrete-valued  time series

EventSeries as a default does not interpolate the values between given times stamps.

TemporalData - general object to store any number of paths of either data type

Examples

Create.

ts=TimeSeries[{2,4,6},Automatic]

es=EventSeries[{2,4,6},Automatic]

td=TemporalData[{2,4,6},Automatic]

TimeSeries and EventSeries at given time stamp return a value while TemporalData at a given time returns a slice distribution.

ts[1]

es[1]

td[1]

TimeSeries and TemporalData interpolate between time stamps while EventSeries does not.

ts[1/2]

es[1/2]

td[1/2]

TemporalData returning DataDistribution really makes sense for multipath data.

data=RandomFunction[WienerProcess[],{0,1,.01},10^4]

Show[Histogram[data["SliceData",.5],40,"CDF"],Plot[CDF[data[.5],x],{x,-3,3}]]

Properties

ts["Properties"]

es["Properties"]

The one property of TimeSeries not defined for EventSeries is “PathFunction”.

ts["PathFunction"]

The path function defines how the values between time stamps are being interpolated. It can be changed via an option.

ts1=TimeSeries[{1,2,3},Automatic,ResamplingMethod->{"Interpolation",InterpolationOrder->2}]

ts2=TimeSeries[{1,2,3},Automatic,ResamplingMethod->{"Constant",0}]

ts1["PathFunction"]

ts2["PathFunction"]

Properties of TemporalData include plural versions for multipath objects.

td["Properties"]

Arithmetic

TimeSeries and EventSeries being single path object allow to define certain arithmetic operations directly.

Addition of TimeSeries

The case of identical time stamps.

ts1=TimeSeries[{2,4,6,8},Automatic]

ts2=TimeSeries[{1,3,5,7},Automatic]

sum12=ts1+ts2

ListLinePlot[{Labeled[ts1,"ts1"],Labeled[ts2,"ts2"],Labeled[sum12,"sum"]}]

What if time stamps are different?

ts3=TimeSeries[{1,3,5,7},{{0,1/2,1,3/2}}]

sum13=ts1+ts3

sum13["Path"]

ListLinePlot[{Labeled[ts1,"ts1"],Labeled[ts3,"ts3"],Labeled[sum13,"sum"]}]

The values of ts1 has been interpolated on the new time stamps from ts3 but within the intersection of the underlying time intervals.

times=ts3["Times"]

ts1[times]

%+ts3["Values"]

The precision of the new time stamps naturally defines the precision of the new values.

ts1[{1/2,.5,.5`20}]

Addition of EventSeries

The identical time stamps are required for addition to be defined.

es1=EventSeries[{2,4,6,8},Automatic];
es2=EventSeries[{1,3,5,7},Automatic];

es1+es2

Try to add EventSeries with different time stamps.

es3=EventSeries[{1,2,3,4},{{1,1.2,2,2.3}}];

es1+es3

When addition of EventSeries to TimeSeries is defined (needs time stamps agreement) the result will be coerced to TimeSeries.

ts1=TimeSeries[{2,4,6,8},Automatic];
es1=EventSeries[{2,4,6,8},Automatic];

ts1+es1

Other operations

ts=TimeSeries[{2,4,6,8},Automatic];

Operations on values.

Min[ts]

Creating new TimeSeries object.

Mod[ts,3]

%["Path"]

Shifting whole TimeSeries.

ts+3

%["Path"]

Double Life of Time Stamps

So far we looked at a simple time series with integer time stamps.

ts=TimeSeries[{2,4,6,8},Automatic]

ts["Times"]

The time stamps in general might be thought as cardinals or ordinals, ordering the observations. Time stamps not only order the values but also specify the distance between the observations.

TimeSeries with Dates

TimeSeries@FinancialData["AAPL","Close",{{2015,5,1},{2015,7,31}}]

data=TimeSeries[];

The function MinimumTimeIncrement returns the smallest distance between any two time stamps.

UnitConvert[Quantity[MinimumTimeIncrement[data],"Seconds"],"Days"]

However, the financial data is given on business days, hence sometimes the distance between time stamps is greater than a day. We can check if all the distances between consecutive time stamps are identical.

RegularlySampledQ[data]

Calculate the differences between time stamps:

MapThread[DateDifference[#1,#2]&,{Most@data["Dates"],Rest@data["Dates"]}]

The data is uniformly sampled on business days. The regularity of time stamps is the underlying assumption for discrete time time series models like ARProcess. The data in our example can be understood as regularly sampled via resampling or by setting option TemporalRegularity to True.

data1=TimeSeriesResample[data,"BusinessDay"]

data2=TimeSeries[data,TemporalRegularity->True]

Check the regularity of time stamps.

{RegularlySampledQ[data1],RegularlySampledQ[data2]}

The new paths are identical to the original one.

{SameQ[data1["Path"],data["Path"]],SameQ[data2["Path"],data["Path"]]}

Time Series Processes

Time series process defined in Mathematica are discrete time processes and are defined in terms of time unit increments. The outcome of simulation is a path indexed by numbers.

RandomFunction[ARProcess[{.3},1],{10}]

%["Path"]

Interpreting time parameter as index allows to compute time slices for non weakly stationary models.

proc=ARProcess[{1.2},1,{x0}];(* x0 is the initial value at time -1 *)

Table[Mean[proc[t]],{t,0,3}]

For that reason any estimating function like TimeSeriesModelFit or EstimatedProcess will either resolve the data to be regularly sampled or fail.

RegularlySampledQ[data]

TimeSeriesModelFit[data]

%["TemporalData"]

Compare to the original data.

data

EstimatedProcess[data,ARMAProcess[1,1]]

The solution is to pretreat the data so it is regularly sampled in a sensible way.

RegularlySampledQ[data1]

TimeSeriesModelFit[data1]

%//Normal

EstimatedProcess[data1,ARMAProcess[1,1],ProcessEstimator->"MaximumLikelihood"]

Modifying Values & Times

Most of the time series processing function have “TimeSeries” in the name:

TimeSeriesAggregate

TimeSeriesInsert

TimeSeriesMap

TimeSeriesMapThread

TimeSeriesResample

TimeSeriesRescale

TimeSeriesShift

TimeSeriesThread

TimeSeriesWindow

MovingMap

Laundry List Part One

We have already seen TimeSeriesResample.

ts=TimeSeries[{1.2,0.3,4.7},{{2,4,6}}]

TimeSeriesResample[ts,3]

%//Normal

TimeSeriesShift will move time stamps.

ts=TimeSeries[{1.2,0.3,4.7},{{2,4,6}}]

TimeSeriesShift[ts,1]

%//Normal

Extract part of a time series with TimeSeriesWindow.

ts=TimeSeries[{1.2,0.3,4.7},{{2,4,6}}]

TimeSeriesWindow[ts,{3,6}]

%//Normal

Insert new value or intertwine two time series with TimeSeriesInsert.

ts=TimeSeries[{1.2,0.3,4.7},{{2,4,6}}]

TimeSeriesInsert[ts,{1,2.8}]

%//Normal

TimeSeriesInsert[ts,TimeSeries[{4,4,4},{{1,3,5}}]]

%//Normal

TimeSeriesRescale is another operation on time stamps.

ts=TimeSeries[{1.2,0.3,4.7},{{2,4,6}}]

TimeSeriesRescale[ts,{Today,DatePlus[Today,Quantity[3,"Days"]]}]

%//Normal

Laundry List Part Two

TimeSeriesMap maps given function over the values.

ts=TimeSeries[{Array[a,{6,2}]},Automatic]

TimeSeriesMap[Max,ts]

%//Normal

The Max applied directly to the time series finds the maximum of all the inputs in values.

Max[ts]

TimeSeriesThread will combine two time series.

ts1=TimeSeries[{Array[a,{6,2}]},Automatic];
ts2=TimeSeries[{Array[b,{6,2}]},Automatic];

TimeSeriesThread[Total,{ts1,ts2}]

%//Normal

TimeSeriesMapThread maps over values and time stamps.

ts=TimeSeries[{Array[a,{6,2}]},Automatic];

TimeSeriesMapThread[#2^#1&,ts] (* #1 refers to times, #2 referes to values *)

%//Normal

TimeSeriesAggregate applies given function over disjoint windows of given length.

ts=TimeSeries[{Array[a,7]},Automatic];

TimeSeriesAggregate[ts,2,f]

%//Normal

data=TimeSeries@FinancialData["SP500",{2015}];

DateListPlot[data]

mdata=TimeSeriesAggregate[data,"Month",Mean]

DateListPlot[mdata]

mdata["DatePath"]

Moving Map 2.0

MovingMap underwent extensive revision in Mathematica 10.2.

?MovingMap

window length vs number of elements

Window length 2:

MovingMap[f,{(Subscript[x, 1]),(Subscript[x, 2]),(Subscript[x, 3]),(Subscript[x, 4]),(Subscript[x, 5])},2]

Number of events 3:

MovingMap[f,{(Subscript[x, 1]),(Subscript[x, 2]),(Subscript[x, 3]),(Subscript[x, 4]),(Subscript[x, 5])},Quantity[3,"Events"]]

Now for irrregularly spaced data:

ts=TimeSeries[{(Subscript[x, 1]),(Subscript[x, 2]),(Subscript[x, 3]),(Subscript[x, 4]),(Subscript[x, 5])}, {{0,3,4,6,9}}];

Window length 2 with relative window position on the Right by default:

MovingMap[f,ts,2]//Normal

All possibilities - note varied number of arguments in f:

Table[MovingMap[f,ts,{2,pos}]//Normal,{pos,{Left,Center,Right}}]//Column

Now with given number of events - constant number of arguments to f:

Table[MovingMap[f,ts,{Quantity[3,"Events"],pos}]//Normal,{pos,{Left,Center,Right}}]//Column

the arguments to the function

ts=TimeSeries[{(Subscript[x, 1]),(Subscript[x, 2]),(Subscript[x, 3]),(Subscript[x, 4]),(Subscript[x, 5])}, {{0,3,4,6,9}}];

Use values only:

MovingMap[f,ts,2]//Normal

MovingMap[f[#Values]&,ts,2]//Normal

Use times only:

MovingMap[f[#Times]&,ts,2]//Normal

Use boundary times - some of them are NOT time stamps in original ts:

MovingMap[f[#BoundaryTimes]&,ts,2]//Normal

Use boundary values - note interpolation used when a boundary is not an element of ts:

MovingMap[f[#BoundaryValues]&,ts,2]//Normal

freedom of window placement

There is no restriction to place the windows on existing time stamps.

ts=TimeSeries[{(Subscript[x, 1]),(Subscript[x, 2]),(Subscript[x, 3]),(Subscript[x, 4]),(Subscript[x, 5])},{{0,2,4,6,8}}];

ts["Path"]

MovingMap[f,ts,{2,Right,{{1,3,5,7,9}}}]//Normal

Give range for time stamps

MovingMap[f,ts,{Quantity[2,"Events"],Left,{1,8}}]//Normal

MovingMap[f,ts,{Quantity[2,"Events"],Left,{1,7}}]//Normal

MovingMap[f,ts,{Quantity[2,"Events"],Right,{1,7}}]//Normal

For dated time series

ts=TimeSeries@FinancialData["SBUX",{{2015,8,1},{2015,9,9}}]

res1=MovingMap[Mean,ts,{5,Right}]

res2=MovingMap[Mean,ts,{Quantity[5,"Days"],Right}]

res1===res2

Now let’s place windows on Saturdays.

DayName/@ts["Dates"]

Tally[%]

dates=Table[DayPlus[ts["FirstDate"],k,Saturday],{k,1,5}]

res3=MovingMap[Mean,ts,{5,Right,{dates}}]

DateListPlot[{ts,res1, res3},PlotLegends->{"data", "res1","res3"}]

res3["DatePath"]

Compare to TimeSeriesAggregate since our windows were nonoverlapping.

res=TimeSeriesAggregate[ts,{Quantity[7,"Days"],Right}]

res["DatePath"]

DateListPlot[{res3,res},PlotLegends->{"moving map", "time series aggregate"}]

So MovingMap can do the job of TimeSeriesAggregates in a more precise way.

specifying padding for values and times

ts=TimeSeries[{(Subscript[x, 1]),(Subscript[x, 2]),(Subscript[x, 3]),(Subscript[x, 4]),(Subscript[x, 5])},{{0,2,4,6,8}}];

ts["Path"]

Padding makes sense when window size is specified, not explicit number of events.

MovingMap[f,ts,{Quantity[2,"Events"],Right},None]//Normal

None padding - take all the windows

MovingMap[f,ts,{3,Right},None]//Normal

Automatic padding - take only non-overhanging windows

MovingMap[f,ts,{3,Right},Automatic]//Normal

Padding for times and values - to see this make function take values and times as arguments.

MovingMap[f[#1,#2]&,ts,{3,Right},{Automatic,p}]//Normal

Filtering

Simplifying the filters that work on TimeSeries can be split into two categories - the ones that refer to the relative neighborhood or the ones that refere to the absolute number of values. Here are examples from each category.

?MeanFilter

?GaussianFilter

ts=TimeSeries[];

ts["Times"]

res1=MeanFilter[ts,1]

res2=GaussianFilter[ts,10]

ListLinePlot[{ts,res1,res2},PlotLegends->{"ts","MeanFilter","GaussianFilter"}]

Filters also work for irregularly sampled TimeSeries.

vals=Accumulate[RandomVariate[NormalDistribution[0,0.1],101]];
times=FoldList[Plus,0.,RandomVariate[ExponentialDistribution[200],100]];

ts=TimeSeries[vals,{times}]

RegularlySampledQ[ts]

MinimumTimeIncrement[ts]

MeanFilter[ts,.2]

The filters also understand Quantity input.

MeanFilter[ts,Quantity[.2,"Seconds"]]===MeanFilter[ts,.2]

ListLinePlot[{ts,MeanFilter[ts,.2]},PlotLegends->{"ts","MeanFilter"}]

Filters also work on TimeSeries with dates.

dts=TimeSeries[];

RegularlySampledQ[dts]

MedianFilter[dts,Quantity[5,"Days"]]

DateListPlot[{dts,MedianFilter[dts,Quantity[5,"Days"]]},PlotLegends->{"dts","MedianFilter"}]

Thank you!

gosiak@wolfram.com

Join the Conversation #WolframTechConf

Init

$PlotTheme={"Presentation","Wide"};