Transform Diffing

A diffing transform computes the difference between consecutive data points.

Quantile properties

Property Type Description
diffing string[] Fields to compute diffings. Because this transformation reshape the data, provide all the fields to compute diffing together.
carryOver boolean (Optional, default: true, only for streaming specs) Whether to get the last value from the lastest instance in history.
keepFirstAsZero boolean (Optional, default: true) If add a zero value before the first datum. For streaming specs, if carryOver is set to true and there are historical data points, then this is ignored.
as string[] (Optional) New field names. The length must match that of diffing.
groupby Array[String] Nominal fields to group the diffed values by.

Behavior

Suppose we have a dataset with three fields: category, speed, and distance, and we compute diffing for speed and distance.

category speed distance
car 10 300
car 11 290
bird 1 12
bird 1.1 13

If groupby is not set, then it will result in:

category speed_diff distance_diff
car 10 300
car 1 -10
bird -10 -278
bird 0.1 1

If groupby is set to ['category], then it will result in:

category speed_diff distance_diff
car 10 300
car 1 -10
bird 1 12
bird 0.1 1

As shown above, ‘next’ values are preserved for the non-diffed fields.

If keepFirstAsZero is set to false (with groupby), it will reslut in:

category speed_diff distance_diff
car 1 -10
bird 0.1 1

Usage pattern

JSON

{
  ...
  "transform" : [
    {
      "diffing": ["speed", "distance"],
      "groupby" ["category"],
      "carryOver": true
    },
  ]
  ...
}

JavaScript

let stream = new Erie.Stream();
...
let diffing = new Erie.Diffing(["speed", "distance"], true, true, ["speed_diff", "distance_diff"]) // fields, carryOver, keepFirstAsZero, new field names
diffing.groupby(['category']);
stream.transform.add(diffing);
...
© Hyeok Kim