The encoding property of a steam defines how to map data variables to auditory variables. While Erie uses many Vega-Lite conventions, each encoding channel may have different properties and usage patterns. This is because data sonification is defined on a sound space. For example, a time channel plays two roles: an encoding channel and a sonification canvas at the same time.

This documentation describes overall structure and common properties of encoding channels.

Supported encoding channels

For more details about the modulation, please refer to frequency modulation and amplitude modulation (or harmonicity).

Common encoding channel properties

Inline aggregate

Possible aggregate values

• count: the count of elements
• valid: the number of valid items in a variable
• distinct: the number of distinct items in a variable
• mean/average: the mean of a variable
• mode: the mode of a variable
• median: the median of a variable
• quantile: the p quantile value of a variable
• stdev: the standard deviation of a variable
• stdevp: the population standard deviation of a variable
• variance: the variance of a variable
• variancep: the population variance of a variable
• sum: the sum of a variable
• product: the product (multiplication) of a variable
• max: the maximum value of a variable
• min: the min value of a variable

Note: double-field aggregates (corr, covariance, and covariancep) are not available for inline aggregate.

See the transform documentation for further details.

Bin

The simplest way to use in-channel binning is setting the bin property as true. To provide more details, one can use a bin Object with details.

Scale

The scale propoerty describes further details about the mapping between the specified data variable and the targeted auditory channel. A scale object can have the following properties. Each encoding channel may have different scale properties, so refer to relevant documentations for details.

API usage

{
  ...
  "encoding" : {
    "[channel]": {
      "field": "FieldName",
      "type": "DataType",
      "aggregate": "AggregateMethod", // only for a `quantitative` channel
      "bin": {
        "nice": true // or false,
        "maxbins": 10,
        "step": 5, // Number
        "exact": false // or true
      }, // or true or false
      "scale": {
        "domain": [ ... ],
        "range": [ ... ],
        "polarity": "positive" // or "negative",
        "maxDistinct": true // or false,
        "times": ..., // Number,
        "zero": false // or true,
        "description": "nonskip" // or "skip",
        // only for `time` channel
        "length": ..., // Seconds,
        "timing": "absolute", // or "relative"
        // only for `time`, `tapSpeed`, and `tapCount` channels
        "band": ..., // Seconds,
      },
      "value": ...,
      "condition": [
        {"test": [ value1, value2, ... ], "value": ... }, // as array
        {"test": {"not": [ value1, value2, ... ]}, "value": ... }, // "not"
        {"test": "d.field > 0", "value": ... } // conditional
      ],
      "speech": true // or false,
      "format": '.2', // (rounding at the second decimal point), no default value
      "formatType": "number" // default value
    }
  }
  ...
}

let stream = new Erie.Stream();

// method 1 (more reusable)
let channel = new Erie.{ChannelName}();
channel.field("Origin", "nominal");
// or let channel = new Erie.Channel("Origin", "nominal");
let maxbins = 10, nice = true, step;
channel.bin(maxbins, step, nice);
// channel.bin(true);
// channel.bin(false);
// let exact = [0, 10, 15, 20];
// channel.bin(exact);
channel.aggregate("mean");
channel.scale("timing", "relative"); // key & value
channel.scale("domain", [0, 10]);
channel.scale("timing", "relative");

// static
channel.value(...);
channel.condition.add([ value1, value2, ... ], ...);
channel.condition.add({"not": [ value1, value2, ... ]}, ...);
channel.condition.add("d.field > 0", ...);

// format
channel.format('.2', 'number');
channel.format('.2');
channel.formatType('number');

stream.encoding.{channelName}.set(channel); // provided for reusability;

// method 2 (more direct)
stream.encoding.{channelName}.field("Origin", "nominal");
stream.encoding.{channelName}.scale("timing", "relative");

// method 1 + 2
stream.encoding.{channelName} = new Erie.Channel("Origin", "nominal");
stream.encoding.{channelName}.scale("timing", "relative");

Scale description markup

For the description property of a scale, you can provide a marked-up expression to be read out before playing the sonification or be separately played.

How to mark up a description text

There are three four types of elements in a description text.

1. Plain text → Just give that text
2. A discrete audio legend → <sound value="X" duration="D">

• X: a “data” value or reserved keywoard to be mapped using the scale.
• D: the duration of this sound (default: 0.5 seconds or 1 beat). If config.timeUnit is beat then the unit is beat counts, and when it is second, then the unit is seconds.

1. A continuous audio legend → <sound v0="X1" v1="X2" duration="D">

• vn: Xn (n = 0, 1, 2, …): breakpoints, the “data” values or reserved keywoards to be mapped using the scale! Provide as many point as you need. They will be timed relatively.
• D: the duration of this sound (default: 0.5 seconds or 1 beat multipled by the number of breakpoints). If config.timeUnit is beat then the unit is beat counts, and when it is second, then the unit is seconds.

1. A datum list → <list item="A" first="F" last="L" join="S" and="N">

• A: an array/list of items or a keyword to be listed. If not provided, all the domain values will be sounded. They should be comma separated. For example, <list item="apple,pear, juice" ... > (O) but <list item="[apple,pear, 'juice']" ... > (may work, but may cause error). Spaces before and after each item is trimmed.
• F: the first F items to read out
• L: the last F items to read out

Note: <list item="apple,pear,juice,melon,grape" first="2" last="2" join=", "> will result in ‘apple, pear, melon, grape’. See the examples for how to use.

• S: a string between each consecutive pair of items. This considers a space (for iternationalization). The default is ,
• N: a string between the last two consecutive items. This considers a space (for iternationalization). There is no default value provided.

Note: For the time channel, while the rest can be “compiled,” it won’t really play the audio.

Reserved keywords

To use in a plan text, use them with < and > (e.g., <domain.min>)

• domain.min: the minimum domain value
• domain.max: the maximum domain value
• domain.length: the number of domain values specified
• domain[n]: the n-th domain value
• domain: the entire domain
• channel: the channel name
• field: the channel’s field
• aggregate: the channel’s aggregate method
• title: the channel’s display name (if not provided, it uses field).

Examples

Case 1: Consider a pitch channel that maps [0, 50] (data, field: 'weight') to [300, 500] (Hz). The description text is

The <channel> channel represents <title>. The minimum value <domain.min> is <sound value="domain.min"> and <domain.max> is <sound value="domain.max">. This is compiled to The pitch channel represents weight. The minimum value 0 is (sound with pitch 300) and 50 is (sound with pitch 500).

Case 2: Consider a pan channel that maps [100, 300] (data, field: 'height') to [-1, 1]. The description text is

The <channel> channel is for <title>. The minimum value of <domain.min> and the maximum value of <domain.max> are mapped to <sound v0="domain.min" v1="domain.max" duration="0.3">.

This is compiled to The pan channel is for height. The minimum value 100 and the maximum value of 300 are mapped to (pan-sweep from -1 to 1 for 0.6 seconds).

Case 3: Consider a time channel that maps [apple,pear,kiwi,melon,grape,mango,orange] (data, field: 'type') to relative timing. The description text is

The <field> has <domain.unique> itmes. They are played from <list first="2" join=" and "> to <list last="2" join=" and ">.

This is compiled to The type has 7 items. They are played from apple and pear to mango and orange.

How to play a description

1. Play it before a sonification

Quick answer: Nothing to be done.

1. Play it separately

In config, first set skipScaleSpeech as true.

{
  ...
  "config": {
    "skipScaleSpeech": true
  }
}

let spec = new Erie.Stream();
...
spec.config('skipScaleSpeec', true);

After you compile the sonification spec, run the playDescription method.

let spec = new Erie.Stream();
...
let audio;
Erie.compileAuidoGraph(spec)
  .then((audio_graph) => {
    let audio = audio_graph;
    ...
    audio.playScaleDescription();
    audio.playScaleDescription('time'); // play only for the time channel
    audio.stopScaleDescription(); // to stop whatever being played
  });
...

If scale.description is set to 'skip' or null, then it is also not played with the playScaleDescription method.

Scale consistency for multi-stream sonifications

It is possible (and default) to use a consistent scales unless otherwise specified.

For example, suppose a sequence consisting of one single stream and one three-stream overlay, so there are four streams defined. Let’s name them ‘Stream-A’, ‘Overlay-1’, ‘Overlay-2’, and ‘Overlay-3’, respectively. ‘Overlay-1’, ‘Overlay-2’, and ‘Overlay-3’ comprise ‘Stream-B’ The specification will look like: {"sequence": [ {... Stream-A ...}, { "overlay": [Overlay-1, Overaly-2, Overlay-3]}]}.

Then, assume these streams use a pitch channel.

Case 1: Global

‘Stream-A’ defines a pitch channel, while the others do not. Then, ‘Overlay-1’, ‘Overlay-2’, and ‘Overlay-3’ use the same pitch scale as defined in ‘Stream-A’ as long as they rely on the same dataset and field, and are of the same encoding type (e.g., nominal, quantitative). If the overlaid streams use a different dataset or of a different encoding type, then the scale will be different.

Case 2: Nested

‘Stream-A’ and ‘Overlay-1’ define a pitch channel, while the others do not. Then, ‘Overlay-2’, and ‘Overlay-3’ use the same pitch scale as defined in ‘Overlay-1’ as long as they rely on the same dataset and field, and are of the same encoding type (e.g., nominal, quantitative). If ‘Overlay-2’, and ‘Overlay-3’ use a different dataset, field, or encoding type, then the scale will be different.

Case 3: I just need individually defined scales

If it is necessary to use individually defined scales, then set overlayScaleConsistency (for overlay) or sequenceScaleConsistency (for sequence) to false, in the closest config object. To make inconsistent scales between ‘Stream-A’ and ‘Stream-B’, then set sequenceScaleConsistency as false in the top-level sequence. To make inconsistent scales among ‘Overlay-1’, ‘Overlay-2’, and ‘Overlay-3’, then set overlayScaleConsistency as false in the overlay object.

When two scale definitions conflicts, a property defined earliest is applied.

This can also done by channels, for example { "sequenceScaleConsistency": { "pitch" : false } }.