Explanation – Flow Representation
←Explanation - Flow · Index↑ · Explanation - Timing→
Please ensure you have read the Summary and Definitions before considering the further detail covered by this page.
Understanding the Flow Representation Entity
Setting the level of a Flow for a scenario defines the interface that will be used for the data – it defines the data type to be used when handling the data (that is, how the data is encoded and how to interpret it). This interface might be implemented in different ways: for example, using different containers or transports for the interchange of the data. These implementations are known as Flow Representations and can take different approaches to managing these practicalities. For example:
- The details of how data is formatted might be governed by a particular API, transport or container.
- The formatting might be explicitly signalled in-band using technical metadata properties.
- An implementation (such as an API) might provide a means for the client to request that the data be formatted in a particular way.
A Flow might be requested from a system using a Flow ID. The expectation is often that a single Flow Representation will be delivered throughout the resulting communication / transportation session even though there might be other Flow Representations associated with the same Flow.
In general, the Flow Representations associated with a Flow will not be identical. Some of the differences might be signalled by technical metadata whereas others might not. Note that no Flow Representation is considered to be canonical.
Notes on the Formal Definition of Flow Representation
For a Flow, any differences in the data communicated by its Flow Representations must be considered negligible for the intended purpose. For example, in a media scenario the Flow Representations must all represent baseband signals that are considered “the same” for practical purposes such that they all result in “the same” presentation being perceived (for example, the differences in video pictures are visually imperceptible).
This constraint allows Flow Representations with very small differences in baseband signal to be members of the same Flow. For example, if a lossy video compression process with a fixed configuration processes a video twice then the Flow Representations produced might not decode to identical baseband video signals for various reasons (such as random processes used in the video encoder). However, as long as these differences are considered negligible (they will not be visually noticeable or affect further graphics processing) then these Flow Representations can still be members of the same Flow. An alternative way to view this is that each Data Object might have a range of values associated with it: values in this range are all regarded as having a variation in quality / accuracy that is negligible for the scenario (in other words, the range of values: results in practically the same presentation; represents practically the same baseband signal).
The Flow Representations belonging to a Flow are constrained to all use the same set of Time Values (because these Time Values are defined by the Flow). However, these Flow Representations are not required to all populate the same set of Time Values with data. For example, one Flow Representation might contain data for a one minute portion of the Flow whereas another Flow Representation might contain data for a three minute portion of the Flow.
A Flow Representation is not given an ID in the model because identifying (in an interchangeable or persistent way) individual Flow Representations is generally not considered to be an important requirement.
Examples of Flow Representations
Example 1: Mono PCM Audio in Different Packaging Formats
In this example, we use a Flow ID of X to represent this audio encoded as PCM. In this scenario the Flow is set at the level of the PCM-coded audio samples (these are endianness-independent).
The same Flow ID is used across a variety of packaging / delivery mechanisms (different Flow Representations).
RTP:
- PCM in RTP is defined to be big endian
WAV:
- PCM in WAV is defined to be little endian
Store:
- Deliver the PCM audio along with signalling of the endianness so that endianness can be signalled upon retrieval
HTTP API:
- Deliver the PCM audio over the API along with signalling of the endianness. Alternatively, define a fixed endianness for the API (and specify it in the API documentation).
C++ function:
- Returns samples as integers; the endianness of the in-memory representation is transparent to the user.
Flow technical metadata would include properties such as the audio codec (PCM), sampling rate, bit-depth, but not the endianness.
Example 2: Uncompressed Video
In this example, we use a Flow ID of X to represent this uncompressed HD video. In this scenario the Flow is set at the level of the uncompressed video frames (these are byte-packing independent). Note: in this example, if the video were losslessly compressed (for example using VC-2 in a lossless mode) then a different Flow ID would be used.
The same Flow ID is used across a variety of packaging / delivery mechanisms (different Flow Representations).
RTP (RFC 4175 or SMPTE ST 2110):
- Uncompressed video frames have a defined line-based byte packing arrangement
HD SDI (video carriage):
- Uncompressed video frames are carried in raster sequence using 10-bit words. In this example, the client should assume that only the “active” area (1920x1080) is relevant and so discard the rest of the video samples in the HD SDI signal.
C++ function:
- Returns video frames as planar with each 10-bit sample carried in a 16-bit integer.
- Alternatively, returns video frames with the 10-bit samples packed into bytes.
- In both cases, the data type is defined in the function documentation.
HTTP API:
- Deliver each video frame as a binary object over the API along with signalling of the byte packing (such as v210 or planar). Alternatively, define a fixed byte packing for the API (and specify it in the API documentation).
Store, MXF file, or similar:
- Same situation as for HTTP API above.
Flow technical metadata would include properties such as: 1920x1080, YUV, 10-bit, 4:2:0, colour space but not the byte packing.
Example 3: AVC Intra
In this example, we use a Flow ID of X to represent this AVC Intra coded video. In this example, we assume that each video frame is encoded only once by one encoder (with one set of settings) across all the Flow Representations with a Flow ID of X.
In this scenario the Flow is set at the level of these compressed video frames. More specifically, all the member Flow Representations are such that only bitstream assembly operations need to be carried out to create an AVC Intra compliant bitstream (or the bitstream is already compliant).
The same Flow ID is used across a variety of packaging / delivery mechanisms (different Flow Representations).
RTP (RFC 6184):
- The H.264 padding and filler are removed from the bitstream prior to carriage; SPS/PPS placement or repetition might be altered to suit the transmission; etc
MXF – example 1:
- Carried as AVC Intra (strictly following the AVC Intra specification – for example, each video frame is padded to be an exact size). Signalled in the MXF Header Metadata as AVC Intra
MXF – example 2:
- Carried as generic H.264 (for example, variable video frame size). Not signalled in the MXF Header Metadata as AVC Intra.
HTTP API, Store, etc:
- Deliver each video frame as a binary object along with signalling of the encoding (either simply “H.264” or “AVC Intra”, as appropriate)
MP4 file (H.264 mapped as per ISO/IEC 14496-15):
- The H.264 padding and filler are removed from the bitstream prior to carriage; the SPS/PPS are moved to the MP4 header; etc
Example 4: Lossy Compression in a Cloud Environment (Distributed Encoders etc)
In this example, we use a Flow ID of X to identify the Flow Representations that can be used interchangeably subject to a set of defined rules.
For example, this could be useful for:
- On-the-fly encoding of content without storing the result. The result might be different each time the encoded content is requested.
- Distributed encoders. The final encoded result might be assembled from multiple encoders running in parallel. These encoders might not produce identical results, for example because: there is a random aspect to the encoding; the encoding produced is affected by the frames that have already been encoded; the different encoders might be optimised for different kinds of content.
The assumption is that the baseband signal is virtually identical in each case so the Flow Representations can belong to the same Flow, but the properties of the encoding itself might be quite different. However, in this example the encoders are configured and operated to ensure that the encodings can be used interchangeably subject to a set of defined rules. For example, GOP placements might be restricted so that Flow Representations can be safely switched between at known intervals. In general, these rules should enable the client to answer questions such as:
- Can an encoded frame from one encoder be replaced with one from another encoder?
- Can one section of content encoded by encoder A be joined to the next section of content encoded by encoder B?
Example 5: Temperature Readings
In this example, a temperature sensor generates readings regularly. Each reading is a 32-bit signed-integer. We use a Flow ID of X to identify these 32-bit signed-integer readings. In this scenario the Flow is set at the “level” of the 32-bit signed-integers (and so there are no restrictions on the serialisation etc of the data).
The same Flow ID is used across a variety of packaging / delivery mechanisms (different Flow Representations).
C++ function:
- Returns the temperature reading for a given time-point. The return value is
int32; the endianness of the in-memory representation is transparent to the user.
HTTP API:
- The API has an end-point allowing the temperature reading to be requested at a specified
Time Value. The API response could be JSON (different schemas are possible, such as:{ "temperature": 32 }or{ "t": 32 }) or XML or another serialisation. The serialisation might be specified in the API documentation, or signalled in the API response, or could be requested by / negotiated with the client (perhaps using HTTP content negotiation headers)
File:
- The data might be stored in various different file formats, such as a MATLAB file or a Python pickle file. The format of the file will need to be defined and signalled.
Example 6: Weather Observations (Alternative 1)
In this example, a weather station makes weather observations regularly (from a number of sensors). The details of each observation are stored in a JSON data structure using JSON Schema Y. We use a Flow ID of X to identify these JSON-formatted weather observations. In this scenario the Flow is set at the level of the JSON-formatted (using Schema Y) weather observations (these are independent of details of the serialisation).
The following Flow Representations would all have the same Flow ID:
- The JSON is encoded as UTF-8
- The JSON is encoded as UTF-16
- Different amounts of superfluous whitespace are included
- Different orderings of JSON dictionary keys are used
Note:
- If a different JSON Schema (or another serialisation such as XML) is used then a different
FlowID is required.
Example 7: Weather Observations (Alternative 2)
In this alternative we define Flow at a higher level – we say that Flow is at the level of the weather observations themselves and so the same Flow ID is used regardless of serialisation (XML, different JSON Schemas, etc).
The data contained within each observation can be expressed as:
WeatherObservation {
int16 temperature;
uint16 humidity;
}
The same Flow ID is used across different implementations of this interface (different Flow Representations).
C++ function:
- Returns each weather observation as a struct as defined in the function documentation.
HTTP API:
- The API has an end-point allowing the weather observation to be requested at a specified
Time Value. The API response could be JSON (different schemas are possible, such as:{ "temperature":20, "humidity":50 }or{ "t":20, "h":50 }) or XML or another serialisation. The serialisation might be specified in the API documentation, or signalled in the API response, or could be requested by / negotiated with the client (perhaps using HTTP content negotiation headers)