Skip to content

Explanation –Flow

Please ensure you have read the Summary and Definitions before considering the further detail covered by this page.

Understanding the Flow Entity

The intention is that a Flow (identified by a Flow ID) provides a simple way to identify a particular expression of a Source. A Flow is a timed-data interface that might be implemented in different ways. These different implementations of a Flow are called Flow Representations.

In general, it is important not to think of a Flow as something that is stored in a file or carried in network packets – that is a Flow Representation and is just one practical realisation of the Flow. The Flow ID carried along with, or associated with, this Flow Representation is of importance here. It is this Flow ID that identifies the content and the interface (that is, as well as identifying the content it communicates what can be expected when handling the data). It helps to inform the user about how the data can be handled. For example:

  • It might be that a decoder is expected to handle either all the Flow Representations of a particular Flow or only a subset of the Flow Representations.
  • It might be that either: each transport used to carry a particular Flow requires a different Flow Representation; or that all transports used can carry any of the Flow Representations.

In each scenario a decision needs to be made about the level at which to set the Flow. The most practical level is chosen: this is the lowest level that is considered interesting for the scenario. A Flow might be set at a low level in which case all associated Flow Representations will be very similar. Alternatively, a Flow can be set at a higher level in which case the associated Flow Representations can be much more varied in how they implement the Flow.

There is a need to work across a very wide range of different scenarios (involving both media and data; that is, all kinds of content Sources). Across these scenarios there are practical reasons for handling data at different levels. In some scenarios, the sequences of bytes used to represent the data are of interest and this is the level we principally want to work at. In other scenarios, these sequences of bytes are not of interest and we principally want to handle the data at a higher (more conceptual) level. So, the lowest level that is interesting is at a different position in each scenario (at least when the most common operations are considered).

The need to handle data at different levels is illustrated in these simple examples:

  • Example 1: A time-based signal consists of a numeric value that varies over time. This is expressed by integers associated with Time Values.
  • We expect the integers to be carried using different serialisations (such as JSON, XML, plain text, binary) on different transports and APIs. The serialisation is known in each carriage (the serialisation is pre-defined, signalled or requested) and it is straightforward to match the serialisation in each case with a suitable decoder.
  • These different serialisations used for the integers are not generally of interest to the consumer of the data – the expectation is that the data will be decoded and all operations carried out on the integers.
  • Example 2: A video signal is represented using lossy compression (each video frame is a sequence of bytes generated by a video encoder).
  • Many operations need to be performed without decoding the video due to the cost of this. Therefore, the encoded data is of interest.
  • Also, there are often complexities and subtleties involved in the video coding process which cannot be fully described in practice (such as with technical metadata) but which do have an impact on how the encoded data can be used or the ability of a device to decode it (for example, not all H.264 coded video can be decoded by all H.264 decoders – the standard provides many options and tools). This too makes the encoded data of interest, and it is useful to be able to identify expressions of the video signal that are encoded in a particular way.

Further examples of the levels at which a Flow might be set are explained alongside Flow Representations.

Even if Flow is set (for a particular scenario) at a much higher "level" than the Flow Representations, this does not mean that all levels below Flow are unimportant. The levels below that defined by the Flow are not considered interesting when interfacing with the data but they do need to be taken care of somewhere in the wider system; that is, these lower levels are critical to a successful data transfer but are not of interest once that transfer has completed.

It is typically important for technical metadata to be provided about a Flow. This "Flow metadata" will apply to all Flow Representations associated with the Flow. Some Flow Representations might have extra Flow Representation-specific technical metadata associated with them. Additionally, it is important that the level of the Flow is clearly defined – this level might also need to be communicated (using additional metadata properties) between devices.

Guidance on how Flows can be applied to the handling of video and audio can be found in 3.0. Practical Guidance for Media

Notes on the Formal Definition of Flow

General Notes

A Flow can be regarded as a collection of Entrys. This collection of Entrys does not have any fixed / defined ordering, although the Entrys can of course be ordered by Time Value if required. However, in some real scenarios there could be Time Value-based restrictions on the order in which Entrys can be added to an existing Flow (for example, for a live media Flow). Note: This relates to mutability of the model entities which is commented on in 4.0. Appendix - Commentary. Also refer to notes on the relationship between Grains and Entrys.

There is no constraint that requires the technical properties of a Flow to be static. For example, the dimensions of a video or its framerate could change over time – but these properties would change in the same way across all Flow Representations associated with the Flow.

There is also no constraint that all Flow Representations which match the technical properties of a particular Flow must be part of that Flow – this constraint would not be practical to enforce. However, clearly a Flow is most useful if all eligible Flow Representations are members (for example, it is generally better if one Flow is used rather than two if this is permitted by the scenario and by the definition of Flow).

Interpretation of a Flow (for example, "rendering" of the Flow) will depend on the exact data types etc used. For example:

  • A Data Object might still contain some time-based information (e.g. information about how long the effect of the Data Object should persist for when rendered).
  • The exact interpretation of the Time Value with which a Data Object is associated will depend on how it is being used:
  • For example, for an SD video frame captured from SDI the Time Value might be the time at which the frame signal begins (Line 1) or it could be the time at which the visible picture for the frame begins (Line 23).
  • In some scenarios each Time Value will correspond with the time at which the Data Object came into existence. However, this is certainly not the case for all scenarios. For example, each Time Value might be used to map the Data Object onto a future point in time.

Notes on Data Objects

  • Data Objects might depend on each other (for example, inter-frame coded video containing B/P frames; with AAC coded audio multiple Data Objects are needed in order to be able to decode the audio). Any such inter-dependencies are not modelled here.
  • In practical media applications, Data Objects will typically contain data for a single media sample but could contain multiple media samples.
  • Regardless of how each Data Object is constructed, multiple Data Objects might be packaged together in Flow Representations. For example, a real system might choose to always transport Data Objects in packages of ten for practical reasons. This packaging must of course be completely reversible so that the original Data Objects and their Time Values can be accessed.
  • In practical media applications, a Data Object will typically be mono-essence, but it could be multi-essence. Essence data might exist in both mono-essence and multi-essence Flows. Mapping between these Flows might be necessary. For example, it might be useful to state that one multi-essence Flow (video + audio) contains all of the data required to (reversibly) create two mono-essence Flows (one video and one audio).
  • A Data Object can appear in a Flow more than once. Data Objects are not owned by a Flow and so a Data Object can be used by more than one Flow.