Basic Media Operations
←Practical Guidance for Media · Index↑ · Composite Media Operations→
This document explores a range of common production operations and the requirements for tracking identity and timing through them. Each granular operation is described and will be referred to in further documents.
This document is not intended to be an exhaustive set of potential operations. However, for any additional operation this document should be sufficient to provide an indication of how to treat it (in terms of its handling of identity and timing).
The headings below state how identity and timing must change as content is handled by the listed operations. The operations are grouped by the number of Flows which are input to and output from the operation purely for convenience in documenting them. For example, ‘1..*:1 Flow’ indicates a ‘many to one’ operation where multiple input Flows are combined in order to produce one output Flow.
These identity changes should be applied in any cases where a process has the potential to make a change to the content, even if it is not currently doing so. For example, a vision switcher’s output should be identified with its own Source ID, even if it is currently acting as a pass-through for a single input.
No Source or Flow Changes
The following operations result in no change made to the content’s existing identity or timing data. They are however likely to generate one or more new Flow Representations of the same Flow.
- Storage Read / Write
- Reads or writes content to/from disk with no changes made to its
Data Objects orTime Values.
- Reads or writes content to/from disk with no changes made to its
- Stored Content Playback – Retrieve Only
- Retrieves data from a store in order to play it back without making modifications to its
Time Values.
- Retrieves data from a store in order to play it back without making modifications to its
- Internal Conversion
- Modifies the byte packing or other aspect of a piece of content in a completely reversible fashion (into a new
Flow Representation), before returning it to its original state and providing it back to the user or system as an output.
- Modifies the byte packing or other aspect of a piece of content in a completely reversible fashion (into a new
- Data Transfer
- Transmission of a piece of content over a link. The content might be re-packed into a new
Flow Representationfor transmission over the link but is converted back to its original state before being returned to the user or system on the receiving end.
- Transmission of a piece of content over a link. The content might be re-packed into a new
- Buffering & Latency
- A time delay caused either intentionally or as a side effect of a processing or transmission operation. This delay makes no modifications to the
Time Values associated with the content.
- A time delay caused either intentionally or as a side effect of a processing or transmission operation. This delay makes no modifications to the
0:1 Flow
New Flow ID, New Source ID
The following operations result in the generation of new Time Values in an appropriate Time Context.
- Capture
- Converts light, sound or other physical inputs into a digital signal or creates purely synthesised content. This operation is intended to signify the point at which practically processable content comes into being, or the first point in the system where identity and timing data can be associated with it.
1:0 Flow
No New Flow ID, No New Source ID
The following operations transition content into an environment outside of the scope of the identity and timing model.
- Output
- Converts a digital signal into light, sound or other physical outputs. This operation is intended to signify the point at which content transitions to no longer being practically processable, or the transition out of an environment which is capable of carrying identity and timing data.
1:1 Flow
New Flow ID, New Source ID
These operations can be broadly summarised as performing one of the following actions:
- Content is fundamentally modified
- Content is re-timed
- New content is created from a single input
The following operations might modify the content’s Time Values and/or associated Time Context. Where applicable this action is described.
- Slow Motion / Speed Up
- Takes content which was recorded at a specific rate and plays it back at a faster or slower rate as desired (that is, its duration and its
Time Values are changed). This maintains a 1:1 relationship between input and outputData Objects and so must not include frame drops / repeats or interpolation between frames.
- Takes content which was recorded at a specific rate and plays it back at a faster or slower rate as desired (that is, its duration and its
- Stored Content Playback – Retrieve and Re-Time
- Retrieves data from a store before modifying its
Time Values for playout purposes. For example, in order to synchronise it with other data which was initially recorded at a different time.
- Retrieves data from a store before modifying its
- Time Shift
- Takes some original content and re-positions it to appear at a different time (modifying its
Time Values). It should maintain the original playback/recording rate and as such the same duration. This does not include changes ofTime Context, or variable offsets to theTime Values being applied.
- Takes some original content and re-positions it to appear at a different time (modifying its
- Variable Time Shift
- A Time Shift operation which includes a user control to modify the offset between the input and output
Flow. This will modify the ancestry between individualData Objects /Time Values over the running time of a system. - Note: Unlike a pure buffer, a traditional delay line modifies the implicit
Time Values in a sequence in order to change its relationship to time (and as a result match up with the timing of other sequences).
- A Time Shift operation which includes a user control to modify the offset between the input and output
- Frame Synchroniser / Rate Conforming
- Creates a new
Flowwhich is well aligned to a time grid where the input had an unintentional frequency or phase offset from the intended rate. This is likely to involve occasional frame drops or repeats,Time Valuere-assignment, and potentially aTime Contextchange.
- Creates a new
Time ContextRe-Mapping- Creates a copy of a
Flowbut with differentTime Values in a differentTime Context.
- Creates a copy of a
- Content Analysis
- Performs an analysis operation on individual media elements or sequences, producing a data set which could be used by other operations. The data set has concrete references to the content which was analysed to produce the results.
- Amplify / Attenuate
- Applies a multiplier to an aspect of a piece of content.
- Video Crop / Zoom
- Selects a lower resolution image from within a larger one.
- Note: Assumes that the picture remains at its lower resolution and is not mixed with anything else (such as a black frame).
- Video Squeeze / Squash
- Reduces the resolution in one dimension only, modifying the aspect ratio whilst maintaining the original field of view.
- Note: Assumes that the picture remains at its lower resolution and is not mixed with anything else (such as a black frame).
- Conversion to Monochrome
- Generates a black and white signal from a colour one.
- EQ Modification
- Amplifies or attenuates using filters affecting particular frequency ranges.
- Note: a real EQ might result in a more complex end result.
New Flow ID, Same Source ID
These operations can be broadly summarised as performing one of the following actions:
- Content is adjusted in terms of quality
- Content is interpolated to fit a different playback rate
The following operations make no changes to the content’s Time Values or Time Context. Time Values must be carried through 1:1 from input to output. One exception to this is described under “Sample Rate Conversion” where the Time Context remains the same, but new (interpolated) Time Values might need to be generated.
- Audio / Video Bit Depth Modification
- Increases or decreases the number of bits available to store each sample.
- Chroma Subsampling Modification
- Changes the video from 4:4:4 to 4:2:2 subsampling or similar.
- Byte Packing Modification
- Converts between (for example) v210 and 10-bit planar, or between UYVY and YUY2, maintaining the original samples intact.
- Colorspace Conversion
- Converts between (for example) BT.2020 and BT.709.
- Dynamic Range Conversion
- Converts between (for example) HDR and SDR.
- Colour Encoding Conversion
- Converts between (for example) YUV and RGB.
- Video Resolution Modification
- For example a UHD to HD modification, maintaining the original aspect ratio.
- Encode / Decode
- Takes input content and produces an approximation of the input content which might or might not be an entirely reversible operation. The content represented should cover the same time period but might be reduced in spatial resolution. Includes the reverse operation.
- Sample Rate Conversion
- Takes time domain sampled content and produces a new output with a different sampling rate, but must convey the same content over exactly the same duration.
- Note: Includes video frame rate conversion (which might involve interpolation) but does not include time stretches such as 24Hz to 25Hz (where the same content is played back without modification but at a different rate and so the duration is modified). For these cases see “Slow Motion / Speed Up” and “Frame Synchroniser / Rate Conforming”.
1..*:1 Flow
New Flow ID, New Source ID
In all of the following operations Time Values at each input must match such that they can be carried through to the output consistently. If the Time Values do not match up they must first be brought into line using an appropriate 1:1 operation.
- Switch
- Produces an output which could take any of a number of inputs but can only ever consist of a single input’s data at any one time.
- Note: This covers the case where we care about associating a single identity with the output
Flow, recording editorial decisions. A more system-level operation such as re-tuning a receiver to handle a stream of a differentFlowneed not change its identity.
- Mix
- Combines multiple inputs to create a new output which is a composite of one or more of the inputs. This might be by performing Switch operations, or by blending a set of inputs together to produce an output which consists of multiple inputs’ data at a single point in time. When blending inputs all of their
Time Values must match and the corresponding outputData Objects must be associated with the sameTime Value. - Note: This could include operations such as adding an image overlay to a video (DOG), or adding an inaudible watermark to a piece of audio.
- Combines multiple inputs to create a new output which is a composite of one or more of the inputs. This might be by performing Switch operations, or by blending a set of inputs together to produce an output which consists of multiple inputs’ data at a single point in time. When blending inputs all of their
- Combine
- Takes multiple pieces of input content which might consist of matching or differing essence formats and merges them into a single entity (a new
Flow). For example, this might involve taking several individual audio channels and combining them into a single multi-channel representation, or combining video and audio into an SDI multiplex.
- Takes multiple pieces of input content which might consist of matching or differing essence formats and merges them into a single entity (a new
- Video Keying
- Takes a “key” and “fill” as inputs, merging them to produce a single output image which is a composite of the two.
1:1..* Flow
New Flow IDs, New Source IDs
In the following operations the Time Values for each output must match the corresponding Data Object’s Time Value from the input.
- Split
- Takes input content which might consist of multiple components (for example, multiple audio channels or an SDI multiplex) and splits it into smaller units (for example, single audio channels or mono-essence components of a mux) without changing their timing relationship.
←Practical Guidance for Media · Index↑ · Composite Media Operations→