A method of instructing an enhancement decoder instance to reconstruct image frames of a video

Abstract

A method of instructing an enhancement decoder instance to reconstruct image frames of a video, comprising the steps of: configuring the enhancement decoder instance with a target level of quality at which the image frames are to be reconstructed; passing, to the enhancement decoder instance, data identifying target image frames to be reconstructed, the target image frames having been reconstructed at a current level of quality by a previous decoder instance; passing, to the enhancement decoder instance, an enhancement layer defining, for each target image frame, a difference between the current level of quality of the target image frame and the target level of quality for the target image frame; and instructing the enhancement decoder instance to: decode the enhancement layer into reconstruction data, and reconstruct the target image frames at the target level of quality using the reconstruction data.

Description

[0001] A METHOD OF INSTRUCTING AN ENHANCEMENT DECODER INSTANCE TO RECONSTRUCT IMAGE FRAMES OF A VIDEO

[0002] FIELD OF THE INVENTION

[0003] The present invention relates to video decoding, and in particular to a method of instructing an enhancement decoder instance to reconstruct image frames of an encoded video.

[0004] BACKGROUND

[0005] MPEG-5 Part 2 Low Complexity Enhancement Video Coding, LCEVC, is a video coding technique in which a lower (or base) quality version of a source video is encoded using an existing (or base) codec (such as AVC, HEVC, VP9, AV1 , EVC, WC, or the like), and an enhancement layer defining, for each image frame of the video data, a difference between the source quality and the decoded (and optionally upscaled) base quality of that image frame is encoded using a different (enhancement) codec. These encoded base quality image frames (that is, the encoded base layer), along with the encoded enhancement layer, may be retrieved by a client device (such as a computer, smartphone, games console, or the like).

[0006] Once the encoded base and enhancement layers are retrieved, the client device may decode the base layer using a base codec, and decode the enhancement layer using an enhancement codec, combine the decoded base and enhancement layers (optionally after upscaling of the decoded base layer) to reconstruct image frames at a quality up to (or equal to) the source image frames. These reconstructed image frames may then be rendered and output for display to a user.

[0007] LCEVC provides greater compression efficiency compared to encoding the source image frames at the source quality using a base codec — encoding a base layer of the source image frames naturally results in a smaller file size, and the high efficiency of the enhancement codec ensures that the resulting encoded enhancement layers are of a size which is less than the difference between encoded source image frames and encoded base layer.

[0008] LCEVC also provides near / at source quality reconstruction with lower computational complexity compared to encoding a base layer and solely relying on upscaling — the enhancement layer includes the high-frequency details that even the most advanced (and thus computational complexity) upscaling methods could not infer.

[0009] As will be appreciated, LCEVC techniques may be applied to numerous different video playing scenarios, such as live-streaming (streaming a live sports game or music concert, for example), video-on-demand streaming (streaming a movie or TV series from an internet-based video sharing platform, for example), playing downloaded content (playing a video sent to a smartphone via a messaging application, for example), or playing stored content (playing a video stored on a portable flash drive, sold-state drive, hard disk drive, or the like).

[0010] In the case of streaming (whether that be live-streaming or video-on-demand streaming), the streaming conditions may change during the course of streaming the video. These changes in streaming conditions may be, for example, a change in the amount of bandwidth available for streaming the video, a user-requested change in quality (the user may prefer a certain quality level, for example), a change in the amount of computational resources available for video decoding (the client device may execute other applications during runtime, for example), and the like.

[0011] To minimise / avoid frame rate drops or stuttering when displaying the video during such changes in streaming conditions, client devices may obtain a different quality version of the same video. For example, if the amount of available bandwidth reduces, then the client device may obtain (from a server, for example) a lower quality version of the video than what it had been decoding and displaying (a current quality, for example), otherwise the frame rate may be reduced, or stuttering (pauses between image frames) may occur due to streaming the current quality video stream over a network connection having reduced bandwidth. To carry out such retrieval of different quality video streams, client devices are typically configured with a stream switching mechanism that is driven based on metadata identifying a stream format. One example is Adaptive Bit-Rate streaming, or ABR, and a typical, standardised mechanism implementing this is Dynamic Adaptive Streaming over HTTP (DASH), also known as MPEG-DASH. Other common HTTP based streaming protocols include HTTP Live Stream (HLS), and Real-Time Messaging Protocol (RTMP), which both function in a similar manner to HTTP (DASH), although RTMP is not based on HTTP but is rather TCP / UDP based.

[0012] In ABR streaming, a video is encoded into multiple resolutions or bitrate combinations (that is, qualities), and each is typically referred to as a ‘rendition’ (or a representation, see below in the context of manifest files). Client devices determine the available bandwidth and retrieve the appropriate rendition according to their available bandwidth and in a format they can support. The collection of renditions is typically referred to as a ladder or a profile.

[0013] To implement this, in DASH, each client device retrieves a manifest file, known as an MPEG-DASH MPD (Media Presentation Description) file, which comprises metadata to enable the client device to construct a uniform resource locator (URL) to retrieve the appropriate stream based on available conditions (such as device capability, resolution, available bandwidth, state of the buffer, etc). Within each manifest file is a set of representations, each describing one of the versions of the content. The manifest file describes how the video has been split into pieces by a packager and how it is arranged so that the client device can request chunks of the video for subsequent playback. Generally, the representations include specific video and audio tracks for describing a version of the media stream, bitrates, resolution, and other information associated with the media content. The manifest may also be referred to as a playlist in other formats.

[0014] In the context of LCEVC, each rendition / representation in an ABR ladder / profile may comprise an encoded base layer and an encoded enhancement layer. Thus, when the client device obtains a new representation in response to changing streaming conditions, a new base layer and a new enhancement layer is typically obtained.

[0015] To ensure a smooth transition between the current and new qualities being displayed by the client device (that is, with minimal / no stuttering between the image frames of current and new qualities), the client device may decode / reconstruct and buffer a plurality of image frames at the current quality for subsequent display while the client device constructs a new locator to retrieve the new quality base and enhancement layers from the server.

[0016] However, variable frame rates and / or stuttering may still occur due to the highly variable computational burden placed on the client device when switching from decoding a current LCEVC representation (that is, a current base layer and a current enhancement layer) to decoding a new (target) LCEVC representation (that is, a target base layer and a target enhancement layer).

[0017] As will be appreciated, a similar issue may arise in non-streaming scenarios such as playing downloaded / stored content. For example, it may be that the stored content comprises multiple representations of the same video, each representation comprising a respective base layer and enhancement layer. The user of the client device may switch between these representations as they desire and / or the representation may be automatically selected based on the computational capability of the client device. In either case, when switching between representations, a new base layer is decoded, leading to a highly variable computational burden placed on the client device.

[0018] The present invention seeks to address this issue.

[0019] SUMMARY OF INVENTION

[0020] In a first aspect, there is provided a (computer-implemented) method of instructing an enhancement decoder instance to reconstruct image frames of a video, comprising the steps of: configuring the enhancement decoder instance with a target level of quality at which the image frames are to be reconstructed; passing, to the enhancement decoder instance, data identifying target image frames to be reconstructed, the target image frames having been reconstructed at a current level of quality by a previous decoder instance; passing, to the enhancement decoder instance, an enhancement layer defining, for each target image frame, a difference between the current level of quality of the target image frame and the target level of quality for the target image frame; and instructing the enhancement decoder instance to: decode the enhancement layer into reconstruction data, and reconstruct the target image frames at the target level of quality using the reconstruction data.

[0021] This way, image frames may be reconstructed at different levels of quality while only relying on the same base layer each time, the different levels of quality being obtained by combining different numbers of enhancement layers to this base layer. This reduces the frequency / occurrence of the client device decoding a new base layer in response to changing the LCEVC representation used for reconstructing and displaying image frames of a video, thereby leading to a reduction in the fluctuation in computational burden being placed on the decoder / client device.

[0022] Optionally, the configuring step comprises configuring the enhancement decoder instance to: receive data indicating the target level of quality, and specify or change one or more settings of the enhancement decoder instance in accordance with the target level of quality. For example, prior to runtime, software may be executed which configures the enhancement decoder instance to automatically specify its settings in response to receiving metadata indicating a target spatial resolution at runtime. This may enable a faster configuration compared to, say, providing a script at runtime due to the metadata being smaller than a script in terms of file size / bit count.

[0023] Preferably, the method comprises the step of determining the target level of quality based on at least one of a streaming condition, an amount of computational resources available for decoding, and a user preference. This way, a target level of quality that is most appropriate for the current situation may be determined, that is, by ensuring that the client device / decoder is not over-burdened and / or that the user may watch the video at their desired level of quality. Preferably, the method comprises the step of determining whether the target level of quality is higher than the current level of quality, wherein the configuring step is carried out if the target level of quality is higher than the current level of quality. This way, it can be ensured that an enhancement decoder instance is employed when the level of quality is to be increased during viewing of the video — if the level of quality is to be reduced, then an enhancement decoder instance (which provides video quality increases) is not needed.

[0024] Preferably, the step of passing, to the enhancement decoder instance, the data identifying the target image frames comprises one or more of: passing, to the enhancement decoder instance, a pointer pointing to a buffer storing the target image frames; and sending the target image frames output from the previous decoder instance directly to the enhancement decoder instance. This is to say that the target image frames may be passed to the enhancement decoder instance without having rendered them first, thereby reducing the amount of computational resources expended during the reconstruction process — rendering the target image frames at the current level of quality prior to further reconstruction at a target level of quality would be redundant, as it is the target quality image frames that are to be displayed, not the current quality image frames.

[0025] Optionally, the method comprises the step of instructing the previous decoder instance to reconstruct the target image frames at the current level of quality. Doing so may streamline the reconstruction of image frames — rather than the client device running respective different computer-implemented methods for each decoder instance (which would utilise a greater amount of the client device’s computational resources), a single computer-implemented method covering all decoder instances may be used.

[0026] Optionally, the previous decoder instance comprises a base decoder instance. That is, the previous decoder instance may be any known and / or available decoder. Typically, client devices comprise such base decoder instances in the form of a hardware decoder. Preferably, the base decoder instance is for decoding image frames encoded in a Moving Picture Experts Group, MPEG, format. This is to say that the base decoder instance is preferably configured to decode images encoded in an MPEG format. Examples of MPEG formats include AVC, HEVC, EVC, WC, and the like. This way, the previously discussed advantages provided by the present invention may be applied broadly to internet video streaming, given that much of the video content on the internet is encoded using an MPEG format (such as AVC, for example). As will be appreciated, internet video streaming is increasingly becoming a staple task for modem client devices.

[0027] Optionally, the previous decoder instance comprises a previous enhancement decoder instance. This is to say that the reconstructed current quality mage frames may have been reconstructed using a previous enhancement layer, meaning that the (configured) enhancement decoder instance is to decode a successive enhancement layer (dependent upon a reconstruction obtained using a previous enhancement layer) for further reconstruction of the image frames at the target level of quality. This way, the same base layer may be used for the target level reconstruction, all while using a different number of enhancement layers.

[0028] Optionally, the step of passing, to the enhancement decoder instance, the enhancement layer comprises one or more of: passing, to the enhancement decoder instance, a pointer pointing to a storage device storing the enhancement layer; and sending the enhancement layer retrieved externally directly to the enhancement decoder instance. In the case where the video to be displayed is being / has already been downloaded to the client device (in response to the client device receiving the video via, say, an instant messaging service, social media application, or the like), the enhancement layer(s) for that video may retrieved from the client device’s storage or directly from the client device’s I / O in the case where the video is still being received during playback.

[0029] Optionally, the step of passing, to the enhancement decoder instance, the enhancement layer comprises: retrieving a manifest file, the manifest file comprising metadata for constructing a locator to segments of the video, the metadata comprising: a representation pointing to the enhancement layer; a previous representation pointing to a previous encoding of the video for reconstructing the target image frames at the current level of quality; and a dependency attribute associating the representation with the previous representation; parsing the manifest file to construct a locator for retrieval of the representation; and retrieving the representation for reconstructing the target image frames at the target level of quality. In the case where the video is being streamed (a live-streamed sports game, or a video-on-demand from a video sharing platform, for example), the client device may retrieve the appropriate enhancement layer for the target level reconstruction from a server by using a manifest file / playlist to construct a locator (such as a URL) for the target level enhancement layer. As mentioned previously, this enhancement layer is to be retrieved without also retrieving a new base layer, yet an association between the enhancement layer and base layer / previous enhancement layer is preferably provided (in the form of a dependency attribute) to ensure that the appropriate enhancement layer is retrieved, as well as ensuring synchronisation.

[0030] Further optionally, at least part of the previous encoding of the video was encoded using a base coding scheme. Specifically, the base coding scheme may be in a Moving Picture Experts Group, MPEG, format. Examples of MPEG formats were discussed previously.

[0031] Further optionally, at least part of the previous encoding of the video comprises the previous enhancement layer. As mentioned previously, the same base layer may be used for the target level reconstruction, all while using a different number of enhancement layers.

[0032] Further optionally, the method comprises the step of retrieving the previous representation for reconstructing the target image frames at the current level of quality. Doing so may streamline the retrieval of representations — rather than the client device running respective different computer-implemented methods for the retrieval of each representation (which would utilise a greater amount of the client device’s computational resources), a single computer-implemented method covering all retrievals may be used. Optionally, the method comprises the step of rendering, for display, the target image frames at the target level of quality.

[0033] Preferably, the method comprises the step of displaying, on a display screen, the rendered image frames. This way, the user is able to view the target quality image frames. However, these rendered target quality image frames may be stored for viewing at a later time.

[0034] Optionally, a given level of quality (such as a current or target level of quality) comprises one or more of: a pixel count; a spatial resolution; a temporal resolution; a bit depth; and a colour space.

[0035] In a second aspect, there is provided a method of instructing an enhancement encoder instance to encode image frames of a video, comprising the steps of: passing, to the enhancement encoder instance, target image frames that have been reconstructed at a current level of quality by a decoder instance; passing, to the enhancement encoder instance, a source video having a target level of quality; and instructing the enhancement encoder instance to: generate reconstruction data defining, for each target image frame, the difference between the current level of quality of the target image frame and the target level of quality for the target image frame, and encode the reconstruction data as an enhancement layer.

[0036] In short, the method of the second aspect relates to the encoding of enhancement layers by using already decoded / reconstructed current quality image frames in lieu of a new base layer.

[0037] Optionally, the decoder instance comprises a base decoder instance (which may be for decoding MPEG format image frames, for example).

[0038] Optionally, the decoder instance comprises an enhancement decoder instance. This is to say that successive enhancement layers (that is, enhancement layers that are dependent upon image reconstructed using a previous enhancement layer) may be encoded. This is achieved by using, in lieu of a new base layer, current quality image frames that have been reconstructed using a base layer and enhancement layer(s). In a third aspect, there is provided a computer program comprising processor- implementable instructions which, when executed by a processor, cause the processor to perform the method according to the first aspect or the method according to the second aspect.

[0039] In a fourth aspect, there is provided a non-transitory computer-readable storage medium having stored thereon the computer program of the third aspect.

[0040] In a fifth aspect, there is provided a decoder, comprising a processor and a memory, the memory having stored thereon processor-implementable instructions which, when executed by the processor, cause the processor to perform the method of the first aspect.

[0041] In a sixth aspect, there is provided an encoder, comprising a processor and a memory, the memory having stored thereon processor-implementable instructions which, when executed by the processor, cause the processor to perform the method of the second aspect.

[0042] BRIEF DESCRIPTION OF DRAWINGS

[0043] Embodiments of the present invention will now be described by way of example with reference to the accompanying drawings, in which:

[0044] Figure 1 shows a method of decoding and rendering a video encoded using enhancement coding according to embodiments of the present invention;

[0045] Figure 2 shows a method of encoding a video using an enhancement coding operation according to embodiments of the present invention;

[0046] Figure 3 shows a known, high-level schematic of an LCEVC encoding, decoding and transport process;

[0047] Figure 4 shows a known high-level schematic of an ABR process of implementing enhancement coding;

[0048] Figure 5 shows an example of a known ABR profile for enhancement coding;

[0049] Figure 6 shows a flowchart illustrating a method according to embodiments of the present invention; Figure 7 shows a high-level schematic of an LCEVC encoding process according to embodiments of the present invention;

[0050] Figure 8 shows a high-level schematic of an ABR process of implementing enhancement coding according to embodiments of the present invention; and Figure 9 shows a high-level approach to encapsulation according to the embodiments of the present invention.

[0051] DETAILED DESCRIPTION

[0052] A method of instructing an enhancement decoder instance to reconstruct image frames of a video disclosed. In the following description, a number of specific details are presented in order to provide a thorough understanding of the embodiments of the present invention. It will be apparent, however, to a person skilled in the art that these specific details need not be employed to practice the present invention. Conversely, specific details known to the person skilled in the art are omitted for the purposes of clarity where appropriate.

[0053] As mentioned previously, switching between LCEVC representations of a video during streaming / playing of the video may result in variable frame rates and / or stuttering due to the highly variable computational burden placed on the client device when switching from decoding a current LCEVC representation (that is, a current base layer and a current enhancement layer) to decoding a new (target) LCEVC representation (that is, a target base layer and a target enhancement layer).

[0054] To address this issue, rather than decoding a new base layer (or stream) and enhancement layer (or stream) as part of changing the LCEVC representation from a currently quality one to a target quality one (as has typically been the case heretofore), the client device may retrieve a new enhancement stream (defining the difference in quality between current and target qualities), and use the already decoded / reconstructed current quality image frames in lieu of a new base layer. This technique may be referred to as a “nested LCEVC” approach, as the output from a LCEVC decoder may used as part of the input to the / another LCEVC decoder. As a non-limiting example, and turning now to Figure 1 , a client device may currently be reconstructing image frames of a video at a first level of quality (Image stage 1). LCEVC decoder 1 may reconstruct these first quality image frames by decoding and combining base layer 1 BL1 and enhancement layer 1 EL1. As will be appreciated, LCEVC decoder 1 may be embodied in a player of the client device (such as dash.js in the case of MPEG DASH), a decoder module or any other module on the device. Alternatively, such functionality of LCEVC decoder 1 may be split over multiple modules such as an ABR handler, Data handler, Media Handler, Player and decoder, for example.

[0055] Alternatively or in addition, browser-based players are contemplated within the scope of the present invention. A non-limiting example of this is Shaka player, which is a browser-executed javascript that calls to the LCEVC decoder of the client device when decoding functionality is required. For example, Shaka player passes encoded base and enhancement layers from a buffer of the browser to the LCEVC decoder of the client device, receives the reconstructed image frames, and then render these image frames to a browser canvas.

[0056] Due to streaming conditions, user preferences, client device capability, or the like, the client device may determine that the image frames are to be rendered at the first level of quality and then displayed to a user (via an associated display screen, for example). In this case, the image frames obtained at Image stage 1 (that is, the first quality image frames) are rendered and output for display.

[0057] In the case where the client device determines that the image frames are to be rendered at a second level of quality (higher than the first level of quality), the client device obtains enhancement layer 2 (defining the difference between first and second levels of quality) and uses the reconstructed first quality images (from Image stage 1) as in lieu of a second base layer. For example, the reconstructed first quality image frames and EL1 are input to LCEVC decoder 2 (which may be the same as or different from LCEVC decoder 1) for reconstruction of the image frames at the second level of quality (Image stage 2). LCEVC decoder 2 may reconstruct these second quality image frames by decoding EL2 and combining it with the first quality image frames. Subsequently, image frames obtained at Image stage 2 (that is, the second quality image frames) are rendered and output for display.

[0058] In the case where the client device determines that the image frames are to be rendered at a third level of quality (higher than the second level of quality), the client device obtains enhancement layer 3 (defining the difference between second and third levels of quality) and uses the reconstructed second quality images (from Image stage 2) as in lieu of a third base layer. For example, the reconstructed second quality image frames and EL2 are input to LCEVC decoder 3 (which may be the same as or different from LCEVC decoder 1 and / or LCEVC decoder 2) for reconstruction of the image frames at the third level of quality (Final Image). LCEVC decoder 3 may reconstruct these third quality image frames by decoding EL3 and combining it with the second quality image frames. Subsequently, image frames obtained at Final Image stage (that is, the second quality image frames) are rendered and output for display.

[0059] It will be appreciated that while the example of Figure 1 outlines three different levels of quality, the above technique are more broadly applicable to two or more levels of quality. Moreover, it will be appreciated that the first level of quality image frames need not be a reconstruction obtained from a base layer and enhancement layer, but may be a reconstruction solely obtained from a base layer.

[0060] In any case, it will be appreciated that the above techniques enable a plurality of different levels of quality to be obtained from one base layer, thereby reducing the frequency / occurrence of the client device decoding a new base layer in response to changing the LCEVC representation used for reconstructing and displaying image frames of a video. As will be appreciated, base layers typically require greater computational resources to decode compared to enhancement layers due to their respective contents — base layers are self-sufficient in that they comprise information for reconstructing image frames outright, whereas enhancement layers define information such as residuals and high-level details that are combined with a base layer to reconstruct image frames. Therefore, decoding base layers of differing qualities during streaming of video may result in a highly fluctuating computational burden on the client device, making it difficult to allot the appropriate computational resources to the task of decoding streams for a given moment in time during the stream. The present invention enables the decoding of image frames at a new quality by only retrieving an enhancement layer that is dependent upon the current quality image frames already being reconstructed by the client device. As enhancement layers require less computational resources to decode compared to base layers, the amount of fluctuation in the computational burden placed on the client device to decode the image frames of differing qualities is reduced — the same base layer may be used in the reconstruction of current quality and new quality image frames, with the number of successive enhancement layers / streams used in such reconstruction changing instead.

[0061] As will be appreciated, the implementation of the enhancement coding operation used to encode video into base and enhancement layers may be carried out in a different manner in order to provide successive enhancement layers, that is, enhancement layers that depend on image frames previously reconstructed using a previous enhancement layer.

[0062] Turning now to Figure 2, a plurality of source videos are obtained. These source video may be obtained by downscaling a single source video multiple times (as is the case in Figure 2), for example. The lowest (that is, first) quality source video may be input to an LCEVC encoder to obtain BL1 and EL1 , BL1 being a downscaled version of the first quality image frames encoded using a base encoding format (such as an MPEG format), and EL1 being an encoding of the residuals defining the difference between the base level of quality of decoded BL1 and the first level of quality of the first quality source video. The LCEVC encoder may be located on the server side of a streaming / distribution network, for example. Once encoded, BL1 and EL1 may be stored / packaged ahead of distribution / streaming to a client device.

[0063] Copies of BL1 and EL1 may be taken and subsequently decoded and combined to reconstruct image frames at the first level of quality — see Recon(BL1+EL1). The reconstructed first quality image frames are then treated as if they were a decoded second base layer. This is to say that the reconstructed first quality image frames are input to the same / a different LCEVC encoder along with a second quality source video. The LCEVC encoder then outputs EL2, that is, an encoding of the residuals defining the difference between the first level of quality of Recon(BL1+EL1) and the second level of quality of the second quality source video. Once encoded, EL2 may be stored / packaged ahead of distribution / streaming to a client device.

[0064] Copies of BL1 , EL1 and EL2 may be taken and subsequently decoded and combined to reconstruct image frames at the first level of quality — see Recon((BL1+EL1)+EL2)). The reconstructed second quality image frames are then treated as if they were a decoded third base layer. This is to say that the reconstructed second quality image frames are input to the same / a different LCEVC encoder along with a third quality source video. The LCEVC encoder then outputs EL3, that is, an encoding of the residuals defining the difference between the second level of quality of Recon((BL1+EL1)+EL2)) and the third level of quality of the third quality source video. Once encoded, EL3 may be stored / packaged ahead of distribution / streaming to a client device.

[0065] It should be noted that embodiments of the present invention are not limited to only streaming scenarios, but may also be implemented in use cases such as playing stored / downloaded content, for example.

[0066] Before a more detailed description as to how the present techniques are implemented at both the decoder (client device) and encoder (server), an enhancement coding scheme, LCEVC, suitable for use with the concepts of the present invention will be described first. LCEVC will be described in the context of Figure 3. Throughout the description the terms hierarchical coding and enhancement coding may be used interchangeably and while examples are described in the context of LCEVC, it will be understood that the concepts described may be suitable to any similar hierarchical or enhancement coding scheme.

[0067] LCEVC adopts a multi-layer approach where any base codec (e.g. H.264, HEVC, AV1 and others), is enhanced via an additional low bitrate stream. LCEVC’s data stream structure is defined by two component streams: a base stream (or layer) decodable by a hardware decoder; and an enhancement stream consisting of one or two enhancement layers suitable for software processing implementation with sustainable power consumption. The enhancement provides improved compression efficiency to existing codecs, and reduces encoding and decoding complexity, for on demand and live streaming applications.

[0068] Figure 3 below illustrates how LCEVC operates on both encoding and decoding pipelines. The base encoding (whether H.264, HEVC or others) is performed on a down-scaled input at a lower resolution, typically a quarter of the desired output resolution. LCEVC enhancement data is calculated at the two resolutions providing two levels of correction and enhancement. The LCEVC encoder generates the enhancement stream from two inputs: the base encoding and the original uncompressed full resolution video, effectively correcting the quality gap between the two. The LCEVC data can be packaged together with the base elementary stream (for example as Supplemental Enhancement Information (SEI) of the Network Abstraction Layer, NAL), as frame metadata in a WebM container or in an additional data Packet Identifier (PID) in a MPEG-2 TS stream.

[0069] The LCEVC decoder works at an individual video frame level. As input it takes the decoded low-resolution picture from the base video decoder, which is typically provided by a hardware decoder on the device, and the LCEVC enhancement decoded in software to produce a full-resolution picture ready for rendering on the display view. Example implementations of decoding LCEVC are set out in WO2022 / 023739 and WO2023 / 118851 which are incorporated by reference.

[0070] As illustrated in Figure 3, in the encoder 100, an input full resolution video, i.e. a source video 102, is processed to generate various encodings. A first encoding (base encoding 110) is produced by feeding a base encoder 106 (e.g., AVC, HEVC, VP9, or any other codec) with a down-sampled version of the input video, which is produced by down-sampling 104 the input video 102. In the example shown, the downsampling is to a quarter resolution but this is optional as will be elaborated on elsewhere. The base encoding 110 may be referred to as a base layer. A second encoding (level 1 encoding 112, an example of an enhancement encoding) is produced to create first level corrections 116 by applying an encoding operation to the residuals obtained by taking the difference between a reconstructed base codec video and the down-sampled version of the input video. The reconstructed base codec video is obtained by decoding the output of the base encoder 106 with a base decoder. In typical implementations, the level 1 encoding 112 is optional. This level 1 encoding 112 may be referred to as a first enhancement layer.

[0071] A third encoding (level 2 encoding 114, another example of an enhancement encoding) is produced to create first level corrections 120 by processing the residuals obtained by taking the difference between an up-sampled version (i.e. normative upsampling 118) of a corrected version of the reconstructed base coded video and the input video 102. This level 2 encoding 114 may be referred to as a second enhancement layer.

[0072] The enhancement layer(s) and the base layer are typically combined (i.e. as illustrated by mux 122) and the full resolution video is encoded in layers. This is then typically transmitted using standard packaging and transmission protocols, an example of which will be described below in the context of Figure 4.

[0073] While often the corrections forming the enhancement layer involve an upsampling to higher resolution, generally any improvement in quality may be provided by the enhancement layer. For example, resolution, visual quality (VQ), bit depth (e.g. 8 to 10b) or colour space (e.g. HDR).

[0074] At the decoder 140, the encoded video is separated (i.e. as illustrated by demux 142) into an ancillary data stream 144 and a video stream 146. The decoder receives the layers (a base encoding, an optional level 1 encoding and a level 2 encoding) together with headers containing further decoding information. The base encoding, i.e. in video stream 146, is decoded by a base decoder 148 corresponding to the base decoder used in the encoder. At an enhancement decoder 150, which receives headers and the enhancement layers in the ancillary data stream 144, its output is combined with the decoded residuals obtained by decoding the level 1 encoding (if present). The combined video is up-sampled and further combined with the decoded residuals obtained by applying a decoding operation to the level 2 encoding to output the full resolution video 152.

[0075] Figure 4 illustrates a non-limiting example of implementing enhancement coding using standard packaging and transmission protocols. It should be noted that while the following relates to streaming scenarios and use cases, the present invention is not limited thereto — the present invention is equally applicable to, say, a case where content that is stored / downloaded on the client device is being played.

[0076] The source video 102 is encoded by the encoder 100 to produce the full resolution video encoded in layers. An example of how the encoder 100 may be implemented using an encoder integration layer is described in WO2022 / 023747 which is incorporated herein by reference. While any suitable container format may be used for the video, an mp4 container format containing the base layer and enhancement layer is illustrated here.

[0077] The encoder will be instructed to output a set of videos in different qualities, referred to here as bitrate variants, depending on the desired configurations. For example, the set of files may include videos at different resolutions (e.g. 1080p, 720p, etc.) and different qualities, frame rates or bitrates at the same resolution (e.g. 720p, 60 fps and 720p, 30fps etc.).

[0078] LCEVC is designed so that legacy devices are still able to decode and play the base video, even if the stream with both the enhancement and base layers is retrieved. In other words, LCEVC is inherently backwards-compatible since devices that have not yet been updated to decode LCEVC happily play the underlying base codec. However, for those legacy devices, in this situation they are streaming data they are not using and therefore underutilising their available bandwidth.

[0079] To address this, as well as video files of the full resolution video encoded in layers, in order to support legacy devices, the encoder will also typically be instructed to output variants encoded using the base codec only. This enables those files to be retrieved by legacy client devices without the accompanying enhancement data and thereby utilising more of their available bandwidth on viewable video data.

[0080] The set of variants, i.e. the set of encodes or renditions, are packaged by a packaging service referred to here as a packager 202. An example packager is the Shaka packager. The packager 202 splits each rendition into chunks of a specified duration (e.g. 2 or 4 seconds) and records how the video is split and the order of the chunks in a media presentation description file (MPD), otherwise referred to as a manifest file 205. The packaged videos and manifest files are then placed into storage 204. Once stored, the chunks of the video may be referred to as segments and the segments and manifest file are made available to retrieve over a network 206.

[0081] The manifest file is a document, typically in Extensible Markup Language (XML), that contains metadata required by a client device to construct appropriate uniform resource locators (URLs) to access the segments. The manifest is structured hierarchically. The media presentation contains a sequence of one or more periods; a period contains one or more Adaptation Sets; an Adaptation Set contains one or more representations; a representation contains one or more segments; and segments carry the locations of the actual media data and associated metadata.

[0082] The Media Presentation (i.e. the overall hierarchy) contains information about all the different media types in the content. The most common media types are video, audio, and closed captioning data. At the top level, the MPD contains information including the MPD Profile, minimum buffer time, presentation duration, and maximum segment duration, and title. The video is structured into multiple Periods, for example for advert insertion or chapters. An Adaptation Set catalogues the available representations of the media. For example, as shown in the schematic in Figure 4, there may be an Adaptation Set for the base video and an Adaptation Set for the LCEVC video. There may also be an Adaptation Set for a subtitle track and an audio track, for example. Within each Adaptation Set, the player will only select one Representation for a particular Segment, but the Representation can change from one Segment to the next. Video Adaptation Sets typically contain multiple Representations, one for each resolution / bitrate, allowing the media player to select the best available quality without buffering. If the video is available in more than one codec, each codec will be in a different Adaptation Set. Within an Adaptation Set, a Representation describes one of the versions of the content. Each Representation typically includes the average bandwidth of the Representation to allow for the client device to choose which representation to retrieve. Each Representation includes a set of Segments which contain the information required to construct the URLs to download the content. The MPD can either provide a list of the segment URLs or a template that the player uses to build the URLs dynamically.

[0083] The structure of an implementation of MPEG DASH will be well understood to the skilled person and so this description is provided for context only. Further information may be found in the ISO / IEC 23009 standards documentation.

[0084] Figure 4 illustrates two client devices 208, 210. Client device #1 208 may be an LCEVC capable device and client device #2 210 may be a legacy device not capable of decoding LCEVC video. When referring to functions of the client device, such functionality may be embodied in a player on the device, a decoder module or any other module on the device, or such functionality may be split over multiple modules such as an ABR handler, Data handler, Media Handler, Player and decoder, etc. The client devices 208, 210 may first retrieve (i.e. download) the manifest file 205 over the network 206, for example as served by a content delivery network CDN. The manifest file is parsed by the client device to understand the mechanism of serving the video. The client device 208, 210 senses its available bandwidth and selects an available representation according to the available bandwidth and the representation supported by the device. The client device then requests chunks from one of the video’s bitrate variants depending on its buffer size and available bandwidth. In other words, the client device selects segments for download according to the available bandwidth and the device status (e.g. video supported and buffer conditions). In the example of Figure 4, client device #1 208 supports LCEVC and has 2Mbps available. It accordingly parses the manifest file to identify that the most appropriate segments for download are those according to the b1e1 representation and from the metadata in the manifest file it constructs an appropriate URL to retrieve the segments to fill its buffer. The retrieved segments are passed to an LCEVC decoder for decoding. The client device 208 continuously senses its bandwidth conditions and buffer levels and adaptively streams content. According to ABR principles, if bandwidth conditions change or buffer levels change for example, the device can retrieve segments from a different representation, e.g. within the same Adaptation Set. As mentioned previously, client device #2 210 does not support LCEVC. Moreover, in this example, client device #2 210 senses the same available bandwidth as client device #1 208. Accordingly, client device #2 210 would choose from the AVC Adaptation Set, since it does not support LCEVC and would choose to retrieve segments from Representation b2 since that indicates an average bandwidth of 2Mbps and would then construct an appropriate URL from the metadata.

[0085] The Representations within an Adaptation Set may be referred to as an ABR ladder or ABR profile. Figure 5 illustrates a sample ABR profile, from a report written by industry analyst Jan Ozer in April 2020. The H.264 ABR Ladder profiles, encoded using x264 (medium preset), are in line what recommended by Apple (RTM) in the HLS Authoring Specification. The LCEVC H.264 ABR Ladder profiles have been optimised to achieve both a quality uplift and a reduction in the total average bitrate streamed. As shown, an ABR ladder using LCEVC balances quality and efficiency. In other words, there may be a lower bitrate for the top profile at a similar or higher quality as x264, while improving quality & resolution of lower profiles. To achieve maximum efficiency, it may be possible to reduce data rate as much as possible while matching x264 quality on all profiles to decrease costs (e.g., transcoding, CDN) and maximise network capacity (e.g., number of simultaneous users).

[0086] In the case of playing stored / downloaded content, switching between multiple stored representations of the video may be triggered in response to a user selection (the user may request a higher quality video output) and / or a change in client device capabilities. For the latter case, it may be that a computer playing the video was also executing, say, word processing software, an internet browser, image editing software, or the like, but now is only decoding the video, for example. This would result in an increase in the amount computational resources that are available for video decoding, meaning that a higher quality video may be output, for example.

[0087] Turning back to the non-limiting streaming example of Figure 4, the amount of bandwidth available for streaming image frames may increase at some point during runtime (from, say, 2Mbps to 3 Mbps for example). Before the bandwidth increase occurs, LCEVC-enabled client device #1 208 may have been retrieving stream b1e1 (after constructing a locator for the b1e1 stream using manifest file 205, for example), comprising a base layer b1 and enhancement layer e1 (dependent on b1) which, when decoded and combined, would result in the reconstruction of image frames at a current level of quality.

[0088] Using LCEVC techniques known heretofore, client device #1 208 would retrieve the b2e2 stream (and cease retrieving the b1e1 stream) in response to the bandwidth increase from 2Mbps to 3Mbps.

[0089] However, in accordance with the techniques of the present invention (described earlier with reference to Figures 1 and 2 above), client device #1 208 may additionally retrieve enhancement stream e2 while still retrieving stream b1e1. Once streams b1e1 and e2 are obtained, client device #1 208 may use a base decoder instance and a first enhancement decoder instance to decode base layer b1 and enhancement layer e1 , respectively. The decoded b1 and e1 layers may then be combined to produce reconstructed image frames at the current level of quality (as was previously the case before the bandwidth increase). A second enhancement decoder instance may be configured to reconstruct images frames at a new (target) level of quality, using the reconstructed current quality image frames (derived from the stream b1e1) as the decoded base layer, and decoding enhancement stream e2. This configuring may occur before or after, independently of or in response to, detection of the change in streaming conditions, construction of a new locator, or retrieval of the enhancement stream. In any case, this second enhancement decoder instance may then combine the reconstructed current quality image frames with the decoded enhancement stream e2 to reconstruct images at the target level of quality (which in this case may be higher than the current level of quality due to the increase in bandwidth).

[0090] As will be appreciated, in Figure 4, enhancement stream e2 is shown as being dependent on base stream b2. This is to say that enhancement stream e2 defines, for each image frame reconstructed using the b2 stream, a difference between the current level of quality of the reconstructed image frame and the target level of quality for the image frame. However, in the present invention, enhancement stream e2 instead defines, for each image frame reconstructed using the b1e1 stream, a difference between the current level of quality of the image frame and the target level of quality for the image frame. The enhancements streams e2 of these two examples are not necessarily defining the same quality changes, as image frames reconstructed using the b1e1 stream is not necessarily at the same level of quality as those image frames reconstructed using the b2 stream. Obtaining an enhancement stream e2 that is dependent on stream b1e1 instead of stream b2 (hereafter called a “successive enhancement stream”) shall be discussed later herein.

[0091] As will be appreciated, if a decrease in available streaming bandwidth occurs at a later point during runtime, client device #1 208 may cease retrieving enhancement stream e2, while continuing to retrieve stream b1e1. As a result the second enhancement decoder instance ceases its decoding and reconstruction, leaving the first enhancement decoder to reconstruct the image frames. Thus, the image frames output for display will revert from the target (higher) quality to the aforementioned current (lower) quality.

[0092] Furthermore, it will be appreciated that this technique is equally applicable in the case where the current quality image frames have been reconstructed using only a base decoder instance. This is to say that the present invention is not limited to only further enhancing already-enhanced image frames, but rather may also extend to enhancing base quality image frames as and when streaming conditions allow it. For example, client device #1 208 may first retrieve base stream b1 only (as steaming conditions may be poor), and may later additionally retrieve enhancement layer e1 while still retrieving base stream b1 if the streaming conditions become better (increased bandwidth, for example). A first enhancement decoder instance may be configured to decode retrieved enhancement stream e1 , and client device #1 208 may combine decoded b1 and e1 streams to reconstruct image frames at a higher quality than that obtainable solely using the b1 stream. If streaming conditions worsen later on, client #1 208 may cease retrieving stream e1 , while continuing to retrieve stream b1 .

[0093] It should be noted that the retrieving of base stream b1 and enhancement streams e1 and e2 in the present example is distinct from that discussed previously with respect to Figure 3. This is because in the present example, the e1 and e2 streams are not combined / multiplexed during the encoding stage, thereby enabling flexible retrieval of e1 and e2 streams as appropriate (that is, depending on streaming conditions, for example). In the example of Figure 3, the two encoding layers 112 and 114 are combined / multiplexed with the base layer 110, meaning that retrieval by the client device is an “all or nothing” procedure — either all of streams 110, 112 and 114 are retrieved or none of them are. This is similarly the case in Figure 4, where manifest file 205 comprises LCEVC representations b1e1 (where base layer b1 is multiplexed with enhancement layer e1) and b2e2 (where base layer b2 is multiplexed with enhancement layer e2).

[0094] Put differently, the example of Figure 3 relates to a case where one representation on an LCEVC ABR ladder is base layer 110 in conjunction with enhancement layers 112 and 114 (all of these layers being comprised in the same mp4 file, for example), whereas the present invention provides a finer granularity — up to three representations would be obtainable using the present invention, the first representation corresponding to base layer 110 alone (in a first mp4 file, for example), the second corresponding to base layer 110 (in the first mp4 file) and enhancement layer 112 (in a second mp4 file but linked to the first mp4 file via a dependency ID attribute, for example), and the third corresponding to base layer 110 and enhancement layer 112 and enhancement layer 114 (in a third mp4 file but linked to the second mp4 file via another dependency ID attribute, for example).

[0095] Alternatively or in addition, the present example may be extended beyond the example presented in Figure 3. For example, there may equally be a case where the first representation corresponds to base layer 110, the second corresponds to base layer 110 in conjunction with enhancement layers 112 and 114 (that is, a reconstruction using a level 1 and level 2 enhancements), and a third to base layer 110 in conjunction with enhancement layers 112 and 114 and yet a further enhancement stream comprising a level 2 enhancement layer and optional level 1 enhancement layer. This is to say that each successive enhancement stream may comprise one or two enhancement layers.

[0096] Additionally, the techniques of the present invention may be extended to other (non-streaming) scenarios. For example, when the user of the client device requests to view the video at a higher quality than is currently being displayed, the client device may continue to reconstruct image frames at the current level of quality (by decoding downloaded / stored base layer and optional first enhancement layer), and then decode a (further) downloaded / stored enhancement layer to obtain residual data defining the difference between the current and higher (target) levels of quality. The current quality image frames and the residual data may subsequently be combined to reconstruct the image frames at the higher level of quality. As will be appreciated, this technique may similarly be carried out in the case where the computational resources available for decoding and reconstructing image frames increases during playing of the video (such as when other software / applications being executed on the client device are closed down).

[0097] This technique eliminates / reduces the frequency with which base layer encodings of different qualities are retrieved and decoded in response changes in streaming conditions. As mentioned previously, base layers typically require greater computational resources to decode compared to enhancement layers due to their respective contents. By reducing the frequency / occurrence of changing the base layer to be decoded, the amount of fluctuation in the computational burden placed on the client device to decode the image frames of differing qualities is reduced — the same base layer may be used in the reconstruction of current quality and new quality image frames, with the number of successive enhancement layers / streams used in such reconstruction changing instead.

[0098] Method Overview

[0099] Turning now to Figure 6, in embodiments of the present invention, a method of instructing an enhancement decoder instance to reconstruct image frames of a video comprises the steps of: configuring the enhancement decoder instance with a target level of quality at which the image frames are to be reconstructed (step S600); passing, to the enhancement decoder instance, data identifying target image frames to be reconstructed, the target image frames having been reconstructed at a current level of quality by a previous decoder instance (step S602); passing, to the enhancement decoder instance, an enhancement layer defining, for each target image frame, a difference between the current level of quality of the target image frame and the target level of quality for the target image frame (step S604); and instructing the enhancement decoder instance to: decode the enhancement layer into reconstruction data, and reconstruct the target image frames at the target level of quality using the reconstruction data (step S606).

[0100] This is to say that an enhancement decoder instance may be instructed to use, in lieu of a decoded base layer, current quality image frames reconstructed by an upstream decoder instance, decode an enhancement layer that is dependent upon the reconstructed current quality image frames, and combine the reconstructed current quality frames and decoded enhancement layer (that is, reconstruction data) to reconstruct the image frames at a target level of quality (which was determined in response to a change in streaming conditions, for example).

[0101] As will be appreciated, the above method may be performed by a client device, player or decoder. As mentioned previously, this configuring of (additional) enhancement decoder instance as and when needed reduces the amount of fluctuation in the computational burden placed on the client device when decoding ABR streams compared to a case where new base layer encodings are required to decoded and output each new ABR representation.

[0102] Configuring the Enhancement Decoder Instance

[0103] In embodiments of the present invention, the method comprises the step of configuring the enhancement decoder instance with a target level of quality at which the image frames (of a video) are to be reconstructed (step S600).

[0104] As mentioned previously, a given level of quality (such as the aforementioned current and target levels of quality) may comprise one or more of: a pixel count, a spatial resolution, a temporal resolution, a bit depth, and a colour space. It will be appreciated that these examples are entirely non-limiting — other metrics pertaining to the level of quality of image frames are contemplated within the scope of the present invention.

[0105] As mentioned previously, LCEVC enhancement layers are typically decoded in software. In this case, it will be appreciated that an enhancement decoder instance (such as enhancement decoder 150 of Figure 3) may be embodied by a processor of a client device (such as a CPU or GPU) acting under suitable software instruction. Preferably, a suitably configured GPU embodies the enhancement decoder instance.

[0106] As mentioned previously, the enhancement decoder instance receives an enhancement stream. This enhancement stream may comprise one or two enhancement layers (such as optional level 1 enhancement layer 112 and level 2 enhancement layer 114). The enhancement decoder instance decodes the enhancement stream into reconstruction data (residuals, high-level details, for example) and combines the reconstruction data with the previously reconstructed image frames (output from base decoder 140 or from a previous enhancement decoder instance 150, for example) to reconstruct the image frames at the target level of quality.

[0107] Prior to this, the settings of the enhancement decoder instance are specified such that it may reconstruct image frames at the target level of quality. This may be achieved by (further) software configuration — the processor may execute software instructions which causes one or more settings of the enhancement decoder instance to be selected / specified such that any subsequent decoding and reconstruction it performs relates to obtaining a target pixel count, bit depth, colour space, or the like, for the image frames it outputs.

[0108] This software configuration may involve inputting a script to the processor, causing the processor to specify the enhancement decoder instance settings to whatever settings values are contained in the script.

[0109] Alternatively, this software configuration may involve executing a set of software instructions prior to runtime, this software configuring the enhancement decoder instance to automatically specify its settings in response to receiving data indicating the target level of quality (metadata indicating a target spatial resolution, for example).

[0110] Hence more generally, configuring step S600 may optionally comprise configuring the enhancement decoder instance to: receive data indicating the target level of quality, and specify one or more settings of the enhancement decoder instance in accordance with the target level of quality.

[0111] As will be appreciated, the target level of quality may be determined based on available streaming conditions, client device capabilities, user preferences, or the like.

[0112] As mentioned previously, in MPEG DASH, the client device may sense its available bandwidth, and accordingly parse a manifest file, and construct a locator (URL, for example) to obtain a representation from the ABR profile that is most appropriate given the available bandwidth. The manifest file may comprise metadata indicating the level of quality of each representation of the ABR profile. Therefore, when the client device is constructing the URL for the representation, it may also obtain metadata indicating that representation’s level of quality, and subsequently use this metadata to configure the enhancement decoder instance (generating a script for execution by the enhancement decoder instance or simply passing the metadata on to the enhancement decoder instance).

[0113] Alternatively or in addition, a user of the client device may provide inputs to the client device (via remote control, mouse and keyboard, video game controller, graphical user interface, touchscreen, or the like) specifying that they desired to watch the video at a preferred level of quality. In response, the client device may use this preferred level of quality to select the appropriate representation from the manifest file (if bandwidth permits) and also configure the enhancement decoder instance.

[0114] Hence more generally, in embodiments of the present invention, the method may optionally comprise the step of determining the target level of quality (at which the image frames are to be reconstructed) based on at least one of a streaming condition (such as available bandwidth), an amount of computational resource available for decoding (such as available processing power and / or memory capacity of a client device) and a user preference (such as the specifying of a preferred video quality level).

[0115] It will be appreciated that the above examples relate to a scenario where an (additional) enhancement stream / layer is retrieved (to increase the level of quality of the to-be-displayed image frames, for example). However, embodiments of the present invention may extend to other scenarios where the level of quality is to be maintained (in light of consistent streaming conditions, for example) and / or where the level of quality is to be reduced (in light of a deteriorating streaming conditions, for example).

[0116] In the case where the level of quality is to be maintained, that is, there is no difference between the current level of quality of the reconstructed image frames and the target level of quality for such frames, then no (additional) enhancement decoder instance needs to be configured — the decoder instance performing the reconstruction at the current level of quality (whether that be the base decoder instance or an enhancement decoder instance) is sufficient.

[0117] In the case where the level of quality is to be reduced, that is, the target level of quality is less than the current level of quality, then similarly no (additional) enhancement decoder instance needs to be configured — instead, the (additional) enhancement decoder instance performing the reconstruction at the current level of quality will become redundant, as the (additional) enhancement layer it was decoding will no longer be provided to it.

[0118] To determine whether an enhancement decoder instance is to be configured, it may be advantageous to firstly determine whether the target level of quality is higher than the current level of quality.

[0119] Hence more generally, in embodiments of the present invention, the method may optionally comprise the step of determining whether the target level of quality (at which the image frames are to be reconstructed) is higher than a current level of quality (at which the image frames are currently being reconstructed), wherein configuring step S600 may be carried out if the target level of quality is higher than the current level of quality.

[0120] In any case, once the enhancement decoder instance is configured with the target level of quality, the enhancement layer and the decoded “base layer” (in this case, the upstream reconstruction at a current level of quality) are input to the configured enhancement decoder instance in order for the enhancement decoder instance to perform the reconstruction.

[0121] Passing Data Identifying Target Image Frames

[0122] In embodiments of the present invention, the method comprises the step of passing, to the enhancement decoder instance, data identifying target image frames to be reconstructed, the target image frames having been reconstructed at a current level of quality by a previous decoder instance (step S602). It will be appreciated that “target image frames” here means a specified plurality of reconstructed image frames that require further reconstruction (via the enhancement decoder instance) to possess the target level of quality (as opposed to their current level of quality). This may be contrasted with the generalised “image frames” referred to previously when discussing the configuring step S600, which should be taken to mean that the target level of quality with which the enhancement decoder instance is configured is independent of the image frames input thereto for reconstruction.

[0123] In order for the enhancement decoder instance to reconstruct the target image frames at the target level of quality, the enhancement decoder instance must first obtain such target image frames.

[0124] To do so, a processor of the client device (such as a CPU or GPU) may provide the enhancement decoder instance with a pointer locating the target image frames in a buffer (associated with the previous decoder instance, for example). In this case, the enhancement decoder instance may pull the target image frames from the buffer identified in the pointer for subsequent reconstruction.

[0125] Alternatively or in addition, the enhancement decoder instance may be directly provided with the target image frames. For example, the target images frames output from the previous decoder instance may be rerouted (by a processor of the client device such as a CPU or GPU, for example) such that instead of being sent to a buffer / renderer, they are now being sent to the enhancement decoder instance.

[0126] Hence more generally, passing step S602 may comprise one or more of: passing, to the enhancement decoder instance, a pointer pointing to a buffer storing the target image frames; and sending the target image frames output from the previous decoder instance directly to the enhancement decoder instance.

[0127] In any case, and preferably, a CPU is used to perform passing step S602.

[0128] Optionally, in embodiments of the present invention, the method may comprise the step of instructing the previous decoder instance to reconstruct the target image frames at the current level of quality. It will be appreciated that this step is not essential for embodiments of the present invention — it may be that a different computer-implemented method is used to instruct the previous decoder instance, while the method of the present invention instructs the enhancement decoder instance to reconstruct target quality image frames by using the current quality image frames of the previous decoder instance.

[0129] However, it may be advantageous for the method of the present invention to cover the instruction of the previous decoder instance, as it would streamline the reconstruction of image frames — rather than the client device running respective different computer-implemented methods for each decoder instance (which would utilise a greater amount of the client device’s computational resources), a single computer-implemented method covering all decoder instances may be used.

[0130] In any case, and as mentioned previously, the previous decoder instance may comprise a base decoder instance. The term “base decoder instance” should be taken to mean any known and / or available decoder. Typically, client devices comprise such base decoder instances in the form of a hardware decoder.

[0131] Given that much of the video content on the internet is encoded using a Moving Picture Experts Group, MPEG, format (such as MPEG-4 Part 10, Advanced Video Coding — also known as MPEG-4 AVC or H.264), the base decoder instance is preferably for decoding image frames encoded in an MPEG, format. This way, the previously discussed advantages provided by the present invention may be applied broadly to internet video streaming, which is increasingly becoming a staple task for modem client devices. However, other types of base decoder instances are contemplated within the scope of the present invention.

[0132] In this case, the reconstruction of target image frames at the current quality level was being carried out using just a base stream, which means that the (configured) enhancement decoder instance is to decode a first enhancement layer and combine it with the decoded base stream (current quality reconstruction) in order to reconstruct the target image frames at the target quality. Alternatively or in addition, the previous decoder instance may comprise a previous enhancement decoder instance. In this case, the reconstruction of target image frames at the current quality level was being carried out using a base stream (decoded by the base decoder instance) and one or more enhancement streams (each of which decoded by a respective previous enhancement decoder instance). This means that the (configured) enhancement decoder instance is to decode a successive enhancement layer (dependent upon a reconstruction obtained using a previous enhancement layer) and combine it with the current quality reconstruction in order to reconstruct the target image frames at the target quality.

[0133] In this case, the enhancement decoder instance that is to decode the successive enhancement layer may be different from the previous decoder instance. For example, the (or a different) processor of the client device may execute software causing it to instantiate a separate enhancement decoder instance to the previous enhancement decoder instance, this separate enhancement decoder instance being configured with settings for reconstructing target quality image frames, with the previous enhancement decoder instance being configured with settings for reconstructing current quality image frames. For example, the client may instantiate, say, three different enhancement decoder instances, and each enhancement decoder instance may be configured to reconstruct image frames at a respective different level of quality. In this case, the reconstructed images of one enhancement decoder instance (having a current level of quality) may be input to a second (different) enhancement decoder instance for subsequent reconstruction at the target level of quality, and this target quality level image reconstruction may be input to a third enhancement decoder instance for subsequent reconstruction at a further target level of quality. As will be appreciated, this case may be more suitable for streaming applications, as the process of retrieving, decoding, reconstructing and displaying image frames is typically carried out in a “just-in-time” manner — buffering large numbers of previous reconstructed image frames ahead of a subsequent (further) reconstruction stage would likely disrupt this continuous stream of data from server to screen. Alternatively or in addition, the enhancement decoder instance that is to decode the successive enhancement layer may be same as the previous enhancement decoder instance. In this case, the previous enhancement decoder instance may be reconfigured with the settings for reconstruction at the target level of quality after having reconstructed some or all of the current quality image frames (which may be stored in a buffer during the reconfiguration). . This case may be more suitable for playing downloaded / stored content. For example, at the start of playback, the user may select a first quality representation, and so the enhancement decoder instance may reconstruct and buffer all image frames of the video at the first quality before playing the video. Should the user decide to increase the level of quality, then the enhancement decoder instance may be reconfigured with the increased level of quality, and subsequently reconstruct the buffered image frames at the higher level of quality (using a successive enhancement layer). As will be appreciated, this approach is more time consuming than the aforementioned use of multiple different enhancement decoder instances, and so would not be the ideal approach to take in a streaming scenario.

[0134] Regardless of the type of previous decoder instance employed, it will be appreciated that reconstructing the target image frames at the current level of quality may optionally involve the use of upscaling / upsampling after the decoding of the base stream and enhancement stream(s) (if any). Such upscaling / upsampling was discussed previously with reference to Figure 3.

[0135] In any case, once the enhancement decoder instance obtains the target image frames, then the appropriate enhancement layer may be passed to the enhancement decoder instance for decoding and combination with the target image frames to reconstruct them at the target level of quality.

[0136] Passing an Enhancement Layer

[0137] In embodiments of the present invention, the method comprises the step of passing, to the enhancement decoder instance, an enhancement layer defining, for each target image frame, a difference between the current level of quality of the target image frame and the target level of quality for the target image frame (step S604).

[0138] In order for the enhancement decoder instance to reconstruct the target image frames at the target level of quality, the enhancement decoder instance must obtain an enhancement layer defining the residuals / high-level details making up the difference between the current and target quality levels.

[0139] To do so, the enhancement decoder instance may be provided with a pointer locating the enhancement layer in a storage device (a buffer of the client device, for example). In this case, the enhancement decoder instance may pull the enhancement layer from the buffer identified in the pointer for subsequent reconstruction.

[0140] Alternatively or in addition, the enhancement decoder instance may be directly provided with the enhancement layer. For example, the enhancement layer retrieved from a server may be rerouted such that instead of being sent to a buffer, it is now being sent to the enhancement decoder instance.

[0141] Hence more generally, passing step S604 may comprise one or more of: passing, to the enhancement decoder instance, a pointer pointing to a storage device storing the enhancement layer; and sending the enhancement layer retrieved externally (that is, outside the enhancement decoder instance or client device) directly to the enhancement decoder instance.

[0142] Alternatively or in addition, in the case of ABR streaming, the client device may construct a locator (a URL, for example) for the enhancement layer by using a manifest file / playlist and the sensed streaming conditions, retrieve the enhancement layer identified by the locator from a server (over a network such as the internet, for example) and pass this retrieved enhancement layer to the enhancement decoder instance.

[0143] In embodiments where the client device uses a manifest file / playlist to retrieve enhancement layers, a more flexible retrieval of enhancement layers for reconstructing target quality image frames (without also retrieving a new base layer, for example) may be obtained by preferably modifying the storage, retrieval and transport of base and enhancement layers.

[0144] As alluded to previously, whereas each LCEVC-compliant representations of manifest file 205 of Figure 4 comprises a base layer and enhancement layer multiplexed / combined together (for example, representations b1e1 and b2e2), the present invention relies on the base layer and enhancement layers all being separated from each other (contained in separate mp4 files, for example). However, the base layer and enhancement layer(s) should be related in such a way that the client device may retrieve the base layer and enhancement layer(s) at separate times, yet combine a given current image frame with its corresponding residuals frame (of the decoded enhancement layer) to obtain target quality image frames. Put differently, given that the base and enhancement layers are no longer multiplexed before transport to the client device, the client device should preferably be given an indication as to which residuals frame is to be applied to which reconstructed current quality image frame.

[0145] Furthermore, as alluded to previously, the present invention also relies on “successive” enhancement layers, that is, enhancement layers which are dependent on a previous (lower quality) enhancement layer, rather than on a base layer. For example, the current quality image frames may have been reconstructed using an enhancement layer already. In this case, a further enhancement layer may be used for combination with the current quality image frames (acting as a decoded base layer in this case) to reconstruct the image frames at a target level of detail.

[0146] Implementing successive enhancements layers and the separated retrieval of base and enhancement layer(s) at the different points of the streaming pipeline shall now be discussed in turn below. While streaming applications are mainly discussed, it will be appreciated that the implementation of the present techniques at encoder and decoder sides is equally applicable to non-streaming use cases.

[0147] Turning now to Figure 7, encoder 500 may obtain a plurality of source videos. In this example, two source videos 502 and 524 are used. However, persons skilled in the art will appreciate that the following process may be extended to more than two source videos.

[0148] Features 500 to 520 of Figure 7 should be understood as corresponding, respectively, to features 100 to 120 of Figure 3. Therefore, to understand features 500 to 520 to Figure 7, see the discussion with reference to Figure 3 above.

[0149] To obtain successive enhancement layer(s) 534 (and 532), base stream 510 and enhancement layers(s) 514 (and 512) are used to reconstruct image frames 530 at the level of quality of source video 500 using a decoder (comprising respective base and enhancement decoder instances, for example).

[0150] Second source video 524 (which is of a higher level of quality than that of source video 500) may optionally be downsampled 526 to the same quality as that of source video 500 using a downsampler. First level corrections (residuals) 536 may be created by taking the difference between the down-sampled second source video 526 and the reconstructed image frames 530 (instead of a reconstruction of a different base layer). Applying an encoding operation to these first level correction 536 results in the obtaining of (optional) enhancement layer 532.

[0151] Top level corrections (residuals) 540 may be created by taking the difference between an up-sampled version (i.e. normative upsampling 538) of reconstructed image frames 530 (corrected using enhancement layer 536, for example) and second source video 524. Applying an encoding operation to these top level correction 540 results in the obtaining of enhancement layer 534.

[0152] Alternatively, second source video 524 may be downsampled to obtain source video 502. This is to say that a plurality of source videos may be obtained by downsampling an original source video. In this case, there may not be any first level corrections to be generated — reconstructed image frames 530 are a reconstruction of source video 502 (that is, downsampled second source video 524), and so taking the difference between reconstructed image frames and downsampled second source video 524 will be produce minimal (if any) first level corrections. In this case, it may be that only top level correction 540 are created and encoded into enhancement layer 534.

[0153] In any case, this enhancement encoding operation results in the generation of enhancement layer(s) 534 (and 532), which is / are dependent upon image frames that are reconstructed using enhancement layer(s) 514 (and 512), rather than a different base stream. Enhancement layer(s) 534 (and 532) may be called “successive” enhancement layer(s), as they depend on previous enhancement layers being used to reconstruct / enhance the image frames that they will further reconstruct / enhance.

[0154] The benefit of these successive enhancement layers is that higher quality representations may be retrieved without having to also retrieve a new base layer — the current and higher (target) quality representation both ultimately depend on the same quality base stream. This means that the fluctuation in computational burden placed on the client device may be reduced — changing levels of quality during streaming may be obtained solely by decoding different (numbers) of enhancement layers, which are less complicated and more efficient to decode than base layers.

[0155] To enable flexible retrieval of the enhancement layers 512, 514, 532 and 534 as and when needed, while also retrieving base stream 510 (upon which the enhancement layers are ultimately dependent for reconstructing image frames), the encoder multiplexer should preferably be modified.

[0156] In the example of Figure 3, base stream 110 and enhancement layers(s) 114 (and 112) are passed to Mux (multiplexer) 122, where they are multiplexed together into a single track of a single mp4 file, subsequently passed to a packager, which presents the single track mp4 file as one representation making up the manifest file. To provide each of the other representations for the manifest files, a respective set of base and enhancement layers is therefore encoded and multiplexed into a respective other single track mp4 file. Thus, when the client device retrieves a different representation, a different base layer is retrieved. As mentioned previously, the retrieval of different base layers during streaming results in a highly fluctuating computational burden being placed on the client device.

[0157] However, in the example of Figure 7, Mux 522 may place base stream 510 and enhancement layer 514, 534 (and 512, 532) into respective (that is, separate) mp4 files and / or respective tracks of mp4 files. For example, Mux 522 may create five mp4 files — one for each of base stream 510 and enhancement streams 512, 514, 532 and 534. Alternatively, Mux 522 may create three mp4 files — a first for base stream 510, a second for enhancement streams 512 and 514, and a third for enhancement streams 532 and 534, the second and third mp4 files respectively having two tracks, one for each of the two enhancement layers it contains, for example. As will be appreciated any combination of mp4 files and tracks may be used here. Hence more generally, mux 522 may be configured to insert a base layer and enhancement layers (suitable) for one or more representations into one or more container files (mp4 files, for example), wherein the mux is configured to each layer (either base or enhancement) into a different container file or different track of a given container file. That is, no two layers are placed into the same track of a container file.

[0158] As a result the packager may present base stream 510 and enhancement layers 512, 514, 532 and 534 as separate representations in a manifest file / playlist, thereby enabling the separate retrieval of base stream 510 and enhancement layers 512, 514, 532 and 534.

[0159] It will be appreciated that while the above examples relate to mp4 files, other container formats are contemplated within the scope of the present invention.

[0160] In any case, and more generally, in embodiments of the present invention, a method of instructing an enhancement encoder instance to encode image frames of a video comprises the steps of: passing, to the enhancement encoder instance, target image frames (530) that have been reconstructed at a current level of quality by a decoder instance; passing, to the enhancement encoder instance, a source video (524, 526) having a target level of quality; and instructing the enhancement encoder instance to: generate reconstruction data (536, 540) — residuals, high-level details — defining, for each target image frame, the difference between the current level of quality of the target image frame and the target level of quality for the target image frame, and encode the reconstruction data as an enhancement layer (532, 534).

[0161] Optionally, the decoder instance comprises a base decoder instance (506). In this case, the reconstructed current quality image frames that are to be passed to the enhancement encoder instance were obtained via decoding a base encoding (such as an MPEG encoded video).

[0162] Alternatively or in addition, the decoder instance comprises an enhancement decoder instance (516, 520). In this case, reconstructed current quality image frames that are to be passed to the enhancement encoder instance were obtained via decoding and combining a base encoding and one or more previous enhancement layers.

[0163] Figure 8 illustrates this separation of base and enhancement layers using the pipeline of Figure 4. As described here, delivering separate base and (successive) enhancement layers can be done using MP4 ISO BMFF carriage and MPEG- DASH delivery. Other mechanisms are considered within the scope of the present disclosure, but examples are presented using MPEG-DASH for brevity. In examples, modifications may be made at the encoder, packager and decoder to adapt them to the delivery method.

[0164] Source videos 502 and 524 are encoded using the encoder 500 to produce a base layer and a plurality of enhancement layers. Here we illustrate the use of LCEVC encoding the video in an mp4 file. Depending on the implementation of the LCEVC encoder, modifications may or may not be necessary. What is important is that that the (successive) enhancement layers and base layer can be separated, as mentioned previously.

[0165] An enhancement layer may be encoded “in loop” with the base layer, downscaling source video 502 before base encode and upscaling the reconstructed image to allow calculation of the EL residual information which is then encoded and transmitted. The LCEVC encoder will produce LCEVC NALUs according to the standard (ISO / IEC23094-2). LCEVC NAL unit types can have the following meaning: It is expected that for every I DR (that is, Instantaneous Decoder Refresh) picture in the base the corresponding LCEVC picture is also I DR. As will be appreciated, for successive enhancement layers, the same procedure may be carried out using second source video 524, albeit that image frames reconstructed using the previously obtained base layer and a previous enhancement layer is used in lieu of downscaling a source video to obtain a new base layer.

[0166] As mentioned previously, despite being separated, the base layer and enhancement layer(s) should be related in such a way that the client device may retrieve the base layer and enhancement layer(s) at separate times, yet combine a given current image frame with its corresponding residuals frame (of a decoded enhancement layer) to obtain target quality image frames. Put differently, given that the base and enhancement layers are no longer multiplexed into a single track before transport to the client device, the client device should preferably be given an indication as to which residuals frame is to be applied to which reconstructed current quality image frame.

[0167] Alternative approaches are contemplated to facilitate synchronisation of segments (and frames) of enhancement layer data and base layer data sent separately. Such approaches may be applied at the encoder or the packager. As will be appreciated, while the approaches below discuss the synchronising of base and enhancement layers, these approaches may similarly be applied to synchronising a successive enhancement layer with a previous enhancement layer.

[0168] For example, an Instantaneous Decoder Refresh picture of the base layer and enhancement layer should ideally be synchronised to facilitate combination of the residuals of the enhancement data. Techniques exist within known multi-layer codecs such as SVC and SHVC.

[0169] Frames of the base layer video and enhancement layer video may also be kept synchronised by techniques set out in WO2023 / 118851 which is incorporated herein by reference. This technique is relevant for HTML element implementations where the player may not have access to information in the base layer. It is described there how a packetised elementary stream (PES) may be constructed of a plurality of T ransport Stream (TS) packets. A presentation time stamp (PTS) is supplied in packets for the base stream. The enhancement stream uses a presentation time stamp in the enhancement stream to synchronise frames of the two streams during HTML reconstructions. In other words, the frames of the enhancement stream are indexed according to the PTS and this is compared with timing metadata such as a “media time” variable or a current playback time provided in the base layer to pair the frame of the enhancement layer with the frame of the base layer. In other words, in HTML5 the player may be provided with RGB data and timing metadata (but not timestamp information) which can be used to figure out timestamp information and combine as appropriate with the enhancement data.

[0170] Alternative approaches include the use of SEI messages to carry timing information which can then be used to synchronise enhancement and base. In order to preserve timing information and ensure synchronisation, an example of encapsulation of the LCEVC (by encoder mux 522, for example) within the M P4 file format is provided. The encapsulation of an LCEVC bitstream within the MP4 file format can be summarised in Figure 9.

[0171] The two main concepts used to link the LCEVC enhancement to its corresponding XVC base (where X stands for "whatever coding format", e.g. AVC, HEVC, EVC, WC) are:

[0172] (1) link the LCEVC enhancement to the XVC base by means of an "sbas" box identifying which is the corresponding base;

[0173] (2) use the same time base for the "stts" boxes in the enhancement, as they are used in the base, to perform picture to picture synchronization.

[0174] A process 800 is shown in Figure 9 in which an LCEVC NALU 802 is linked to the base NALU 804 using an «sbas>> box. THE LCEVC NALU 802 is turned into an LCEVC sample with an «stts> box the same as the <stts> time to sample as the Base sample 808 created from the Base NALU. These samples can then be combined into the LCEVC picture 810.

[0175] Example encoder implementations may be configured to return two separate outputs, one for the BL NALUs and one for the EL NALUs with the necessary synchronisation between BL and EL. The packaging into the carriage mechanism by the encoder mux (such as Mux 522) may then generate either: a single mp4 file with the BL and EL in two separate tracks, or two files, one with BL and one with EL, both in a single track.

[0176] In an example implementation, the former can be produced, i.e. a single mp4 with BL and EL in separate tracks. The separation of the tracks into different independent chunks may then be left to the packager.

[0177] Again, it should be noted that these approaches may also be extended to synchronising / aligning successive enhancement layers (such as layers 532 and 534) with previous enhancement layers (such as layers 512 and 514). Returning to Figure 8, a modified packager 702 is configured to separate the encoded base and enhancement layers for retrieval as a set of Segments and construct a manifest file comprising metadata suitable for constructing URLs to retrieve those segments. As usual, that metadata may comprise a modified URL or a template URL to be combined with information for each segment to construct a URL to retrieve that segment. The separated segments are then stored in storage 204 for later retrieval as usual. Each profile created by the packager may also comprise a suitable indication of average bandwidth and other available metadata usable by an ABR handler or ABR rules to identify and retrieve the appropriate segments.

[0178] In an implementation, the packager will generate segments that are represented by separate MP4 files (or a single file with byte ranges), those only containing BL and those containing EL as a single track representation within. The BL and EL are linked at an elementary stream by the encoder with the method described above. The packager optionally provides a way for the player to associate the BL MP4 chunks and the EL MP4 chunks. This can be done at manifest level. In the implementations described here, this association is using a dependency attribute in the manifest file.

[0179] In order to allow for smooth playback, the packager may signal chunks in the BL and EL at the same start and end times / frames. In certain implementations, this means that the packager will have to recognise the LCEVC I DR NALUs (nal_unit_type = 29) and cut the segment in correspondence of the equivalent AVC / HEVC / WC I DR.

[0180] Matching I DR frames is preferably ensured by the encoder, as explained above.

[0181] As explained elsewhere, a dependency attribute is one possible implementation used to create an association between the EL segments and the BL segments that they are enhancing, and create an association between the successive EL segments and the EL segments that they are enhancing. A dependency attribute may be set by the packager 702 in the manifest file 705. In the example shown in Figure 8, this is shown as ‘dID’. As is also shown in Figure 8, a base layer b1 is contained within a separate Adaptation Set to that of the enhancement layers e1-e4. However, it will be appreciated that a single Adaptation Set comprising base layer b1 and enhancement layers e1-e4 are also contemplated within the scope of the present invention — the use of dependency ID attributes within a single Adaptation Set has previously been used for purposes such as HDR. In any case, manifest file 705 is distinct from manifest file 205 of Figure 4 in that base layer b1 enhancement layers e1 -e4 of manifest file 705 are presented as separate representations, whereas each representation in manifest file 205 comprises a respective base layer in conjunction with an enhancement layer (typically multiplexed together by the Mux of encoder 100).

[0182] The ‘dID’ of manifest file 705 links the separated representations to each other. For example, the e1 representation is linked to (that is, made dependent upon) the b1 representation, the e2 representation is linked to the e1 representation, and so on, with each successive representation being linked to its immediate predecessor in a chain-link manner.

[0183] The bandwidth for a given representation in the LCEVC Adaptation Set (that is, a given one of representations e1-e4) may represent the bandwidth of the given representation in combination with the other representations on which it is dependent, or optionally the bandwidth of that given representation alone for the client device to combine it with the bandwidth requirements of representations on which the given representation depends in order to identify the total bandwidth. The latter may be preferable but that may depend on the standards and how they are designed.

[0184] The present disclosure presents the concept of separating the base and enhancement layers for storage and presentation and provides mechanisms to allow ABR logic to move between representations representing an encoded video stream using the combined base and enhancement layers as well as base encoded video alone. Use of a dependency ID attribute to link enhancement layer segments with respective base layer segments (or segments of a previous enhancement layer) is just one possible preferred implementation. In the below exemplary scenario illustrated, we expect one single Adaptation Set but with separate Representations. Other adaptations sets may of course be included in the MPD, but for the sake of explanation here, there are multiple representations within each Adaptation Set for each MIME type. Each Representation points to an MP4 Segment which contains a single track — for the BL only the legacy bitstream, e.g., AVC; for the EL only the enhancement encoded in LCEVC. The link from EL to BL (or previous EL) is external, from the EL MP4 to the BL MP4 (or previous EL MP4), and is provided using the ©Dependency! D attribute in MPEG-DASH.

[0185] This mechanism is described in DASH-IF Interoperability - Guidelines for Implementations, Annex G: Sample Dual-Layer MPD. The example reported in the guidelines is shown as follows:

[0186] <Period>

[0187] <!- Video ->

[0188] <AdaptationSet mimeType="video / mp4" subsegmentAlignment- 'true" subsegmentStartsWithSAP="1 " frameRate="24">

[0189] Representation id="base-layer" codecs="hev1.2.4.H153.bO" bandwidth="29200362" width="3840" height="2160">

[0190] <BaseURL>BL_Dashlf_BL-HDR10_UHD.mp4< / BaseURL> <SegmentBase indexRange="687-862"> clnitialization range="0-686" / > < / SegmentBase>

[0191] < / Representation>

[0192] Representation id="enhancement-layer" dependencyld- 'base-layer" codecs="dvhe.07.03" bandwidth="9320458" width="1920" height="1080"> <BaseURL>EL_Dashlf_BL-HDR10_UHD.mp4< / BaseURL> <SegmentBase indexRange="723-898"> clnitialization range="0-722" / > < / SegmentBase>

[0193] < / Representation> < / AdaptationSet>

[0194] < / Period>

[0195] < / MPD>

[0196] In the case of an LCEVC enhancement, the only attribute changing is the 4cc for codecs, which for LCEVC is “Ivc1”. A sample manifest for LCEVC is set out as follows:

[0197] <?xml version="1.0" encoding="utf-8"?>

[0198] <MPD xmlns:xsi=http: / / www.w3.org / 2001 / XMLSchema-instance xmlns="urn:mpeg:dash:schema:mpd:2011" xmlns:xlink=http: / / www.w3.org / 1999 / xlink xsi:schemaLocation="urn:mpeg:DASH:schema:MPD:2011 http: / / standards.iso.org / ittf / PubliclyAvailableStandards / MPEG-

[0199] DASH_schema_files / DASH-MPD.xsd" profiles="urn:mpeg:dash:profile:isoff-live:2011" type="static" mediaPresentationDuration="PT 10.0S" maxSegmentDuration="PT5.0S" minBufferTime="PT20.0S">

[0200] <Programlnformation>

[0201] < / Programlnformation>

[0202] <ServiceDescription id="0">

[0203] < / ServiceDescription>

[0204] <!- In this Period the LCEVC + Base stream is split into two representations: BL and EL ->

[0205] <!- In this example 3 representations are used to inform the player of the existence of 3x profiles, one of which is derived from the combination of base and enhancement, and another being derived from the combination of base and two successive enhancements ->

[0206] <!- the linking mechanism is dependency! D->

[0207] <Period id="O" start="PTO.OS"> <AdaptationSet id="0" contentType- 'video" startWithSAP="1" segmentAlignment- 'true" bitstreamSwitching- 'true" frameRate="25 / 1 " maxWidth- ' 1920" maxHeight="1080" par="16:9">

[0208] <!- Independent Representation (BL) of profile 0: 360p-> Representation id="0" mimeType="video / mp4" codecs="avc1 .64001 f" bandwidth="1100000" width="640" height="360" sar="1 :1"> <SegmentTemplate timescale="12800" initialization- 'init- stream$RepresentationlD$.m4s" media="chunk-stream$RepresentationlD$- $Number%05d$.m4s" startNumber="1 "> <SegmentTimeline>

[0209] <S t="1024" d="128000" / >

[0210] <S d="1024" / > < / SegmentTimeline> < / SegmentTemplate> < / Representation>

[0211] <!- Dependent Representation (EL) of profile 1 : combines with profile 0 to produce 720p->

[0212] Representation id="1" mimeType="video / mp4" codecs="lvc1" bandwidth="800000" width="1280" height="720" sar="1 :1" dependencyld="0"> <SegmentTemplate timescale="12800" initialization- 'init- stream$RepresentationlD$.m4s" media="chunk-stream$RepresentationlD$- $Number%05d$.m4s" startNumber="1 "> <SegmentTimeline>

[0213] <S t="1024" d="129024" / > < / SegmentTimeline> < / SegmentTemplate> < / Representation>

[0214] <!- Dependent Representation (EL) of profile 2: combines with profile 1 to produce 1080p-> Representation id="2" mimeType="video / mp4" codecs="lvc1" bandwidth="800000" width="1280" height="720" sar="1 :1" dependencyld="1">

[0215] <SegmentTemplate timescale="12800" initialization- 'init- stream$RepresentationlD$.m4s" media="chunk-stream$RepresentationlD$- $Number%05d$.m4s" startNumber="1 ">

[0216] <SegmentTimeline>

[0217] <S t="1024" d="129024" / >

[0218] < / SegmentTimeline>

[0219] < / SegmentTemplate> < / Representation>

[0220] < / AdaptationSet>

[0221] < / Period>

[0222] < / MPD>

[0223] The concept of Dependent Representation (which is the case for an LCEVC EL, depending on its corresponding BL), is introduced in 23009-1 , clause 5.3.5.1.

[0224] The following is a quote from 23009-1 , clause 5:

[0225] “Dependent Representations are described by a Representation element that contains a @dependencyld attribute. Dependent Representations are regular Representations except that they depend on a set of complementary Representations for decoding and / or presentation. The @dependencyld contains the values of the @id attribute of all the complementary Representations, i.e. Representations that are necessary to present and / or decode the media content components contained in this dependent Representation.”

[0226] @codingDependency is defined in 23009-1 , clause 5.3.7.2 (Semantics), and clause 5.3.7.3 (Syntax), as a common attribute for AdaptationSet and Representation.

[0227] Further relevant information can be found in 230090 - Dynamic adaptive streaming over HTTP (DASH), Part 3: Implementation guidelines, Section 10. Mapping of ISOBMFF’s track reference type to DASH MPD. The following is a quote from 23009-3, clause 10:

[0228] “Dependencies (See definition of “Dependent Representations” in Section 5.3.5.1 . of DASH Part-1):

[0229] For a given track reference type (one row of the Table), when the “Mapping to DASH” column of the Table indicates “Dependency”, it means that a Representation@dependencyld should be set from the Representation corresponding to the track declaring the track reference of the given type to Representation(s) corresponding to the track(s) having their track_ID listed in the ‘tref box. The list of Representation ids in the @dependencyld attribute shall match the track D order from the ‘tref’ box.”

[0230] In a further possible implementation detail, concerning the MIME types for the Base Layer and the Enhancement Layer, the Base Track will have the Sample Description Box according to the base codec (e.g. "avc1", "hvc1", "evs1", "vvc1"...) while the latter will have consistently its Sample Description Box (e.g. "Ivc1").

[0231] As stated above, the link between the EL and its corresponding BL may be implemented by means of the "sbas" box. Note in other embodiments set out herein, the current time from the media element is used to compare pts on enhancement side and so this example is not needed.

[0232] Returning to Figure 8, in the known manner, in order to retrieve a video stream, a client device will first retrieve the manifest file. In the known manner, a legacy device 210 of Figure 4, i.e. a non-LCEVC enabled device, will retrieve and parse the manifest file and will retrieve a representation according to the Adaptation Set which they support by constructing a URL to the base layer encoded video and downloading segments of the video to fill their available buffer according to present conditions.

[0233] A modified client device according to embodiments of the present invention will retrieve the manifest file and parse the manifest file to identify a representation of the enhancement layer and from the dependency attribute may identify an associated base layer (optionally via the dependency attribute of a previous enhancement layer on which the enhancement layer depends). In a first example, the client device will determine the appropriate representation from the enhancement Adaptation Set and will the retrieve the associated base layer segments (and segments of a previous enhancement layer) as appropriate. In a second example, the client device may build a set of virtual representations from the manifest file, the virtual representations indicating the combination of the base layer and enhancement layer(s) indicated by the dependency attribute.

[0234] References to client devices here may equally refer to a player or decoder. An example player to be modified is dash.js.

[0235] The client device will retrieve the base layer and enhancement layer segments and pass these to an appropriate enhancement decoder for decoding and combination for rendering by the client device.

[0236] In an example process, a client device referred to here as a decoder 708, will have to parse the manifest file and choose from a set of renditions based on the bandwidth it has available. It is expected for the player (i.e. the player on the client device or decoder such as dash.js) to build a combined virtual set of representations where an enhanced profile will have a bandwidth equal to the sum of EL and BL. The example process may include the following:

[0237] 1. The DASH manifest parser should parse the @DependencylD attribute to set up the dependency between BL and EL Representations.

[0238] 2. A DASH ABR engine may maintain its list of Representations considering BL as a Representation, EL togetherwith its dependency BL as another Representation. In the manifest example outlined above, the DASH ABR engine would maintain 3 Representations: a. Representation 1 with 360p BL only, with bandwidth=1100000 b. Representation 2 with 720p EL with its dependency 360p BL, with bandwidth=1900000, which is sum of 720p EL bandwidth(Representation id 2) and 360p BL bandwidth (Representation id 0). c. Representation 3 with 1080p EL with its dependency 720p EL, which in turn has its dependency 360p BL , with bandwidth=2300000, which is sum of 1080p EL bandwidth (Representation id 3), 720p EL bandwidth (Representation id 2) and 360p BL bandwidth (Representation id 1).

[0239] 3. When the DASH ABR engine detects enough bandwidth and codec ability (i.e. LCEVC support) to be able to choose the bitrate matching the bandwidth for the Representation 2: a. The DASH downloader may download the BL segment and its related EL segment. The downloader may make sure BL and EL segments are downloaded in asynchronous mode as much as possible so all required EL and BL data are available to be able to decode a full picture. In other words, the segments are to be downloaded together as opposed to synchronous download which would mean one after the other. Synchronous mode is also contemplated. b. The playback pipeline may feed the BL data to the base decoder through the available interface (e.g. mediacodec in Android) and pass the EL data (passing step S606, for example) to an LCEVC Decoder instance (configured in configuring step S600) to decode the full picture. Further details may be provided in WO2022 / 023739 which is incorporated by reference, which describes a Decoder Integration Layer (DIL).

[0240] 4. When the DASH ABR engine detects enough bandwidth and codec ability (i.e. LCEVC support) to be able to choose the bitrate matching the bandwidth for the Representation 3: a. The DASH downloader may download the BL segment and the segments of the related ELs. The downloader may make sure sure segments of the BL and ELs are downloaded in asynchronous mode as much as possible so all required EL and BL data are available to be able to decode a full picture. In other words, the segments are to be downloaded together as opposed to synchronous download which would mean one after the other. Synchronous mode is also contemplated. b. The playback pipeline may feed the BL data to the base decoder through the available interface (e.g. mediacodec in Android) and pass the two sets of EL data (passing step S604, for example) to respective LCEVC Decoder instances (configured in configuring step S600) to decode the full picture. As will be appreciated, in this instance, multiple LCEVC decoder instances are used to decode enhancement layers in a one-to-one correspondence. As will be appreciated, each LCEVC decoder instance may be instantiated via software configuration of the client device.

[0241] 5. If the ABR engine detects a bandwidth or a codec ability matching the BL stream only, only BL data may be downloaded and fed into base decoder as it is currently done in contemporary processes.

[0242] Due to computational resources available for decoding being finite, the total number of enhancement decoder instances being implemented by decoder 708 may be limited. For example, decoder 708 may only have enough computational resources to implement 4 different enhancement decoder instances in conjunction with the base decoder instance.

[0243] As will be appreciated, representation e4 of manifest file 705 would be the highest quality representation that is ultimately dependent on base layer b1. In order to provide higher quality representations than that of e4, a further base layer representation b2 may be provided, as well as further enhancement layer representations (not shown). For example, an enhancement layer e5 may be encoded and made dependent on b2, an enhancement layer e6 may be encoded and made dependent on e5, and so on.

[0244] The provision of multiple base layer representations in this way still provides the benefit of reducing the amount of fluctuation in computational burden placed on the decoder, as the frequency with which the base layer is changed during ABR streaming is reduced to compared to known techniques (shown in Figure 4).

[0245] In any case, it will be appreciated that in light of the above discussion, embodiments of the present invention may optionally comprise the following: Passing step S604 may optionally comprise: retrieving a manifest file (such as manifest file 705), the manifest file comprising metadata for constructing a locator to segments of the video, the metadata comprising: a representation pointing to the enhancement layer (representations e1-e4, for example); a previous representation pointing to a previous encoding of the video for reconstructing the target image frames at the current level of quality (representations b1 and e1 -e3, for example); and a dependency attribute associating the representation with the previous representation (‘dID’, as mentioned previously); parsing the manifest file to construct a locator for retrieval of the representation; and retrieving the representation (from storage 204, for example) for reconstructing the target image frames at the target level of quality.

[0246] Optionally, at least part of the previous encoding of the video was encoded using a base coding scheme. Optionally, the base coding scheme may be in a Moving Picture Experts Group, MPEG, format.

[0247] Optionally, at least part of the previous encoding of the video may comprise the previous enhancement layer.

[0248] Optionally, the method comprises the step of retrieving the previous representation (from the manifest file) for reconstructing the target image frames at the current level of quality.

[0249] Instructing Enhancement Decoder Instance

[0250] Once the enhancement decoder instance is configured, and the reconstructed current quality image frames and enhancement layer are passed to it, the enhancement decoder instance may be instructed decode the enhancement layer (to obtain the residuals to be applied to current quality image frames), and reconstruct the current quality image frames to the target quality using the residuals.

[0251] Thus, in embodiments of the present description, the method comprises the step of instructing (via software instruction, for example) the enhancement decoder instance to: decode the enhancement layer into reconstruction data (that is, residuals / high-level details obtained during the enhancement encoding operation), and reconstruct the target image frames at the target level of quality using the reconstruction data (step S606).

[0252] Accordingly, the enhancement decoder instance may carry out the instructions, and output reconstructed image frames at the target level of quality.

[0253] As will be appreciated, these reconstructed image frames may then be rendered, and subsequently displayed on a display screen associated with (that is comprised within or coupled to) the client device.

[0254] Hence optionally, embodiments of the present invention may comprise step of rendering, for display, the target image frames at the target level of quality. Further optionally, embodiments of the present invention may comprise step of displaying, on a display screen, the rendered image frames.

[0255] Computer Program, Storage Medium, Decoder and Encoder

[0256] As indicated above, concepts set out herein may be implemented at a client device, player on the device or decoder. Similarly, concepts may be embodied by modifications to an encoder, packager and / or content delivery network. At each of these entities, methods and processes described herein can be embodied as code (e.g., software code) and / or data. The functionality may be implemented in hardware or software as is well-known in the art of data compression and video streaming. For example, hardware acceleration using a specifically programmed Graphical Processing Unit (GPU) or a specifically designed Field Programmable Gate Array (FPGA) may provide certain efficiencies. For completeness, such code and data can be stored on one or more computer-readable media, which may include any device or medium that can store code and / or data for use by a computer system. When a computer system reads and executes the code and / or data stored on a computer-readable medium, the computer system performs the methods and processes embodied as data structures and code stored within the computer-readable storage medium. In certain embodiments, one or more of the steps of the methods and processes described herein can be performed by a processor (e.g., a processor of a computer system or data storage system).

[0257] Generally, any of the functionality described in this text or illustrated in the figures can be implemented using software, firmware (e.g., fixed logic circuitry), programmable or nonprogrammable hardware, or a combination of these implementations. The terms “component” or “function” as used herein generally represents software, firmware, hardware or a combination of these. For instance, in the case of a software implementation, the terms “component” or “function” may refer to program code that performs specified tasks when executed on a processing device or devices. The illustrated separation of components and functions into distinct units may reflect any actual or conceptual physical grouping and allocation of such software and / or hardware and tasks.

[0258] Hence more generally, embodiments of the present description further contemplate:

[0259] 1. a computer program comprising processor-implementable instructions which, when executed by a processor, cause the processor to perform embodiments of the above-described methods;

[0260] 2. a non-transitory computer-readable storage medium having stored thereon said computer program;

[0261] 3. a decoder comprising a processor and a memory, the memory having stored thereon processor-implementable instructions which, when executed by the processor, cause the processor to perform embodiments of the above-described method of instructing an enhancement decoder instance; and

[0262] 4. an encoder comprising a processor and a memory, the memory having stored thereon processor-implementable instructions which, when executed by the processor, cause the processor to perform embodiments of the above-described method of instructing an enhancement encoder instance. The foregoing discussion discloses and describes merely exemplary embodiments of the present invention. As will be understood by those skilled in the art, the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. Accordingly, the disclosure of the present invention is intended to be illustrative, but not limiting of the scope of the invention, as well as other claims. The disclosure, including any readily discernible variants of the teachings herein, defines, in part, the scope of the foregoing claim terminology such that no inventive subject matter is dedicated to the public.

Claims

CLAIMS1 . A method of instructing an enhancement decoder instance to reconstruct image frames of a video, comprising the steps of: configuring the enhancement decoder instance with a target level of quality at which the image frames are to be reconstructed; passing, to the enhancement decoder instance, data identifying target image frames to be reconstructed, the target image frames having been reconstructed at a current level of quality by a previous decoder instance; passing, to the enhancement decoder instance, an enhancement layer defining, for each target image frame, a difference between the current level of quality of the target image frame and the target level of quality for the target image frame; and instructing the enhancement decoder instance to: decode the enhancement layer into reconstruction data, and reconstruct the target image frames at the target level of quality using the reconstruction data.

2. A method according to claim 1 , wherein the configuring step comprises configuring the enhancement decoder instance to: receive data indicating the target level of quality, and specify or change one or more settings of the enhancement decoder instance in accordance with the target level of quality.

3. A method according to claim 1 or claim 2, comprising the step of determining the target level of quality based on at least one of a streaming condition, an amount of computational resources available for decoding, and a user preference.

4. A method according to any preceding claim, comprising the step of determining whether the target level of quality is higher than the current level of quality, wherein the configuring step is carried out if the target level of quality is higher than the current level of quality.

5. A method according to any preceding claim, wherein the step of passing, to the enhancement decoder instance, the data identifying the target image frames comprises one or more of: i. passing, to the enhancement decoder instance, a pointer pointing to a buffer storing the target image frames; and ii. sending the target image frames output from the previous decoder instance directly to the enhancement decoder instance.

6. A method according to any preceding claim, comprising the step of instructing the previous decoder instance to reconstruct the target image frames at the current level of quality.

7. A method according to any preceding claim, wherein the previous decoder instance comprises a base decoder instance.

8. A method according to claim 7, wherein the base decoder instance is for decoding image frames encoded in a Moving Picture Experts Group, MPEG, format.

9. A method according to any one of claims 1 to 8, wherein the previous decoder instance comprises a previous enhancement decoder instance.

10. A method according to any preceding claim, wherein the step of passing, to the enhancement decoder instance, the enhancement layer comprises one or more of: i. passing, to the enhancement decoder instance, a pointer pointing to a storage device storing the enhancement layer; and ii. sending the enhancement layer retrieved externally directly to the enhancement decoder instance.

11. A method according to any preceding claim, wherein the step of passing, to the enhancement decoder instance, the enhancement layer comprises: retrieving a manifest file, the manifest file comprising metadata for constructing a locator to segments of the video, the metadata comprising: a representation pointing to the enhancement layer;a previous representation pointing to a previous encoding of the video for reconstructing the target image frames at the current level of quality; and a dependency attribute associating the representation with the previous representation; parsing the manifest file to construct a locator for retrieval of the representation; and retrieving the representation for reconstructing the target image frames at the target level of quality.

12. A method according to claim 11 , wherein at least part of the previous encoding of the video was encoded using a base coding scheme.

13. A method according to claim 12, wherein the base coding scheme is in a Moving Picture Experts Group, MPEG, format.

14. A method according to any one of claims 11 to 13, wherein at least part of the previous encoding of the video comprises the previous enhancement layer.

15. A method according to any one of claims 11 to 14, comprising the step of retrieving the previous representation for reconstructing the target image frames at the current level of quality.

16. A method according to any preceding claim, comprising the step of rendering, for display, the target image frames at the target level of quality.

17. A method according to claim 16, comprising the step of displaying, on a display screen, the rendered image frames.

18. A method according to any preceding claim, wherein a given level of quality comprises one or more of: i. a pixel count; ii. a spatial resolution; iii. a temporal resolution; iv. a bit depth; and v. a colour space.

19. A method of instructing an enhancement encoder instance to encode image frames of a video, comprising the steps of: passing, to the enhancement encoder instance, target image frames that have been reconstructed at a current level of quality by a decoder instance; passing, to the enhancement encoder instance, a source video having a target level of quality; and instructing the enhancement encoder instance to: generate reconstruction data defining, for each target image frame, the difference between the current level of quality of the target image frame and the target level of quality for the target image frame, and encode the reconstruction data as an enhancement layer.

20. A method according to claim 19, wherein the decoder instance comprises a base decoder instance.

21. A method according to claims 19 to 20, wherein the decoder instance comprises an enhancement decoder instance.

22. A computer program comprising processor-implementable instructions which, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 18, or any one of claims 19 to 21 .

23. A non-transitory computer-readable storage medium having stored thereon the computer program of claim 23.

24. A decoder, comprising a processor and a memory, the memory having stored thereon processor-implementable instructions which, when executed by the processor, cause the processor to perform the method of any one of claims 1 to 18.

25. An encoder, comprising a processor and a memory, the memory having stored thereon processor-implementable instructions which, when executed by the processor, cause the processor to perform the method of any one of claims 19 to 21.

Images