Video rate control based on user input at client

Abstract

A method for encoding pictures. The method includes obtaining user control input data (UCID). The method also includes obtaining a current picture generated by a first application. The method also includes encoding the current picture using the obtained UCID and / or data derived using the obtained UCID, thereby producing an encoded current picture. The method also includes outputting the encoded current picture.

Description

P112310W001 1VIDEO RATE CONTROLTECHNICAL FIELD

[0001] Disclosed are embodiments related to video coding.BACKGROUND

[0002] 1. Video Coding

[0003] A video consists of a series of pictures where each picture consists of one or more components. A picture of a video is sometimes denoted ‘image’ or ‘frame’. Each component in a picture can be described as a two-dimensional rectangular array of sample values (or “samples” for short). It is common that a picture in a video sequence consists of three components; one luma component Y where the sample values are luma values and two chroma components Cb and Cr, where the sample values are chroma values. Other common representations include ICtCb, IPT, constant-luminance YCbCr, YCoCg and others.

[0004] Before a video is transmitted over a network it is compressed (a.k.a., encoded) into a coded video bitstream (or bitstream for short) using a video encoder of a video codec (a.k.a., coder / decoder). The bitstream is then decompressed (a.k.a., decoded) on the receiving side using a video decoder of the video codec before further processing of the video and / or display of the pictures of the video. Examples of widely deployed video codecs include H.264 (a.k.a., Advanced Video Coding (AVC)), H.265 (a.k.a., High Efficiency Video Coding (HEVC), and H.266 (a.k.a., Versatile Video Coding (VVC)), all of which were developed and standardized jointly by MPEG and ITU-T. MPEG and ITU-T are currently working towards a new video coding standard, likely to be called H.267, within the Joint Video Experts Team (JVET). Other widely spread video codecs include VP9 and AVI.

[0005] A video codec typically supports both intra-coded and inter-coded pictures. For example, VVC and its predecessors, AVC and HEVC, are block-based video codecs that utilize both temporal and spatial prediction. Spatial prediction is achieved using intra (I) prediction from within the current picture. Temporal prediction is achieved using unidirectional (P) or bi-directional inter (B) prediction on the block level from previously decoded reference pictures.P112310W001 2

[0006] An intra-coded picture may only predict from samples of the same picture, whereas inter-coded pictures may also predict from previously decoded pictures, referred to as reference pictures. Inter-coded pictures may be divided into P-pictures, which may only predict from one reference picture at a time for each coding block and bidirectional B-pictures, which may predict from up to two reference pictures simultaneously for each coding block.

[0007] In the above codecs, the difference between the original pixel data and the predicted pixel data, referred to as the residual, is transformed into the frequency domain at the encoder side, then quantized and entropy coded before transmitted together with necessary prediction parameters such as prediction mode and motion vectors, which are also entropy coded. The level of quantization is controlled with a quantization parameter (QP) value. The decoder performs the reverse process, i.e., applying inverse quantization and inverse transformation to obtain the residual, and then adds the residual to an intra or inter prediction to reconstruct a picture.

[0008] Pictures in AVC, HEVC and WC are identified by their picture order count (POC) values. POC determines the output order of decoded pictures. A picture in AVC, HEVC and VVC is divided into one or more slice network abstraction layer (NAL) units for easier packetization into network packets, where a slice may be a full picture or a part of a picture.

[0009] a. Random access point pictures

[0010] In AVC, HEVC, WC, and most other modern video codecs random access point pictures (a.k.a., key pictures) may be used in a coded video bitstream typically for three reasons; as a start of the bitstream, to refresh the video, or to let a decoder tune into the bitstream, e.g. when switching channel in broadcasted TV or when moving to a different part of the video in video-on-demand (VOD).

[0011] Random access point pictures are typically intra coded, which means the picture doesn’t predict from other pictures than itself, resulting in a much lower compression efficiency compared to if inter-picture prediction was allowed. Intra random access point (IRAP) pictures therefore tend to produce bitrate spikes in the bitstream, resulting in increased jitter and overall latency. WC support two types of IRAP pictures; instantaneous decoding refresh (IDR) pictures and clean random access (CRA) pictures, where the IDR picture ensures that all pictures following the IDR picture in decoding order in the bitstream can be decoded correctly. CRAP112310W001 3pictures may contain leading pictures, i.e., pictures that follow the CRA picture in decoding order but precedes the CRA picture in output order. These leading pictures may need to be dropped in case the decoding is starting at the CRA picture.

[0012] b. Reference picture resampling (RPR)

[0013] A new feature in WC, which is not supported in AVC or HEVC, is reference picture resampling (RPR). RPR allows the spatial resolution to change in the middle of the bitstream without the need to code an IRAP picture, something that is not allowed for AVC or HEVC.

[0014] In WC, the spatial resolution (width and height) of a picture may be modified on a picture level and is therefore signaled in a picture parameter set (PPS), in comparison to AVC and HEVC where it is signaled on the sequence level in the sequence parameter set (SPS).

[0015] When the current picture and a reference picture have different spatial resolutions, RPR enables the reference picture to be used for prediction of the current picture by scaling the reference picture to the same spatial resolution as the current picture before prediction. This scaling is done on the block level.

[0016] When the term RPR is used in this disclosure it is not necessarily limited to RPR of WC but may be interpreted in a more generic usage of reference picture resampling including other codecs, such as AVI, and future video codecs not yet developed at the time of writing.

[0017] c. Subpictures

[0018] WC supports the concept of subpictures where a subpicture is defined as a rectangular region of one or more slices within a picture. This means a subpicture contains one or more slices that collectively cover a rectangular region of a picture. Each subpicture is encoded and decoded independently from each other, both spatially and temporally, meaning a block in a first subpicture may not reference any block in another subpicture of the current picture or any non-collocated subpictures of any previously coded pictures.

[0019] Bitstream extraction and merge operations are supported through subpictures in WC and could for instance comprise extracting one or more subpictures from a first bitstream,P112310W001 4extracting one or more subpictures from a second bitstream and merging the extracted subpictures into a new third bitstream.

[0020] d. Screen content coding (SCC)

[0021] Both HEVC and WC include specific coding tools for more efficient coding of computer-generated or screen content, as occurs especially in computer screen sharing and gaming. These tools include transform skip residual coding, intra block copy, adaptive color transform, and the palette mode. For HEVC, the screen content coding tools are supported in the Screen-Extended Main 10 profile, and for WC screen content coding tools are supported directly in the Main 10 profile.

[0022] 2. Auxiliary data for video compression of gaming content

[0023] JVET currently has an ongoing exploratory activity on coding of gaming content, where the draft common test conditions (CTC) for this activity is captured in JVET-AI2027. In this work, auxiliary data may be used as input to the encoding in addition to the source sequence. The auxiliary data comprises depth images and camera parameters used in the rendering, which is something that can be output by a game engine and made available to the encoder during compression of gaming content.

[0024] 3. Low latency video coding and streaming

[0025] Low latency video coding is a key requirement for many applications of video, including conversational services, cloud gaming, and services for extended reality (XR) (e.g., virtual reality (VR), augmented reality (AR) and mixed reality (MR)). Factors that affect the latency of a coded video bitstream include: a) reference pictures, b) rate control and buffer management, c) transport protocols, d) congestion control, and e) resolution change,

[0026] a. Reference pictures

[0027] Video for broadcast TV and video on demand (VOD) streaming is typically encoded using a group of pictures (GOP) structure where pictures are referencing both past and future pictures to achieve a good prediction with high-level of compression, often using bidirectional predicted (B) pictures that predicts from two reference pictures at a time. To achieve low latency, it is not recommended to reference future pictures as this builds up latency.P112310W001 5Furthermore, in professional encoders for broadcasting and VOD it is common with a look-ahead of the coming pictures. This is also not recommended as it increases latency significantly.

[0028] b. Rate control and buffer management

[0029] The number of bits needed to compress video pictures highly depends on the content of the video. Low motion scenes with little detail are relatively easy to compress, while scenes with lots of motion and detailed structures are more complex and require much more bits. To keep a constant quality of the video a variable bitrate (VBR) scheme is typically used in broadcasting and VOD services. To handle variable bitrate the encoding and decoding buffers must be sufficiently large which adds to the latency. To achieve lower latency with smaller buffer sizes the video can be encoded with constant bitrate (CBR). CBR has a lower compression efficiency than VBR, and, therefore, is recommended for real-time video services. The rate is controlled with a rate control algorithm that typically is implemented by a rate controller in the encoder.

[0030] From a configured target bitrate, a rate controller often calculates a bit budget to spend on a certain GOP, picture, or encoding blocks. Overshooting the bit budget may lead to a lower budget for following picture(s) or blocks.

[0031] Rate controllers typically modify the QP value per picture, per slice or per block to achieve desired bitrates. Alternatively, or additionally, the lambda value can be changed of the Lagrange cost function in the encoder rate-distortion optimizations.

[0032] c. Transport protocols for low latency video streaming

[0033] When streaming video with low latency, it is important to use a transport protocol that meets the latency requirements. A popular transport protocol for achieving latencies below 500ms is the Real-time Transport Protocol (RTP) (see, e.g., reference [2]), which is typically run over the User Datagram Protocol (UDP). The Transmission Control Protocol (TCP) is normally not recommended as it requires retransmission of lost packets, which rapidly increases the latency. AVC, HEVC, WC all have RTP payload formats to properly pack the bitstream in RTP.

[0034] d. Congestion controlP112310W001 6

[0035] A congestion control algorithm is helpful for achieving low latency when network conditions vary greatly. Congestion occurs when the transmitted bitrate is higher than the available capacity over a given transmission path. Applications for video streaming may employ congestion control to achieve robust performance and to avoid congestion collapse. The congestion control algorithm may adapt the bitrate to follow a target bitrate by, for example, increasing QP values, adjusting a lambda value, lowering a frame rate and / or changing the resolution of the encoded stream.

[0036] e. Resolution change

[0037] Typical VOD services use adaptive bitrate (ABR) streaming to control the quality delivered to an end-user while mitigating effects of a varying bandwidth in transmission channel. In ABR streaming, several renditions (a.k.a., representations) of the same content are produced on server side (network notes, CDNs) and the renditions differ in terms of codec resolution and bitrates. In such a case, the decision on which resolution is fetched for playback depends on the client device. In other cases, such as in broadcast where content is encoded at a chosen resolution encoder needs to make decision about selecting the resolution that is optimal for coding performance. The same is for low latency applications that use 1-to-l distribution model, i.e., where content served to user is specific to that user and is not redistributed to other users.Example of this is cloud gaming content where each session will differ depending on the interactions from the user.

[0038] In all these video distribution cases, where video resolution changes based on available bandwidth (either by client device or by encoding device) video can be seamlessly decoded on the client device if resolution changes in the resulting bitstream do not impact decoding process. In most cases, it requires insertion of an intra frame to signal to decoder that subsequent pictures are coded at a new resolution. The WC RPR feature, however, allows resolution change without insertion of an intra frame.

[0039] 4. Signalling of game control data

[0040] Signalling of user-controlled application data for cloud gaming is commonly done using proprietary solutions on the application level and typically packaged and sent as payload within the UDP protocol. IETF is working on an RTP payload format for sending game moves and state of game objects over RTP (see, e.g., reference [3]).P112310W001 7SUMMARY

[0041] Certain challenges presently exist. For instance, bit allocation in conventional rate controllers is often done in using a look-ahead scheme that looks ahead several future pictures, and this look ahead scheme introduces significant delay and is sub-optimal for live video streaming and other video services relying on having very low latency of the transmitted bitstream. In some cases, the rate controller employs full encoding of a certain number of future pictures (e.g. “2-pass encoding”) or different metrics calculating the complexity of the pictures and estimating the required bit allocation.

[0042] For latency sensitive video services using encoders specifically configured for low-delay, the rate controller of such an encoder might only have time to process the current frame to estimate the bit allocation. This estimation may then have very limited accuracy compared to look-ahead schemes. Inaccurate estimations may lead to overshooting or undershooting bit target which could result in exceeding buffers or decreased compression efficiency. Some low latency rate controllers rely solely on reacting to bitrate fluctuations leaving the coding very vulnerable to drastic changes in the content of the input video.

[0043] Accordingly, in one aspect there is provided a method for encoding pictures. The method includes obtaining user control input data (UCID). The method also includes obtaining a current picture generated by a first application. The method also includes encoding the current picture using the obtained UCID and / or data derived using the obtained UCID, thereby producing an encoded current picture. The method also includes outputting the encoded current picture.

[0044] In another aspect there is provided a method performed by a client application. The method includes determining a state of the client application. The method also includes sending an indication of the determined state of the client application to a server application, wherein the indication of the state is used by an encoder to encode a picture generated by the server application.

[0045] In another aspect there is provided a computer program comprising instructions which when executed by processing circuitry of an apparatus causes the apparatus to perform any of the methods disclosed herein. In one embodiment, there is provided a carrier containing the computer program wherein the carrier is one of an electronic signal, an optical signal, a radioP112310W001 8signal, and a computer readable storage medium. In another aspect there is provided an apparatus that is configured to perform the methods disclosed herein. The apparatus may include memory and processing circuitry coupled to the memory.

[0046] An advantage of the embodiments disclosed herein is that they reduce bitrate spikes, which has several benefits, including reduced stress on the delivery system and increased compression efficiency by spending bits where they are best for future referencing.BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate various embodiments.

[0048] FIG. 1 illustrates a system according to an embodiment.

[0049] FIG. 2 illustrates functional components of an encoder.

[0050] FIG. 3 illustrates functional components of a decoder.

[0051] FIG. 4 is a flowchart illustrating a process according to an embodiment.

[0052] FIG. 5 is a flowchart illustrating a process according to an embodiment.

[0053] FIG. 6 is a block diagram of an apparatus according to some embodiments.

[0054] FIG. 7 illustrates data showing an increase in compression quality.DETAILED DESCRIPTION

[0055] FIG. 1 illustrates a system 100 according to one embodiment. System 100 includes a video encoder 102 (or encoder 102 for short) and a video decoder 104 (or decoder 104 for short). Encoder 102 may be a video encoder for the HEVC, WC, AVC, or AVI video codecs or any other supported video codec.

[0056] In the embodiment shown in FIG. 1, encoder 102 is in communication with decoder 104 via a network 110 (e.g., the Internet and / or other network). Encoder 102 encodes source video 163 into a coded video bitstream 190 (or bitstream 190 for short) comprising an encoded video sequence and may transmit bitstream 190 to decoder 104 via network 110. In this example embodiment, encoder 102 is part of an apparatus 130 (e.g., an application server) thatP112310W001 9includes an application 160 that produces source video 163 (e.g., pictures rendered by application 160).

[0057] Decoder 104 decodes the encoded pictures included in the bitstream to produce video data (e.g., reconstructed pictures) for display and / or post processing (e.g. a machine vision task, such as generating a 3D of an object-of-interest (OOI) in the pictures of the video sequence). Accordingly, decoder 104 may be part of a user apparatus 103 having an image processor 105 (e.g., a postfilter or other image processor) and / or a display 106. The image processor 105 may perform machine vision tasks on the decoded pictures. One such machine vision task may be identifying an object in the picture and creating a 3D model of the object. The image processor 105 may also comprise or consist of a postfilter, such as a neural network postfilter (NNPF), that may receive reconstructed pictures from decoder 104 and that may process the reconstructed pictures using a specified postfilter. In the embodiment shown, processor 105 is separate from decoder 104, but in other embodiments, processor 105 may be a component of decoder 104. The user apparatus 103 may be a mobile device, a set-top device, a head-mounted display, or any other device. Additionally, as shown in FIG. 1, in this example, apparatus 103 includes an application (app) 170 that is operable to communicate with application 160 via network 110.

[0058] FIG. 2 illustrates functional components of encoder 102 according to some embodiments. It should be noted that encoders may be implemented differently so implementation other than this specific example can be used. Encoder 102 employs a subtractor 241 to produce a residual block which is the difference in sample values between an input block and a prediction block (i.e., the output of a selector 251, which is either an inter prediction block output by an inter predictor 250 (a.k.a., motion compensator) or an intra prediction block output by an intra predictor 249). In the intra predictor 249, the inter predictor 250, and the selector 251 an prediction mode for each block is chosen with lowest cost where the cost function is a weighted sum of distortion and bits necessary to encode the block, the weighting factor in the cost function is called the lambda value. Then a forward transform 242 is performed on the residual block to produce a transformed block comprising transform coefficients. A quantization unit 243 quantizes the transform coefficients based on a quantization parameter (QP) value (e.g., a QP value obtained based on a picture QP value for the picture in which the input block is a part and a block specific QP offset value for the input block), thereby producing quantized transformP112310W001 10coefficients which are then encoded into the bitstream by encoder 244 (e.g., an entropy encoder) and the bitstream with the encoded transform coefficients is output from encoder 102. Next, encoder 102 uses the quantized transform coefficients to produce a reconstructed block. This is done by first applying inverse quantization 245 and inverse transform 246 to the transform coefficients to produce a reconstructed residual block and using an adder 247 to add the prediction block to the reconstructed residual block, thereby producing the reconstructed block, which is stored in the reconstruction picture buffer (RPB) 266. Loop filtering by a loop filter (LF) stage 267 is applied and the final decoded picture is stored in a decoded picture buffer (DPB) 268, where it can then be used by the inter predictor 250 to produce an inter prediction block for the next picture to be processed. LF stage 267 may include three sub-stages: i) a deblocking filter, ii) a sample adaptive offset (SAO) filter, and iii) an Adaptive Loop Filter (ALF).

[0059] FIG. 3 illustrates functional components of decoder 104 according to some embodiments. It should be noted that decoder 104 may be implemented differently so implementations other than this specific example can be used. Decoder 104 includes a decoder module 361 (e.g., an entropy decoder) that decodes from the bitstream quantized transform coefficient values of a block. Decoder 104 also includes a reconstruction stage 398 in which the quantized transform coefficient values are subject to an inverse quantization process 362 and inverse transform process 363 to produce a residual block. This residual block is input to adder 364 that adds the residual block and a prediction block output from selector 390 to form a reconstructed block. Selector 390 either selects to output an inter prediction block or an intra prediction block. The reconstructed block is stored in a RPB 365. The inter prediction block is generated by the inter prediction module 350 and the intra prediction block is generated by the intra prediction module 369. Following the reconstruction stage 398, a loop filter stage 367 applies one or more filters to the reconstructed blocks and the final decoded picture may be stored in a decoded picture buffer (DPB) 368 and output to image processor 105 for post processing and / or display 106. Pictures are stored in the DPB for two primary reasons: 1) to wait for picture output and 2) if the picture is a reference picture, to be used for reference when decoding future pictures. In some embodiments, image processor 105 may receive the reconstructed picture before all loop filters are applied.P112310W001 11

[0060] As described above, a challenge presently exists because, as noted above, the bit allocation schemes used in conventional rate controllers may introduce significant delay or decrease compression efficiency.

[0061] Accordingly, to overcome these deficiencies, this disclosure describes, among other things, encoder 102 using user control input data (UCID) 161 from application 160 to adapt the encoding of the pictures produced by application 160, such as, for example, adjusting the rate control algorithm employed by the encoder and / or parameters used in the rate control process. UCID 161 may, in some embodiments, comprise UCID 171 provided by apparatus 103 and / or data derived using the UCID 171 provided by apparatus 103. In certain applications, the UCID 161, 171 may provide information indicating an expected content complexity of future pictures to be produced by application 160, and encoder 102 may then for instance, based on the information, increase its QP value or lambda or other encoding parameter to compensate for any expected complexity increase at the start of encoding a current picture and thus reduce or avoid bitrate spikes.

[0062] The UCID 161, 171 may comprise data indicating a user action with respect to the video generated by decoder 104. For example, in the case where application 160 is an XR application or a game, the user action may be movement within the XR environment or within the game such as, for example, the user walking, rotating, change viewing angle etc. The user action may also be for example changing slide in a streamed presentation. The UCID 161, 171 may also comprise timing between user actions. A user action after a period of passivity may trigger the rate control offset fully or weighted with the passed passive time.

[0063] How encoder 102 uses the UCID 161, 171 could be predetermined and / or adaptively adjusted during a session depending on the type of application, the type of UCID and the type of content of the video being encoded.

[0064] Additional Details of Various Embodiments

[0065] As described above, in one embodiment, UCID 161 (a.k.a., metadata) is passed from the application 160 to encoder 102 and encoder 102 uses input data 161 to make one encoding decisions when encoding pictures rendered by application 160 and provided to encoder 102. The pictures produced by application 106 are known as the “application viewport.”P112310W001 12

[0066] In one version, as illustrated in FIG. 1, application 160 may be in communication with application 170. Application 170 may obtain UCID 171 through a user interface, e.g. connected to a keyboard, mouse, touchpad, joystick, game pad, head mounted display (HMD) or mobile phone touchpad or buttons or other types of sensors, and the UCID 171 is sent from application 170 to application 160.

[0067] The UCID 171 may be used to control application 160, e.g. to control a game in a game engine. The UCID 171 may be passed from application 160 to encoder 102 in the form of UCID 161. That is, input data 161 may comprise input data 171 and / or data derived from input data 171. However, in an another embodiment, application 160 sends the input data 171 directly to encoder 102. In such an embodiment, application 160 may be component of encoder 102 or encoder 102 may be a component of application 160. The input data 161 and / or input data 171 is used by the encoder to encode the rendered pictures from the application 160 to an encoded video bitstream 190 which is sent to the user apparatus 103.

[0068] In one embodiment, application 160 and application 170 are components of a cloud gaming service. In this embodiment, application 160 is a game engine and / or game server that controls the game and renders the game to an output video stream 163. The rendered output video stream 163 produced by application 160 is encoded by the encoder 102 and sent to the cloud game application 170. The UCID 171 may include commands for moving, viewing or carrying out tasks in the game, e.g., move forward, move back, move left, move right, view down, view up, view left, view right, jump, aim, and shoot, or commands for starting, stopping the game or accessing certain menus.

[0069] In another embodiment, application 160 and application 170 are components of a video call or video conference service, where application 160 may be used to mix video streams from multiple participants, screen sharing from the participants, or other online collaboration tools, such as white boards, etc. The UCID 171 may in this version include indications of participants entering or leaving a conference meeting, indications of participants becoming active or passive in a call, e.g. by toggling a mute button, gyroscopic sensor data from a phone being used for a call, mouse movements or keyboard strokes in screen sharing, zoom in / out commands, etc.P112310W001 13

[0070] In another embodiment, application 160 and application 170 are components of an XR service that is split rendered in the application 160. This may be an extension of the cloud gaming service mentioned above where the cloud game may be an XR game rendered in the cloud game application server 160, or an extension to the video call or video conference service, e.g. with an AR video call with mixed reality. Here the UCID 171 may additionally come from an HMD (change of movements, viewing direction, etc), an XR controller (aiming, shooting, menus, etc), eye-tracking, etc.

[0071] In another embodiment, application 160 and application 170 are components of a peer-to-peer application where two or more entities communicate with each other without the involvement of a server. In this version, similar to the server-client application versions, at least a first apparatus comprises the encoder and uses the UCID as input to the encoder for encoding application rendered pictures. The UCID may be obtained from the at least first entity, or from another entity in the peer-to-peer application framework.

[0072] FIG. 4 is a flowchart illustrating a process 400 for encoding a picture. Process 400 may begin in step s402.

[0073] Step s402 comprises obtaining UCID 161 and / or 171.

[0074] Step s404 comprises obtaining the picture. The picture may be rendered from application 160, e.g. a game engine of a cloud gaming application. Application 160 may produce the picture using the obtained UCID.

[0075] Step s406 comprises encoding the picture using the obtained UCID, thereby producing an encoded picture.

[0076] Step s408 comprises outputting the encoded picture. For example, the encoded picture may be provided to a transmitter for transmitting the encoded picture to an apparatus comprising a decoder for decoding the encoded picture.

[0077] In some embodiments, outputting the encoded picture comprises transmitting the encoded picture to a user apparatus or providing the encoded picture to a transmitter for transmitting the encoded picture to the user apparatus, obtaining the user control input data (UCID) comprises receiving the UCID from the user apparatus or from the first application.P112310W001 14

[0078] In some embodiments, the user apparatus comprises a user interface, and the user apparatus obtained the user control input data through the user interface.

[0079] In some embodiments, an encoder performs the steps of obtaining the current picture and encoding the current picture, the encoder is a component of an application server further comprising the first application, the first application obtains the UCID from a second application and generates the current picture using the UCID, the second application is a component of the user apparatus, and the first application provides to the encoder the UCID and / or data derived using the UCID.

[0080] In some embodiments, the first application is a component of a cloud gaming service, the first application is a component of a video call or video conference service, the first application is a component of a split rendered XR service, or the first application is a component of a peer-to-peer application

[0081] In some embodiments, the application server is a cloud gaming server, and the user apparatus is a cloud gaming client.

[0082] In some embodiments, the UCID includes at least: a movement command (e.g., move in an indicated direction), a change view command (e.g., tilt, view up, left, right, etc.; zoom in; zoom out), a task command indicating a task to be carried (e.g., jump, aim, and shoot), a start command for starting an application, a pause command pausing the application, a resume command for un-pausing the application, a stop command for stopping the application, a menu access command for accessing an indicated menu or menu item, information indicating a participant entering or leaving a conference meeting, information indicating a participant of the conference meeting becoming active or passive (e.g. indicating the participant activated or deactivated a mute button), gyroscopic sensor data from a phone being used for a call, information indicating a mouse movement, and / or information indicating a keyboard stroke.

[0083] In some embodiments, encoding the current picture using the obtained UCID and / or data derived using the obtained UCID includes at least making a first encoding decision using the obtained UCID and / or data derived using the obtained UCID.P112310W001 15

[0084] In some embodiments, the first encoding decision is a decision for setting a value for at least a first rate control variable, or the first encoding decision is a decision for setting a value for a variable that affects (directly or indirectly) the first rate control variable.

[0085] In some embodiments, encoding the current picture using the obtained UCID and / or data derived using the obtained UCID further comprises, based on the first encoding decision that was made using the UCID and / or the data derived using the obtained UCID (the “derived data” for short), setting a value for at least a first rate control variable or setting a value for a variable that affects (directly or indirectly) the first rate control variable.

[0086] In some embodiments, the first rate control variable is: a quantization parameter, QP (e.g., picture QP, slice QP, block QP), a lambda parameter, a bit budget, a frame rate parameter, a resolution parameter, a target bitrate parameter.

[0087] In some embodiments, the first encoding decision is a decision whether or not to: change a quantization parameter, change a lambda parameter, activate or deactivate a first coding tool, change a frame rate, skip coding of a picture (i.e. drop the picture), skip coding of residuals for a picture, change a picture or a slice type (I, P or B), change a resolution (e.g., either by inserting a new random access point picture or through reference picture resampling (RPR)), change a target bitrate, change an actual bitrate.

[0088] In some embodiments, encoding the current picture using the obtained UCID and / or data derived using the obtained UCID further includes making a second encoding decision using the obtained UCID and / or data derived using the obtained UCID, and the second encoding decision is a decision for setting a value for at least a first rate control variable or the second encoding decision is a decision for setting a value for a variable that affects (directly or indirectly) the first rate control variable.

[0089] In some embodiments, the obtained UCID identifies a state or the process further comprises determining the state based on the obtained UCID, and making the first encoding decision using the obtained UCID and / or data derived using the obtained UCID comprises making the first encoding decision based on the identified or determined state.P112310W001 16

[0090] In some embodiments, making the first encoding decision based on the identified or determined state comprises determining whether an encoder for encoding the current picture has a current state corresponding to the identified or determined state.

[0091] In some embodiments, making the first encoding decision based on the identified or determined state comprises selecting a coding strategy or coding parameter value based on the identified or determined state.

[0092] In some embodiments, the identified or determined state is one or more of: a low complexity state, a high complexity state, a passive state, an active state, a scene cut, a global motion, a local activity.

[0093] In some embodiments, the identified or determined state is determined using a threshold value, wherein the identified or determined state is determined to be a first state if a first value is lower than the threshold value, or, the identified or determined state is determined to be a second state if the first value is higher than the threshold value.

[0094] In some embodiments, a number of passive pictures precede the current picture in display order, and the current picture is encoded further based on the number of passive pictures that precedes the current picture in display order.

[0095] In some embodiments, the state is a passive state and encoding the current picture based on the passive state comprises restricting the quality of the encoding of the current picture as a result of determining that the state is the passive state.

[0096] In some embodiments, a weight value is used in the encoding of the pictures, and the process further comprises: after making a rate control adjustment based on the UCID and / or the derived data, a bitrate and / or picture quality is determined; and determining whether or not to adjust the weight value based on the determined bitrate and / or determined picture quality.

[0097] In some embodiments, process 400 further comprises, prior to encoding the current picture using the obtained UCID and / or the derived data, obtaining correlation information indicating a correlation between the obtained UCID and one or more characteristics of an encoded picture (e.g., bitrate change, motion, block-partitioning, frequency) previously encoded based on UCID matching the obtained UCID, and making the first encoding decisionP112310W001 17using the obtained UCID and / or data derived using the obtained UCID comprises making the first encoding decision using the correlation information.

[0098] In some embodiments, making the first encoding decision using the correlation information comprises selecting an encoding strategy (e.g. activate or deactivate a first coding tool or selecting a rule for setting a QP value or lambda value) based on the correlation information and encoding the current picture based on the selected encoding strategy.

[0099] In some embodiments, the current picture is divided into at least a first part (e.g., a first subpicture) and a second part (e.g., a second subpicture), the first encoding decision is an encoding decision for encoding the first part of the current picture, encoding the current picture using the obtained UCID and / or the derived data further includes making a second encoding decision for the second part using the obtained UCID and / or the derived data, and the first encoding decision is different than the second encoding decision.[000100] In some embodiments, a first encoding instance makes the first encoding decision and encodes the first part of the current picture based on the first encoding decision, a second encoding instance makes the second encoding decision and encodes the second part of the current picture based on the second encoding decision.[000101] In some embodiments, process 400 further comprises determining a region-of-interest (ROI) within the current picture, and the first part of the current picture comprises the ROI.[000102] Rate Control Using the UCID[000103] In this embodiment, encoder 102 include a rate controller (RC) 165 that implements a rate control algorithm and the UCID 161 and / or 171 is passed to RC 165, which adjusts one or more rate control parameters based on the UCID.[000104] Adjusting one or more rate control parameters may include:[000105] 1. adjusting a QP value for a picture;[000106] 2. adjusting a QP value for a slice of a picture QP;[000107] 3. adjusting a QP value for a block (e.g., CTU or CU);[000108] 4. adjusting a lambda value;P112310W001 18[000109] 5. adjusting a value or values that may affect (directly or indirectly) the derivation of one or more QP values and / or lambda values.;[000110] 6. adjusting a bit budget;[000111] 7. selecting one or more coding tools, e.g. toggling certain tools on or off;[000112] 8. adjusting a frame rate;[000113] 9. skipping the coding of a picture (i.e., drop the picture) ;[000114] 10. skipping coding of residuals for a picture ;[000115] 11. adjusting a picture or slice type (I, P or B);[000116] 12. adjusting a resolution (e.g., by inserting a new random access point picture or through reference picture resampling (RPR));[000117] 13. adjusting a target bitrate;[000118] 14. adjusting the actual bitrate[000119] User Control Categorization[000120] In one embodiment, RC 165 adjust the one or more rate control parameters based on a category identified by the UCID or determined category to which the UCID belongs. In one embodiment, the category to which the UCID belongs depends on its impact on coding complexity.[000121] In one embodiment, the UCID belongs to one of the following two categories: 1) Low Complexity (a.k.a., Passive) and 2) High Complexity (a.k.a., Active).[000122] In one embodiment, UCID that does not include any user input belongs to the Low Complexity category. Similarly, UCID that indicates slow movements (movement below a certain threshold) or other controls that have no or little visual impact belong to the Low Complexity category. In short, in one embodiment, UCID that does not results in visual impact and thus higher video coding complexity belongs to the Low Complexity category.[000123] On the other hand, UCID that corresponds to non-slow movements such as, for example, walking, running, turning, strafingjumping, shooting, accelerating, braking etc, belongs to the High Complexity category. Similarly UCID that corresponds to camera angleP112310W001 19changes, panning, rotating, zooming etc. belongs to the High Complexity category. In short, UCID that results in visual impact and thus higher video coding complexity belongs to the High Complexity category.[000124] In another embodiment, a third category may be introduced: the Medium Complexity category. In this embodiment UCID that corresponds to scene cuts in the video, e.g. a menu button press, belong to the Medium Complexity category.[000125] In another embodiment, further categories may be defined to further differentiate the coding impact of the UCID. This may include several steps from low to high complexity or different types (e.g. differentiate between global motion and local activity).[000126] In yet another embodiment each possible user control inputs is treated independently and can be assumed to have a separate category for each.[000127] Low Complexity to High Complexity[000128] In this embodiment, pictures rendered by application 160 are classified as passive pictures or active pictures. In one embodiment, a passive picture is 1) a picture rendered by application 160 wherein the rendering was not based on any UCID (e.g., no UCID was received from application 170), 2) a picture having passive video content or 3) a picture rendered by application 160 wherein the rendering was based on UCID categorized as low complexity. Passive video content may be defined as static or low-motion video content, low-fidelity content and / or easily compressed video content (i.e., video content that may be encoded at relatively low bitrate for good quality). Examples of passive video content include scenes in games with little or no action or movement, and static or low motion passages in a video call or video conference.[000129] In one embodiment, rate control adjustment is applied when non-low complexity UCID (e.g., high or medium complexity UCID) is received after one or more passive pictures have been rendered.[000130] In one embodiment, RC 165 operates in one of two states, a passive state and an active state, and choses one of these operating states depending on the UCID 161. When operating in the passive state, RC 165 applies a first coding strategy that may include adjusting the rate control as disclosed herein, and when operating in the active state, RC 165 applies aP112310W001 20second coding strategy different than the first coding strategy that may include adjusting the rate control as disclosed herein[000131] In another embodiment, RC 165 operates in three or more states ranging from a most passive state to a most active state, where different coding strategies are applied for each state, where the coding strategies may have different levels of adjustments of the rate control output. Also here, the selected state is based on the UCID 161.[000132] In one version the UCID 161 comprises information explicitly indicating which operating state should be employed. The explicit information may be signaled in a standardized or otherwise agreed format such as the Game State over the Real Time Transport (RTP) protocol.[000133] FIG. 5 is a flowchart illustrating a process 500 performed by a client application. Process 500 may begin in step s502.[000134] Step s502 comprises determining a state of the client application (e.g., application 170). The state may be a state descriptive of the content, such as a low complexity state, a high complexity state, a passive state, an active state, a scene cut, global motion or local activity.[000135] Step s504 comprises sending an indication of the determined state of the client application to a server application (e.g., application 160), wherein the indication of the state is used by an encoder to encode a picture generated by the server application. The encoded picture may then be stored for later retrieval or transmitted to the client application. This process 500 may be performed by user apparatus 103.[000136] In some embodiments, process 500 further comprises receiving the encoded picture from the server application.[000137] In some embodiments, the state comprises: state descriptive of the content, Low complexity state, High complexity state, Passive state, Active state, Scene cut, Global motion, Local activity, and / or A region-of-interest (ROI) state, that may comprise information for identifying a ROI area in the picture.[000138] In some embodiments, the client application is a component of a cloud gaming service, the client application is a component of a video call or video conference service, the client application is a component of a split rendered XR service, or the client application is a component of a peer-to-peer applicationP112310W001 21[000139] In some embodiments, the client application is a component of a cloud gaming service, and the server application is a component of the cloud gaming service.[000140] In one embodiment, a threshold is used to determine when to trigger a rate control adjustment based on the UCID. The threshold may for instance comprise of a certain time or a certain number of passive pictures since the last active UCID was received. Exceeding the threshold would then switch the state from active to passive which would trigger the rate control adjustments. Another example would be receiving a sufficient amount of certain UCID during a certain time window, e.g. one user control message to move forward may not trigger the active state, whereas ten consecutive user control messages to move forward would. The threshold may be a fixed number or deduced from encoding configuration e.g. frame rate.[000141] Rate Control Offsets Based on Number of Passive Pictures[000142] In this embodiment, the number of passive pictures rendered within a given time window is used to determine the magnitude of the adjustment offsets to the rate control at the start of an active state. A high number of pictures in preceding passive state increases the rate control adjustment, and vice versa.[000143] In one version, an offset weight (w) is scaled linearly with number of passive pictures (PP) up to a desired maximum (APIA) value: w = min (MAX, PP) / MAX. This offset weight may be used to scale an adjustment offset variable (O) and add to the QP value, e.g. QP + (w * O), for further encoding of the picture. Similarly, the weight may be used to scale lambda value offsets or variables that is used to derive the QP and / or lambda value.[000144] Prevent QP Reduction in Passive Intervals[000145] In this embodiment, RC 165 restricts the quality for pictures when the UCID used in encoding the picture is passive UCID. This may be done by preventing QP and lambda values from falling below a certain threshold. Preventing too low values of QP and lambda prevents the encoder to code these pictures at higher quality than desired, resulting in a more consistent viewing experience as well as saved bits. Avoiding going to high in quality also reduces bit rate spikes when the content changes to more complex, reducing the difference the rate control has to adapt to.[000146] Updating Rate Control Strategy During a SessionP112310W001 22[000147] In one embodiment, the magnitude of a rate control adjustment (e.g., the change in QP value) based on the UCID is continuously updated. The weight scalar described above may be used to set adjustment magnitude starting at a default value. The weight’s default value may depend on corresponding user control category.[000148] After an action where the UCID has resulted in a rate control adjustment, the resulting bit rate and / or picture quality is evaluated, and the corresponding weight scalar adjusted accordingly. For example, if there was a bit rate spike, then the weight scalar may be updated towards adjusting the rate control for lower bit rate when the same or similar UCID is later received, or vice versa.[000149] In one embodiment, RC 165 may have no prior knowledge of the UCID from the game but could learn and adapt its strategy from the received UCID on the fly during a session. RC 165 could for instance look for correlations between the specific UCID and corresponding changes / increases in bitrate, motion, block-partitioning, frequency, etc, of the following frames. Based on a found correlation for a specific UCID, RC 165 could adapt its behavior for the next time that specific UCID is received, e.g. by modifying QP or lambda values in preparation of the expected event.[000150] Having an adaptive rate control strategy unaware of specific inputs has its advantages that the rate control may be agnostic to what game is played and could thus be used for a multitude of different games without having to setup the behavior for specific user control input in advance.[000151] In one embodiment, apparatus 130 performs the following steps:[000152] Step 1: Receive an instance of UCID 171 indicating a certain user control input (user control input A);[000153] Step 2: Render a new picture based on the user control input A;[000154] Step 3 : Encode the new picture of the video using a certain strategy (e.g. a rule how to set the QP and lambda values) associated with user control input A;[000155] Step 4: Determine a correlation between the user control input A and a difference in characteristics between of the encoded current picture and a previous encoded picture; theP112310W001 23characteristics may for instance be a significantly large (e.g. over a certain threshold) increase or change in bitrate, motion, block-partitioning, or frequency of the encoded picture;[000156] Step 5: Update the certain strategy based on the determined correlation; this may for instance comprise adding a delta value to the QP and lambda values when the UCID indicates user control input A;[000157] Step 6: Receive a new instance of UCID that indicates user control input A;[000158] Step 7: Render a new picture based on the new instance of the UCID;[000159] Step 8: Encode the picture rendered in step 7 using the updated strategy associated with user control input A.[000160] Different Encoding Decisions for Different Parts of a Picture[000161] In this embodiment, the picture is divided into two or more parts - i.e., at least a first part (e.g., first subpicture) and a second part (e.g., second subpicture). Encoding the picture using the obtained UCID and / or the derived data includes using the obtained UCID and / or the derived data to make a first encoding decision for the first part and a second encoding decision for the second part, where the first encoding decision is different than the second encoding decision.[000162] In one embodiment, encoder 102 includes at least a first encoder instance and a second encoder instance, where the first encoder instance performs the encoding of the first part of the picture and the second encoder instance performs the encoding of the second part of the picture. Each encode instance may receive the same UCID 161 but may choose to treat it differently.[000163] For example, some UCID may trigger motion in only the first part of the picture, but not in the other(s), so the rate control strategy for each picture part may be different. The difference in strategy could be configured based on prior knowledge of the content to be encoded or determined during runtime as described above.[000164] In one embodiment, UCID 161 may be used to guide the coding of a region-of-interest (ROI) in the picture. The guidance may be done such that the ROI is coded with better quality, e.g. lower QP, than other parts of the picture. For instance, if an aim command or shoot command is received in a third-person shooter game, the area at which the user is aiming isP112310W001 24encoded with higher quality than other parts of the picture. In another example, the UCID may be used to foresee the content complexity in screen sharing for a specific ROI area, e.g. the mouse control commands in a drag-and-drop action could be used to make a decision on the encoding for the particular area, e.g. to lower the QP for that ROI area.[000165] In one embodiment, the UCID 161 comprises information for deriving the ROI area. This information may for instance comprise one or more of vertical and horizontal coordinates of the ROI area, the size of the ROI area, e.g. width or height, or an index identifying the area among a set or list of predefined areas, e.g. a coding structure such as a CTU.[000166] FIG. 6 is a block diagram of an apparatus 600 for implementing encoding apparatus 130 and or user apparatus 103, according to some embodiments. As shown in FIG. 6, apparatus 600 may comprise: processing circuitry (PC) 602, which comprises one or more processors (P) 655 (e.g., one or more general purpose microprocessors and / or one or more other processors, such as an application specific integrated circuit (ASIC), field-programmable gate arrays (FPGAs), and the like), which processors may be co-located in a single housing or in a single data center or may be geographically distributed (e.g., apparatus 600 may be a distributed, cloud computing system comprising two or more computers or a monolithic computing system consisting of a single computer); at least one network interface 648 (e.g., a physical interface or air interface) comprising a transmitter (Tx) 645 and a receiver (Rx) 647 for enabling apparatus 600 to transmit data to and receive data from other nodes connected to a network 110 (e.g., an Internet Protocol (IP) network) to which network interface 648 is connected (physically or wirelessly) (e.g., network interface 648 may be coupled to an antenna arrangement comprising one or more antennas for enabling encoder apparatus 600 to wirelessly transmit / receive data); and a storage unit (a.k.a., “data storage system”) 608, which may include one or more nonvolatile storage devices and / or one or more volatile storage devices. In embodiments where PC 602 includes a programmable processor, a computer readable storage medium (CRSM) 642 may be provided. CRSM 642 may store a computer program (CP) 643 comprising computer readable instructions (CRI) 644. CRSM 642 may be a non-transitory computer readable medium, such as, magnetic media (e.g., a hard disk), optical media, memory devices (e.g., random access memory, flash memory), and the like. In some embodiments, the CRI 644 of computer program 643 is configured such that when executed by PC 602, the CRI causes encoder apparatus 600 to perform steps described herein (e.g., steps described herein with reference to the flow charts). InP112310W001 25other embodiments, encoder apparatus 600 may be configured to perform steps described herein without the need for code. That is, for example, PC 602 may consist merely of one or more ASICs. Hence, the features of the embodiments described herein may be implemented in hardware and / or software.[000167] Conclusion and Results[000168] The embodiments disclosed herein provide an advantage in that they reduce bitrate spikes and / or increase compression quality. FIG. 7 illustrates data showing an increase in compression quality (i.e., equivalent to increased compression efficiency for a given quality), achieved using an embodiment described herein.[000169] While various embodiments are described herein, it should be understood that they have been presented by way of example only, and not limitation. Thus, the breadth and scope of this disclosure should not be limited by any of the above-described exemplary embodiments. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed by the disclosure unless otherwise indicated herein or otherwise clearly contradicted by context.[000170] As used herein transmitting a message “to” or “toward” an intended recipient encompasses transmitting the message directly to the intended recipient or transmitting the message indirectly to the intended recipient (i.e., one or more other nodes are used to relay the message from the source node to the intended recipient). Likewise, as used herein receiving a message “from” a sender encompasses receiving the message directly from the sender or indirectly from the sender (i.e., one or more nodes are used to relay the message from the sender to the receiving node). Further, as used herein “a” means “at least one” or “one or more.”[000171] Additionally, while the processes described above and illustrated in the drawings are shown as a sequence of steps, this was done solely for the sake of illustration. Accordingly, it is contemplated that some steps may be added, some steps may be omitted, the order of the steps may be re-arranged, and some steps may be performed in parallel.[000172] References[000173] [1] JVET-AI2027P112310W001 26[000174] [2] Schulzrinne, H., “RTP: A Transport Protocol for Real-Time Applications,” Internet Engineering Task Force (IETF), Network Working Group, Request for Comments (RFC) 3550, July 2003 (available at datatracker.ietf.org / doc / html / rfc3550).[000175] [3] Jennings, C., “Game State over Real Time Protocol,” draft-jennings-dispatch-game-state-over-rtp-01, 24 October 2022 (available at www.ietf.org / archive / id / draft-jennings-dispatch-game-state-over-rtp-01.txt).

Claims

1. P112310W001 27CLAIMS1. A method (400) for encoding pictures, the method comprising:obtaining (s402) user control input data, UCID (161, 171);obtaining (s404) a current picture generated by a first application (160);encoding (s406) the current picture using the obtained UCID (161, 171) and / or data derived using the obtained UCID, thereby producing an encoded current picture; and outputting (s408) the encoded current picture.

2. The method of claim 1 , whereinoutputting the encoded picture comprises transmitting the encoded picture to a user apparatus (103) or providing the encoded picture to a transmitter for transmitting the encoded picture to the user apparatus (103), andobtaining the user control input data, UCID, comprises receiving the UCID from the user apparatus (103) or from the first application (160).

3. The method of claim 2, whereinthe user apparatus comprises a user interface, andthe user apparatus is configured to obtain the user control input data through the user interface.

4. The method of any of claims 1-3, whereinan encoder (102) performs the steps of obtaining the current picture and encoding the current picture,the encoder is a component of an application server (130) further comprising the first application (160),the first application (160) obtains the UCID from a second application (170) and generates the current picture using the UCID,the second application (170) is a component of the user apparatus (103), andthe first application (160) provides to the encoder (102) the UCID and / or data derived using the UCID.P112310W001 285. The method of claim 4, whereinthe first application is a component of a cloud gaming service,the first application is a component of a video call or video conference service, the first application is a component of a split rendered XR service, orthe first application is a component of a peer-to-peer application.

6. The method of claim 4 or 5, whereinthe application server is a cloud gaming server, andthe user apparatus is a cloud gaming client.

7. The method of any of claims 1-6, wherein the UCID comprises:a movement command,a change view command,a task command indicating a task to be carried,a start command for starting an application,a pause command pausing the application,a resume command for un-pausing the application,a stop command for stopping the application,a menu access command for accessing an indicated menu or menu item, information indicating a participant entering or leaving a conference meeting, information indicating a participant of the conference meeting becoming active or passive,gyroscopic sensor data from a phone being used for a call,information indicating a mouse movement, and / orinformation indicating a keyboard stroke.

8. The method of any one of claims 1-7, whereinencoding the current picture using the obtained UCID and / or data derived using the obtained UCID includes at least making a first encoding decision using the obtained UCID and / or data derived using the obtained UCID.P112310W001 299. The method of claim 8, whereinthe first encoding decision is a decision for setting a value for at least a first rate control variable, orthe first encoding decision is a decision for setting a value for a variable that affects the first rate control variable.

10. The method of claim 9, wherein encoding the current picture using the obtained UCID and / or data derived using the obtained UCID further comprises, based on the first encoding decision that was made using the UCID and / or the data derived using the obtained UCID, setting a value for at least a first rate control variable or setting a value for a variable that affects the first rate control variable.

11. The method of claim 9 or 10, wherein the first rate control variable is:a quantization parameter, QP,a lambda parameter,a bit budget,a frame rate parameter,a resolution parameter, ora target bitrate parameter.

12. The method of claim 8, wherein the first encoding decision is a decision whether or not to:change a quantization parameter,change a lambda parameter,activate or deactivate a first coding tool,change a frame rate,skip coding of a picture,skip coding of residuals for a picture,change a picture or a slice type,change a resolution,P112310W001 30change a target bitrate, orchange an actual bitrate.

13. The method of claim 12, whereinencoding the current picture using the obtained UCID and / or data derived using the obtained UCID further includes making a second encoding decision using the obtained UCID and / or data derived using the obtained UCID, andthe second encoding decision is a decision for setting a value for at least a first rate control variable or the second encoding decision is a decision for setting a value for a variable that affects the first rate control variable.

14. The method of any one of claims 8-13, whereinthe obtained UCID identifies a state or the method further comprises determining the state based on the obtained UCID, andmaking the first encoding decision using the obtained UCID and / or data derived using the obtained UCID comprises making the first encoding decision based on the identified or determined state.

15. The method of claim 14, whereinmaking the first encoding decision based on the identified or determined state comprises determining whether the encoder for encoding the current picture has a current state corresponding to the identified or determined state.

16. The method of claim 14 or 15, whereinmaking the first encoding decision based on the identified or determined state comprises selecting a coding strategy or coding parameter value based on the identified or determined state.

17. The method of any of claims 14-16, wherein the identified or determined state is one or more of:a low complexity state,a high complexity state,P112310W001 31a passive state,an active state,a scene cut,a global motion,a local activity.

18. The method of any of claims 14-17, wherein the identified or determined state is determined using a threshold value, whereinthe identified or determined state is determined to be a first state if a first value is lower than the threshold value, or,the identified or determined state is determined to be a second state if the first value is higher than the threshold value.

19. The method of any of claims 14-18, whereina number of passive pictures precede the current picture in display order, andthe current picture is encoded further based on the number of passive pictures that precedes the current picture in display order.

20. The method of any of claims 14-19, wherein the identified or determined state is a passive state and encoding the current picture based on the passive state comprises restricting the quality of the encoding of the current picture as a result of determining that the state is the passive state.

21. The method of any of claims 1-20, whereina weight value is used in the encoding of the pictures, andthe method further comprises:after making a rate control adjustment based on the UCID and / or the data derived using the obtained UCID, a bitrate and / or picture quality is determined; and determining whether or not to adjust the weight value based on the determined bitrate and / or determined picture quality.P112310W001 3222. The method of any one of claims 8-21, whereinthe method further comprises, prior to encoding the current picture using the obtained UCID and / or the data derived using the obtained UCID, obtaining correlation information indicating a correlation between the obtained UCID and one or more characteristics of an encoded picture previously encoded based on UCID matching the obtained UCID, and making the first encoding decision using the obtained UCID and / or data derived using the obtained UCID comprises making the first encoding decision using the correlation information.

23. The method of claim 22, wherein making the first encoding decision using the correlation information comprises selecting an encoding strategy based on the correlation information and encoding the current picture based on the selected encoding strategy.

24. The method of any one of claims 8-23, whereinthe current picture is divided into at least a first part and a second part,the first encoding decision is an encoding decision for encoding the first part of the current picture,encoding the current picture using the obtained UCID and / or the data derived using the obtained UCID further includes making a second encoding decision for the second part using the obtained UCID and / or the data derived using the obtained UCID, andthe first encoding decision is different than the second encoding decision.

25. The method of claim 24, whereina first encoding instance makes the first encoding decision and encodes the first part of the current picture based on the first encoding decision,a second encoding instance makes the second encoding decision and encodes the second part of the current picture based on the second encoding decision.

26. The method of claim 24 or 25, whereinthe method further comprises determining a region-of-interest, ROI, within the current picture, andthe first part of the current picture comprises the ROI.P112310W001 3327. A method (500) performed by a client application (170), the method comprising: determining (s502) a state of the client application; andsending (s504) an indication of the determined state of the client application to a server application (160), wherein the indication of the state is used by an encoder to encode a picture generated by the server application.

28. The method of claim 27, wherein the method further comprises receiving the encoded picture from the server application.

29. The method of claim 27 or 28 wherein the state comprises:a state descriptive of the content,low complexity state,high complexity state,passive state,active state,scene cut,global motion,local activity, and / ora region-of-interest, ROI, state, that may comprise information for identifying a ROI area in the picture.

30. The method of any one of claims 27-29, whereinthe client application is a component of a cloud gaming service,the client application is a component of a video call or video conference service, the client application is a component of a split rendered XR service, orthe client application is a component of a peer-to-peer application31. The method of claim 30, whereinthe client application is a component of a cloud gaming service, andthe server application is a component of the cloud gaming service.P112310W001 3432. A computer program (643) comprising instructions (644) which when executed by processing circuitry (602) of an apparatus causes the apparatus to perform the method of at least one of claims 1-31.

33. A carrier containing the computer program of claim 32, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium (642).

34. An apparatus (600) for encoding pictures, the apparatus being configured to perform a method comprising:obtaining (s402) user control input data, UCID (161, 171);obtaining (s404) a current picture generated by a first application (160);encoding (s406) the current picture using the obtained UCID (161, 171) and / or data derived using the obtained UCID, thereby producing an encoded current picture; and outputting (s408) the encoded current picture.

35. The apparatus of claim 34, wherein the apparatus is further configured to perform the method of any one of claims 2-26.

36. An apparatus (600), the apparatus being configured to perform a method comprising: determining (s502) a state of a client application; andsending (s504) an indication of the determined state of the client application to a server application (160), wherein the indication of the state is used by an encoder to encode a picture generated by the server application.

37. The apparatus of claim 36, wherein the apparatus is further configured to perform the method of any one of claims 28-31.

Images