Encoding method, decoding method, rendering system, bitstream, encoder, and decoder

Abstract

An encoding method, a decoding method, a rendering system, a bitstream, an encoder, and a decoder. The encoding method comprises: acquiring multiple pieces of rendering API data, wherein the multiple pieces of rendering API data is used for generating a rendered image; acquiring the type of each piece of the multiple pieces of rendering API data; on the basis of the type of each piece of the multiple pieces of rendering API data, acquiring an encoding mode corresponding to each piece of the multiple pieces of rendering API data; and on the basis of the encoding mode corresponding to each piece of the multiple pieces of rendering API data, encoding each piece of the multiple pieces of rendering API data to obtain a bitstream.

Description

Encoding and decoding methods, rendering systems, bitstreams, and codecs

[0001] This application claims priority to Chinese patent application filed on January 17, 2025, with application number 202510082907.5 and entitled "Encoding / Decoding Method, Rendering System, Bitstream and Encoder / Decoder", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of encoding and decoding, and in particular to an encoding and decoding method, a rendering system, a bitstream, and a codec. Background Technology

[0003] Due to limitations in power supply, heat dissipation, and space, the performance of graphics processing units (GPUs) in small devices such as mobile phones is restricted, resulting in poor quality rendered images. In contrast, GPUs in large devices such as computers and servers are much more powerful. Therefore, it is worth considering offloading GPU rendering tasks from small terminal devices like mobile phones to more powerful large terminal devices through multi-device collaboration.

[0004] However, the large volume of operation instructions, algorithms, and data required for rendering (collectively referred to as Rendering Application Programming Interface (API) data) generates significant power consumption during transmission. To address this issue, the rendering API data can be encoded before transmission. Currently, the common method for encoding rendering API data involves converting it into strings or fixed-point data; then, a dictionary compression algorithm is used to compress the residual between the rendered API data and the converted data to obtain the bitstream. This method of encoding rendering API data has a relatively low compression ratio. Summary of the Invention

[0005] In view of this, this application provides an encoding / decoding method, a rendering system, a bitstream, and a codec. This encoding / decoding method can improve the compression ratio and reduce the amount of data transmitted.

[0006] The encoding and decoding methods involved in this application can be applied to any computing power sharing scenario or any GPU computing power offloading scenario (also known as GPU computing power offloading).

[0007] For example, computing power sharing scenarios may include scenarios where one device shares the computing power of another device or one device shares the computing power of multiple other devices.

[0008] For example, a scenario for unloading GPU computing power can include a scenario for unloading all GPU computing power or a scenario for unloading part of GPU computing power.

[0009] This application uses a scenario of shared rendering computing power or a scenario of unloading GPU rendering computing power as an example for illustration.

[0010] In a first aspect, this application provides an encoding method, which includes: first, acquiring multiple rendering application programming interface (API) data, the multiple rendering API data being used to generate a rendered image; then, acquiring the type of each rendering API data in the multiple rendering API data; subsequently, based on the type of each rendering API data in the multiple rendering API data, acquiring the encoding method corresponding to each rendering API data in the multiple rendering API data, wherein the encoding method corresponding to at least one type of rendering API data is a lossy encoding method; and then, based on the encoding method corresponding to each rendering API data in the multiple rendering API data, encoding each rendering API data in the multiple rendering API data to obtain a bitstream.

[0011] In other words, this application takes into account the types of rendering API data and encodes the rendering API data based on its type during the encoding process. Specifically, this application uses corresponding encoding methods for different types of rendering API data. Since data of the same type has high structural consistency, the encoding method of this application can reduce redundancy in different types of rendering API data, thereby improving the compression rate of the rendering API data and reducing the amount of transmitted rendering API data.

[0012] In addition, at least one type of rendering API data in this application is encoded using a lossy encoding method; this can reduce the compression rate of rendering API data with a lossy encoding method.

[0013] For example, the rendering API data involved in this application may include operation instructions, algorithms and data for controlling the GPU to perform rendering, such as the standard OpenGL API; the GPU needs to process several rendering API data to render one frame of image.

[0014] For example, a rendering API data can be an operation instruction, a piece of data, or a set of algorithms. For example, the data included in the rendering API data can be such as texture data, mesh data, etc.; the algorithms included in the rendering API data (also known as rendering operation logic) can be such as geometry processing, shading, lighting calculation, etc.; the operation instructions included in the rendering API data can be such as translation operations, rotation operations, etc.

[0015] Mesh data: A data structure used in computer graphics to model various irregular objects. It can be used to represent the shape and structure of three-dimensional objects. Mesh data can include vertex coordinates, normals, topology, etc.

[0016] Texture data: Texture data can be used to describe the details and texture of an image or object's surface (or, in other words, texture data can be used to describe the texture of an image or object's surface). Texture data can be viewed as an image, which can be used in computer graphics to attach to the surface of a 3D model, increasing the realism and detail of the graphics. For example, texture data can include the color information of vertices on a 3D model (such as RGB, YUV, etc.); texture data can also include the material information of the 3D model, such as roughness, normal information, ambient light information, etc.

[0017] Rendering Computation Logic: Algorithms that control and guide various calculations and operations during the rendering process, determining how a 3D scene is converted into a 2D image.

[0018] In one possible implementation, the encoding method corresponding to each type can be pre-set; in this way, the encoding method corresponding to each rendering API data can be determined according to the pre-set encoding method corresponding to each type and the type of each rendering API data.

[0019] In one possible implementation, the impact of each rendering API data on the quality of the rendered image and the data volume of each rendering API data can be analyzed based on the type of each rendering API data. Then, based on the impact of each rendering API data on the quality of the rendered image and the data volume of each rendering API data, the encoding method corresponding to each rendering API data can be determined.

[0020] For example, two rendering API data of the same type can have the same encoding method, while two rendering API data of different types can have different encoding methods.

[0021] For example, the encoding method may include the compression algorithm used for compression. Optionally, the encoding method may also include whether to refer to data in the cache queue for compression. Optionally, the encoding method may also include preprocessing methods before compression and / or post-processing methods after compression.

[0022] Each rendering API data point is encoded to obtain encoded rendering API data (also known as encoded data of each rendering API data point). Then, multiple encoded rendering API data points (also known as encoded data of multiple rendering API data points) are used to generate the bitstream. Furthermore, bitstream description information can be written (or packaged) into the bitstream. This description information may include, but is not limited to: information describing the rendering API data (such as the type of rendering API data, the number of rendering API data points, etc.), information describing the compression algorithm, information describing the preprocessing method, information describing the post-processing method, and information describing whether data in the cache queue is referenced for compression.

[0023] For example, a bitstream can be called an encoded bitstream, a bitstream, or a bit stream.

[0024] For example, the multiple rendering API data obtained may be all or part of the rendering API data required to generate the rendered image for final display.

[0025] For example, you can obtain the encoding method corresponding to each rendering API data in the total data of these multiple rendering API data, or you can obtain the encoding method corresponding to each rendering API data in the partial data of these multiple rendering API data.

[0026] For example, each of the entire data of multiple rendering API data can be encoded, or each of the partial data of multiple rendering API data can be encoded.

[0027] According to the first aspect, based on the type of each rendering API data in the multiple rendering API data, the encoding method corresponding to each rendering API data in the multiple rendering API data is obtained, including: classifying the multiple rendering API data based on the semantics of each rendering API data in the multiple rendering API data to obtain the type of each rendering API data in the multiple rendering API data.

[0028] Among them, rendering API data with the same or similar semantics have a high degree of structural consistency. Therefore, rendering API data with the same or similar semantics can be classified into one type to ensure the structural consistency of rendering API data of the same type after classification.

[0029] For example, the multiple rendering API data can be classified according to pre-set data classification rules and the semantics of each rendering API data in the multiple rendering API data.

[0030] According to the first aspect, or any implementation of the first aspect above, based on the type of each rendering API data in the plurality of rendering API data, the encoding method corresponding to each rendering API data in the plurality of rendering API data is obtained, including: determining a first rendering API data in the plurality of rendering API data based on the type of each rendering API data in the plurality of rendering API data and a first preset type; determining the encoding method corresponding to the first rendering API data as a first encoding method, the first encoding method including data structure decomposition and a second encoding method, the first rendering API data being one or more; encoding each rendering API data in the plurality of rendering API data based on the encoding method corresponding to each rendering API data in the plurality of rendering API data to obtain a bitstream, including: encoding each first rendering API data based on the first encoding method corresponding to each first rendering API data to obtain each encoded first rendering API data, the bitstream including the encoded one or more first rendering API data.

[0031] For example, data structure decomposition can also be called data structure parsing, which is used to obtain data structure information such as fields, semantics of fields, data length of fields, and data characteristics of fields for each rendered API data.

[0032] For example, the first encoding method corresponding to each first rendering API data can be the same or different.

[0033] One possible implementation involves encoding only all first-rendering API data; this approach corresponds to scenarios where some GPU computing power is offloaded.

[0034] One possible implementation involves encoding all first rendering API data and other rendering API data; this approach corresponds to scenarios where the entire GPU portion is offloaded.

[0035] For example, each encoded first rendering API data can also be referred to as the encoded data of each first rendering API data.

[0036] According to the first aspect, or any implementation of the first aspect above, the second encoding method includes at least one of the following: lossy processing and compression using a general compression algorithm; compression using a dedicated compression algorithm; and compression based on reconstructed data of the encoded rendering API data in the cache queue.

[0037] For example, a dedicated compression algorithm can refer to a lossy or lossless compression algorithm that can only be used to compress a specific type of data. For example, dedicated compression algorithms can include a variety of algorithms, such as the JPEG compression algorithm (for compressing images), the H.264 compression algorithm (for compressing images), the H.265 compression algorithm (for compressing images), the Draco dedicated compression algorithm (for compressing mesh data), and the V-mesh dedicated compression algorithm (for compressing mesh data).

[0038] For example, a general compression algorithm can refer to a lossless compression algorithm that can be used to compress all types of data (including other types besides the rendering API data types involved in this application). For example, a general compression algorithm can include a variety of algorithms, such as LZ77 compression algorithm, Deflate compression algorithm, Zstd compression algorithm, etc.

[0039] For example, lossy processing can be used to reduce data volume, data complexity, and data entropy to improve data consistency; lossy processing can include, but is not limited to, quantization and clustering. It should be understood that lossy processing methods can include a variety of approaches.

[0040] For example, the cache queue can be located inside or outside the encoder and is used to store the reconstructed data of the encoded rendering API data and the identifier value of the reconstructed data of the encoded rendering API data.

[0041] The identifier value of the reconstructed data of the encoded rendering API data can be calculated by using algorithms such as hash algorithm and cyclic redundancy check (CRC) algorithm.

[0042] According to the first aspect, or any implementation of the first aspect above, based on the type of each rendering API data in the plurality of rendering API data, obtaining the encoding method corresponding to each rendering API data in the plurality of rendering API data, further includes: determining a second rendering API data in the plurality of rendering API data based on the type of each rendering API data in the plurality of rendering API data and a second preset type; determining that the encoding method corresponding to the second rendering API data is a third encoding method, and the second rendering API data is one or more; encoding each rendering API data in the plurality of rendering API data based on the encoding method corresponding to each rendering API data in the plurality of rendering API data to obtain a bitstream, further includes: encoding each second rendering API data based on the third encoding method corresponding to each second rendering API data to obtain each encoded second rendering API data, and the bitstream also includes the one or more encoded second rendering API data.

[0043] In other words, this application encodes all the rendering API data required to generate the rendered image, that is, it encodes all the first rendering API data and all the second rendering API data; this implementation corresponds to a scenario where the GPU is completely offloaded.

[0044] For example, the second encoding method and the third encoding method are different.

[0045] According to the first aspect, or any implementation of the first aspect above, the third encoding method includes compression using a general compression algorithm.

[0046] According to the first aspect, or any implementation of the first aspect above, the type of the rendering API data includes at least one of the following: function type, operation logic type, texture data type, mesh data type, or other type.

[0047] For example, the data classification rules are as follows: rendering API data whose semantics are function names are classified into one category, called function type; rendering API data whose semantics are rendering operation logic are classified into one category, called operation logic type; rendering API data whose semantics are texture data of rendering objects are classified into one category, called texture data type; rendering API data whose semantics are mesh data of rendering objects are classified into one category, called mesh data type; and rendering API data whose semantics are other (other semantics besides function names, rendering operation logic, texture data of rendering objects, and mesh data of rendering objects) are classified into another category, called other types.

[0048] It should be understood that multiple rendering API data can also be divided into other types or more granular types, and this application does not impose any restrictions on this.

[0049] According to the first aspect, or any implementation of the first aspect above, the first preset type includes an operational logic type, a texture data type, or a mesh data type, and the second preset type includes a function type or other types; or, the first preset type includes a texture data type or a mesh data type, and the second preset type includes a function type, an operational logic type, or other types.

[0050] According to the first aspect, or any implementation of the first aspect above, each first rendering API data is encoded based on the first encoding method corresponding to each first rendering API data to obtain each encoded first rendering API data, including: performing data structure decomposition on each first rendering API data to obtain data structure information of each first rendering API data; and encoding each first rendering API data based on the data structure information of each first rendering API data and the second encoding method corresponding to each first rendering API data to obtain each encoded first rendering API data.

[0051] In other words, this application takes into account the structural characteristics of each rendering API data and encodes each rendering API data according to its structural characteristics. Since rendering API data of the same type often have the same data structure, data redundancy of rendering API data of the same type can be further reduced.

[0052] According to the first aspect, or any implementation thereof, the second encoding method involves lossy processing and compression using a general compression algorithm. Based on the data structure information of each first rendering API data and the corresponding second encoding method, each first rendering API data is encoded to obtain encoded first rendering API data. This includes: determining sub-data in each first rendering API data that satisfies a first preset condition based on the data structure information of each first rendering API data; performing lossy processing on the sub-data in each first rendering API data that satisfies the first preset condition to obtain loss-processed sub-data, where lossy processing includes quantization or clustering; and compressing the loss-processed sub-data using a general compression algorithm to obtain encoded sub-data. Thus, by performing lossy processing on the first rendering API data, the consistency of the first rendering API data can be increased; furthermore, data redundancy in the first rendering API data can be reduced.

[0053] In this case, the first preset type includes operational logic type, texture data type, or mesh data type, and the second preset type includes function type or other types.

[0054] In one possible approach, the first preset condition can be set in advance according to the amount of data and / or the degree of impact on the quality of the rendered image.

[0055] For example, the first preset condition is float type data. Another example is that the first preset condition is a specific field; where the specific field differs depending on the type of the first rendering API data.

[0056] In one possible approach, the first preset condition could be that the data volume is greater than a first threshold, and / or that the impact on the quality of the rendered image is less than a second threshold. The impact on the quality of the rendered image can be represented by the distortion determined between a rendered image obtained using this data and a rendered image obtained without using this data. The first and second thresholds can be set as needed, and this application does not impose any restrictions on them.

[0057] For example, for sub-data that does not meet the first preset condition, a general compression algorithm can be directly used for compression.

[0058] It should be understood that the lossy processing performed on sub-data that meet the first preset condition in different types of first rendering API data can be the same or different. For example, the lossy processing performed on sub-data that meet the first preset condition in first rendering API data of texture type is clustering, while the lossy processing performed on sub-data that meet the first preset condition in first rendering API data of operation logic type is quantization. As another example, the lossy processing performed on sub-data that meet the first preset condition in first rendering API data of texture type and sub-data that meet the first preset condition in first rendering API data of operation logic type is either clustering or quantization.

[0059] It should be understood that the general compression algorithm used for compressing lossy sub-data in different types of first rendering API data can be the same or different. For example, after lossy processing of sub-data that meets the first preset condition in first rendering API data of texture type, the LZ77 compression algorithm can be used; after lossy processing of sub-data that meets the first preset condition in first rendering API data of operation logic type, the Deflate compression algorithm can be used. As another example, after lossy processing of sub-data that meets the first preset condition in first rendering API data of texture type, and after lossy processing of sub-data that meets the first preset condition in first rendering API data of operation logic type, the LZ77 compression algorithm can be used for both.

[0060] It should be understood that the general compression algorithm used to compress different sub-data that do not meet the first preset condition within the same first rendering API data can be the same or different. Similarly, the general compression algorithm used to compress sub-data that do not meet the first preset condition within different types of first rendering API data can be the same or different.

[0061] According to the first aspect, or any implementation of the first aspect above, the second encoding method is to use a dedicated compression algorithm for compression; based on the data structure information of each first rendering API data and the second encoding method corresponding to each first rendering API data, each first rendering API data is encoded to obtain each encoded first rendering API data, including: determining the dedicated compression algorithm corresponding to each first rendering API data based on the data structure information of each first rendering API data; and using the dedicated compression algorithm corresponding to each first rendering API data to compress each first rendering API data to obtain each encoded first rendering API data.

[0062] Compared to using a general compression algorithm to compress the first rendering API data, using a dedicated compression algorithm to compress the first rendering API data can improve the compression ratio of the first rendering API data, thereby improving the overall data compression ratio.

[0063] In this case, the first preset type includes texture data type or mesh data type, and the second preset type includes operation logic type, function type or other types.

[0064] It should be understood that the dedicated compression algorithms corresponding to two first rendering API data of the same type can be the same or different; and the dedicated compression algorithms corresponding to two first rendering API data of different types can be the same or different.

[0065] According to the first aspect, or any implementation of the first aspect above, the second encoding method is to compress the reconstructed data of the encoded rendering API data in the cache queue; the encoding of each first rendering API data based on the data structure information of each first rendering API data and the second encoding method corresponding to each first rendering API data to obtain each encoded first rendering API data includes: determining the sub-data in each first rendering API data that meets the second preset condition based on the data structure information of each first rendering API data; determining whether there is reconstructed data of the sub-data that meets the second preset condition in the cache queue; when there is reconstructed data of the sub-data that meets the second preset condition in the cache queue, writing the identifier value of the reconstructed data of the sub-data that meets the second preset condition into the bitstream; when there is no reconstructed data of the sub-data that meets the second preset condition in the cache queue, compressing the sub-data that meets the second preset condition to obtain the encoded sub-data.

[0066] Compared to encoding all the data of a certain type of rendering API data, this method only encodes the sub-data that does not have reconstructed data in the cache queue, that is, only encodes a portion of the data of that type of rendering API data; thus, fewer bits are required, which can further improve the compression rate of that type of data, thereby improving the overall data compression rate.

[0067] In this case, the first preset type includes operational logic type, texture data type, or mesh data type, and the second preset type includes function type or other types.

[0068] For example, different second preset conditions can be pre-set for different types of first rendering API data. Specifically, for one type of first rendering API data, it can be pre-checked whether there is any identical sub-data among multiple first rendering API data of the same type corresponding to any two frames; when there is identical sub-data among multiple first rendering API data of the same type in two adjacent frames, the second preset condition can be set to that sub-data (or that field or that parameter).

[0069] For example, the second preset condition for the first rendering API data setting for the operation logic type can be set to: shader information.

[0070] For example, the second preset condition for the first rendering API data setting for texture data types could be: texture information (which could refer to a texture image).

[0071] For example, the second preset condition for the first rendering API data setting for texture data types could be: mesh information (which could refer to vertex coordinates).

[0072] It should be understood that within the same first rendering API data set, different sub-data points that meet the second preset condition may use the same or different general compression algorithms; similarly, within the same first rendering API data set, different sub-data points that meet the second preset condition may use the same or different dedicated compression algorithms. Furthermore, within different types of first rendering API data sets, sub-data points that meet the second preset condition may use the same or different general compression algorithms; and similarly, within different types of first rendering API data sets, sub-data points that meet the second preset condition may use the same or different dedicated compression algorithms.

[0073] It should be understood that within the same first rendering API data set, different sub-data points that do not meet the second preset condition may use the same or different general compression algorithms; similarly, within the same first rendering API data set, different sub-data points that do not meet the second preset condition may use the same or different dedicated compression algorithms. Similarly, within different types of first rendering API data sets, sub-data points that do not meet the second preset condition may use the same or different general compression algorithms; and similarly, within different types of first rendering API data sets, sub-data points that do not meet the second preset condition may use the same or different dedicated compression algorithms.

[0074] According to the first aspect, or any implementation of the first aspect above, the method further includes: when there is no reconstructed data of the sub-data that satisfies the second preset condition in the cache queue, adding the reconstructed data of the sub-data that satisfies the second preset condition and the identifier value of the reconstructed data of the sub-data that satisfies the second preset condition to the cache queue.

[0075] Secondly, this application provides a bitstream including encoded rendering application programming interface (API) data and a first identifier indicating the type of the rendering API data used to generate a rendered image.

[0076] According to the second aspect, the bitstream also includes: a second identifier that indicates the compression algorithm corresponding to the rendering API data.

[0077] According to the second aspect, or any implementation of the second aspect above, the bitstream further includes: a third identifier, which indicates whether the rendering API data has undergone lossy processing.

[0078] According to the second aspect, or any implementation of the second aspect above, the bitstream further includes: a fourth identifier, which indicates whether reconstructed data of the rendering API data exists in the cache queue.

[0079] According to the second aspect, or any implementation of the second aspect above, the bitstream further includes: a fifth identifier, which indicates the reconstructed data of the rendering API data in the cache queue.

[0080] According to the second aspect, or any implementation of the second aspect above, the bitstream further includes at least one of the following: frame identifier, total number of render API data entries, total number of render API data types, data length or lossy processing parameters for each type of render API data.

[0081] In another description, the bitstream may include encoded rendering API data, classification information, and compression information. The classification information may include a first identifier (also called a classification enumeration value, ranging from 0 to 4, representing the five types of rendering API data mentioned above), a frame identifier, the total number of data entries in the rendering API data, the total number of data types in the rendering API data, and the data length of each type of rendering API data. The compression information may include: a second identifier (also called a compression method enumeration value, ranging from 0 to 2, representing the LZ77 compression algorithm, Deflate compression algorithm, and Zstd compression algorithm, respectively), a third identifier (used to indicate whether the rendering API data has undergone lossy processing; the third identifier can also be called a lossy identifier, with a value of 0 or 1, where 0 indicates no lossy processing and 1 indicates lossy processing), and lossy processing parameters (such as the quantization level and clustering threshold). It should be understood that when more common compression algorithms are involved in the encoding process, the second identifier will have more values ​​(or the compression method enumeration values ​​will have more values).

[0082] Thirdly, this application provides a decoding method, which includes: first, receiving a bitstream; then, parsing the bitstream to obtain encoded multiple rendering application programming interface (API) data and a first identifier, the first identifier indicating the type of each rendering API data in the multiple rendering API data; next, determining the decoding method corresponding to each encoded rendering API data based on the first identifier; then, decoding each encoded rendering API data based on the decoding method corresponding to each encoded rendering API data to obtain reconstructed data of each rendering API data, wherein the reconstructed data of the multiple rendering API data is used to generate a rendered image, and the reconstructed data of at least one type of rendering API data differs from the reconstructed data of the at least one type of rendering API data.

[0083] According to the third aspect, based on the first identifier, determining the decoding method corresponding to each encoded rendering API data includes: determining the first encoded rendering API data based on the first identifier, and determining the decoding method corresponding to the first encoded rendering API data as the first decoding method; decoding each encoded rendering API data based on the decoding method corresponding to each encoded rendering API data to obtain the reconstructed data of each rendering API data includes: decoding each encoded first rendering API data based on the first decoding method corresponding to each encoded first rendering API data to obtain the reconstructed data of each first rendering API data.

[0084] According to the third aspect, or any implementation of the third aspect above, the first decoding method includes at least one of the following: inverse processing of lossy processing and decompression using a decompression algorithm corresponding to a general compression algorithm; decompression using a decompression algorithm corresponding to a special compression algorithm; and decompression based on reconstructed data of the decoded rendering API data in the cache queue.

[0085] According to the third aspect, or any implementation of the third aspect above, determining the decoding method corresponding to each encoded rendering API data based on the first identifier further includes: determining the encoded second rendering API data based on the first identifier, and determining the decoding method corresponding to the encoded second rendering API data as the second decoding method; decoding each encoded rendering API data based on the decoding method corresponding to each encoded rendering API data to obtain the reconstructed data of each rendering API data further includes: decoding each encoded second rendering API data based on the second decoding method corresponding to each encoded second rendering API data to obtain the reconstructed data of each second rendering API data.

[0086] According to the third aspect, or any implementation of the third aspect above, the second decoding method includes decompression using a decompression algorithm corresponding to a general compression algorithm.

[0087] According to the third aspect, or any implementation of the third aspect above, the type of the first rendering API data is an arithmetic logic type, a texture data type, or a mesh data type, and the type of the second rendering API data is a function type or other type; or, the type of the first rendering API data is a texture data type or a mesh data type, and the type of the second rendering API data is an arithmetic logic type, a function type, or other type.

[0088] According to the third aspect, or any of the implementations of the third aspect above, the reconstructed data of different types of first rendering API data have different lossy characteristics.

[0089] Among them, lossy features can also be called distortion features.

[0090] According to the third aspect, or any implementation of the third aspect above, each encoded first rendering API data is decoded based on the first decoding method corresponding to each encoded first rendering API data to obtain reconstructed data of each first rendering API data, including: determining the compression algorithm corresponding to each first rendering API data based on the second identifier parsed from the bitstream, the second identifier indicating the compression algorithm corresponding to each rendering API data; and decompressing each encoded first rendering API data based on the decompression algorithm corresponding to the compression algorithm corresponding to each first rendering API data to obtain reconstructed data of each first rendering API data.

[0091] According to the third aspect, or any implementation of the third aspect above, each encoded first rendering API data is decoded based on the first decoding method corresponding to each encoded first rendering API data to obtain the reconstructed data of each first rendering API data, and further includes: determining whether the sub-data in each first rendering API data has undergone lossy processing based on the third identifier parsed from the bitstream; for the reconstructed data of the sub-data in each first rendering API data that has undergone lossy processing, performing inverse lossy processing based on the lossy processing parameters parsed from the bitstream, the inverse lossy processing including inverse quantization processing or inverse clustering processing.

[0092] According to the third aspect, or any implementation of the third aspect above, each encoded first rendering API data is decoded based on the first decoding method corresponding to each encoded first rendering API data to obtain the reconstructed data of each first rendering API data, including: determining whether the cache queue contains the reconstructed data of the sub-data in the first rendering API data based on the fourth identifier parsed from the bitstream; when it is determined that the cache queue contains the reconstructed data of the sub-data in the first rendering API data, the reconstructed data of the sub-data in the first rendering API data is read from the cache queue based on the fifth identifier parsed from the bitstream.

[0093] According to the third aspect, or any implementation of the third aspect above, each encoded first rendering API data is decoded based on the first decoding method corresponding to each encoded first rendering API data to obtain reconstructed data of each first rendering API data, and further includes: when it is determined that the cache queue does not contain reconstructed data of the sub-data in the first rendering API data, determining the compression algorithm corresponding to the sub-data in the first rendering API data based on the second identifier parsed from the bitstream; and decompressing the sub-data in the encoded first rendering API data based on the decompression algorithm corresponding to the compression algorithm of the sub-data in the first rendering API data to obtain reconstructed data of the sub-data in the first rendering API data.

[0094] According to the third aspect, or any implementation of the third aspect above, the method further includes: when it is determined that the cache queue does not contain reconstructed data of the sub-data in the first rendering API data, adding the reconstructed data of the sub-data in the first rendering API data and the identifier value of the reconstructed data of the sub-data in the first rendering API data to the cache queue.

[0095] The third aspect and any implementation thereof correspond to the first aspect and any implementation thereof, respectively. The technical effects of the third aspect and any implementation thereof are similar to those of the first aspect and any implementation thereof, and will not be repeated here.

[0096] Fourthly, this application provides a rendering system, which includes a first device and a second device, wherein:

[0097] The first device is configured to: acquire multiple rendering application programming interface (API) data; acquire the type of each rendering API data; based on the type of each rendering API data, acquire the encoding method corresponding to each rendering API data, wherein at least one type of rendering API data corresponds to a lossy encoding method; encode each rendering API data based on the encoding method corresponding to each rendering API data to obtain a bitstream; write a first identifier into the bitstream, the first identifier indicating the type of each rendering API data; and send the bitstream.

[0098] The second device is configured to: receive a bitstream; parse the bitstream to obtain multiple encoded rendering API data and a first identifier; determine the decoding method corresponding to each encoded rendering API data based on the first identifier; decode each encoded rendering API data according to the decoding method corresponding to each encoded rendering API data to obtain reconstructed data for each rendering API data, wherein the reconstructed data of at least one type of rendering API data differs from the reconstructed data of the at least one type of rendering API data; render a 3D scene based on the reconstructed data of the multiple rendering API data to obtain a rendered image; and send the rendered image.

[0099] The first device is also used to display the rendered image.

[0100] According to the fourth aspect, the first device is specifically configured to classify the multiple rendering API data based on the semantics of each rendering API data to obtain the type of each rendering API data; determine the first rendering API data in the multiple rendering API data based on the type of each rendering API data and a first preset type; determine the second rendering API data in the multiple rendering API data based on the type of each rendering API data and a second preset type; encode each first rendering API data according to a first encoding method corresponding to each first rendering API data to obtain encoded first rendering API data; and encode each second rendering API data according to a third encoding method corresponding to each second rendering API data to obtain encoded second rendering API data; the bitstream includes one or more encoded first rendering API data and one or more encoded second rendering API data.

[0101] According to the fourth aspect, or any implementation of the fourth aspect above, the first preset type includes an operational logic type, a texture data type, or a mesh data type, and the second preset type includes a function type or other types; or, the first preset type includes a texture data type or a mesh data type, and the second preset type includes a function type, an operational logic type, or other types.

[0102] It should be noted that when the first device only encodes the portion of the rendering API data required for the final rendered image, the second device (the device performing the decoding) can render and generate rendered image_1 based on that portion of the rendering API data, and the GPU of the first device (the device performing the encoding) can generate rendered image_2 based on the remaining portion of the rendering API data. Then, the first device can fuse rendered image_1 and rendered image_2 to obtain the final rendered image for display. Rendered image_1 and rendered image_2 can be complete images or portions of complete images; the fused rendered image for final display is a complete image.

[0103] Fifthly, this application provides an electronic device, including: a memory and a processor, the memory being coupled to the processor; the memory storing program instructions, which, when executed by the processor, cause the electronic device to perform the method of the first aspect or any possible implementation thereof.

[0104] In a sixth aspect, this application provides an electronic device, including: a memory and a processor, the memory being coupled to the processor; the memory storing program instructions, which, when executed by the processor, cause the electronic device to perform the method of the third aspect or any possible implementation thereof.

[0105] In a seventh aspect, this application provides a chip including one or more interface circuits and one or more processors; the one or more processors receive or transmit data through the one or more interface circuits, and when the one or more processors execute computer instructions, cause the electronic device to perform the method in the first aspect or any possible implementation of the first aspect.

[0106] Eighthly, this application provides a chip including one or more interface circuits and one or more processors; the one or more processors receive or transmit data through the one or more interface circuits, and when the one or more processors execute computer instructions, cause the electronic device to perform the method in the third aspect or any possible implementation of the third aspect.

[0107] Ninthly, this application provides a computer-readable storage medium storing a computer program that, when run on a computer or processor, causes the computer or processor to perform the method of the first aspect or any possible implementation thereof.

[0108] In a tenth aspect, this application provides a computer-readable storage medium storing a computer program that, when run on a computer or processor, causes the computer or processor to perform the method of the third aspect or any possible implementation thereof.

[0109] In one aspect, this application provides a computer program product including computer instructions that, when executed by a computer or processor, cause the computer or processor to perform the method in the first aspect or any possible implementation thereof.

[0110] In a twelfth aspect, this application provides a computer program product including computer instructions that, when executed by a computer or processor, cause the computer or processor to perform the method in the third aspect or any possible implementation thereof.

[0111] In a thirteenth aspect, this application provides an encoder for performing the methods in the first aspect or any possible implementation thereof.

[0112] In a fourteenth aspect, this application provides a decoder for performing the methods in the third aspect or any possible implementation thereof.

[0113] In a fifteenth aspect, this application provides a decoder for performing the methods in the first aspect or any possible implementation thereof; or for performing the methods in the third aspect or any possible implementation thereof.

[0114] In a sixteenth aspect, this application provides a computer-readable storage medium that stores a bitstream in any possible implementation of the second or third aspect.

[0115] In a seventeenth aspect, this application provides a method for storing a bitstream, the method comprising: receiving a bitstream in any possible implementation of the second or third aspect; and storing the bitstream.

[0116] Eighteenthly, this application provides a method for transmitting a bitstream, the method comprising: storing the bitstream in any possible implementation of the second or third aspect; and transmitting the bitstream.

[0117] In a nineteenth aspect, this application provides a method for transmitting a code stream, the method comprising: receiving a code stream in any possible implementation of the second or third aspect; and sending the code stream.

[0118] In a twentieth aspect, this application provides a system for storing a bitstream, the system comprising: a receiving module and a storage module; the receiving module is used to receive a bitstream in any possible implementation of the second or third aspect; the storage module is used to store the bitstream.

[0119] In a twentieth aspect, this application provides a system for transmitting a code stream, the system comprising: a transceiver module and a storage module; the storage module is used to store the code stream in any possible implementation of the second or third aspect; the transceiver module is used to transmit the code stream.

[0120] In a twentieth aspect, this application provides a system for transmitting a code stream, the system comprising: a transceiver module for receiving a code stream in any possible implementation of the second or third aspect; the transceiver module is also configured to transmit the code stream.

[0121] In a twentieth aspect, this application provides an encoding device, the encoding device comprising:

[0122] The acquisition module is used to acquire multiple rendering application programming interface (API) data, which are used to generate rendered images.

[0123] The encoding module is used to obtain the type of each rendering API data in the multiple rendering API data; based on the type of each rendering API data in the multiple rendering API data, obtain the encoding method corresponding to each rendering API data in the multiple rendering API data; and based on the encoding method corresponding to each rendering API data in the multiple rendering API data, encode each rendering API data in the multiple rendering API data to obtain a bitstream.

[0124] Furthermore, this encoding module may also include a classification module, a data structure decomposition module, a lossy / lossless evaluation module, a lossy processing module, a general compression module, and a dedicated compression module. The classification module can be used to classify the rendering API data. The data structure decomposition module can be used to decompose the rendering API data into data structures. The lossy / lossless evaluation module can be used to determine which sub-data of the rendering API data requires lossy processing and which does not. The lossy processing module can be used to perform lossy processing on the rendering API data. The general compression module can be used to compress the rendering API data using a general compression algorithm, and the dedicated compression module can be used to compress the rendering API data using a dedicated compression algorithm. These modules can be used to execute the corresponding method steps in the first aspect and any possible implementation thereof, which will not be elaborated further here.

[0125] In a twentieth aspect, this application provides a decoding apparatus, the decoding apparatus comprising:

[0126] The receiving module is used to receive the bit stream;

[0127] A decoding module is used to parse the bitstream to obtain multiple encoded rendering application programming interface (API) data and a first identifier, the first identifier indicating the type of each rendering API data in the multiple rendering API data; based on the first identifier, the decoding method corresponding to each encoded rendering API data is determined; based on the decoding method corresponding to each encoded rendering API data, each encoded rendering API data is decoded to obtain reconstructed data of each rendering API data, the reconstructed data of the multiple rendering API data is used to generate a rendered image, and there is a difference between the reconstructed data of at least one type of rendering API data.

[0128] The decoding module may include a general decompression module, a dedicated decoding module, a lossy inverse processing module, and a data reconstruction module. The general decompression module can decompress the encoded rendering API data using a decompression algorithm corresponding to a general compression algorithm (which can be called a general decompression algorithm). The dedicated decompression module can decompress the rendering API data using a decompression algorithm corresponding to a dedicated compression algorithm (which can be called a dedicated decompression algorithm). The lossy inverse processing module can perform lossy inverse processing on the reconstructed data. The data reconstruction module can reconstruct the data output by the general decompression module, the dedicated decompression module, and the lossy inverse processing module to generate reconstructed rendering API data for final rendering. These modules can execute the corresponding method steps in the third aspect and any possible implementation thereof, which will not be elaborated further here.

[0129] In this embodiment, the electronic device, computer-readable storage medium, computer program product, chip or codec, decoder system, encoding device and decoding device are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects in the corresponding methods provided above. Attached Figure Description

[0130] Figure 1A is a schematic diagram of a rendering computing power sharing scenario 100 according to an embodiment of this application;

[0131] Figure 1B is a schematic diagram of another rendering computing power sharing scenario 200 according to an embodiment of this application;

[0132] Figure 1C is a schematic diagram of another rendering computing power sharing scenario 120 according to an embodiment of this application;

[0133] Figure 1D is a schematic diagram of another rendering computing power sharing scenario 130 according to an embodiment of this application;

[0134] Figure 2A is a schematic diagram of a game scene 200 according to an embodiment of this application;

[0135] Figure 2B is a schematic diagram of a meeting scenario 210 according to an embodiment of this application;

[0136] Figure 2C is a schematic diagram of a virtual concert scene 220 according to an embodiment of this application;

[0137] Figure 2D is a schematic diagram of a digital exhibition scene 230 according to an embodiment of this application;

[0138] Figure 3A is a schematic diagram of the structure of a rendering system 300 according to an embodiment of this application;

[0139] Figure 3B is a schematic diagram of the structure of a rendering API data encoding module 115 according to an embodiment of this application;

[0140] Figure 3C is a schematic diagram of the structure of a rendering API data decoding module 122 according to an embodiment of this application;

[0141] Figure 4 is a schematic diagram of an encoding process 400 according to an embodiment of this application;

[0142] Figure 5 is a schematic diagram of a decoding process 500 according to an embodiment of this application;

[0143] Figure 6 is a schematic diagram of another encoding process 600 according to an embodiment of this application;

[0144] Figure 7 is a schematic diagram of another decoding process 700 according to an embodiment of this application;

[0145] Figure 8A is a schematic diagram of the compression, decompression, and rendering process of rendering API data according to an embodiment of this application;

[0146] Figure 8B is a schematic diagram of another encoding process 800 according to an embodiment of this application;

[0147] Figure 8C is a schematic diagram of a code stream structure according to an embodiment of this application;

[0148] Figure 9 is a schematic diagram of another decoding process 900 according to an embodiment of this application;

[0149] Figure 10A is a schematic diagram of the compression, decompression, and rendering process of rendering API data according to an embodiment of this application;

[0150] Figure 10B is a schematic diagram of another encoding process 1000 according to an embodiment of this application;

[0151] Figure 11 is a schematic diagram of another decoding process 1100 according to an embodiment of this application;

[0152] Figure 12A is a schematic diagram of the compression, decompression, and rendering process of rendering API data according to an embodiment of this application;

[0153] Figure 12B is a schematic diagram of another encoding process 1200 according to an embodiment of this application;

[0154] Figure 13 is a schematic diagram of another decoding process 1300 according to an embodiment of this application;

[0155] Figure 14 is a schematic diagram of the structure of a device provided in an embodiment of this application. Detailed Implementation

[0156] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0157] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0158] The terms "first" and "second," etc., used in the specification and claims of this application are used to distinguish different objects, not to describe a specific order of objects. For example, "first target object" and "second target object," etc., are used to distinguish different target objects, not to describe a specific order of target objects.

[0159] In the embodiments of this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of the words "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0160] In the description of the embodiments in this application, unless otherwise stated, "multiple" means two or more. For example, multiple processing units means two or more processing units; multiple systems means two or more systems.

[0161] In the embodiments of this application, the modules / components shown in the framework diagram (or structural diagram or system diagram) are merely examples of this application. The actual framework (or structure or system) may include more or fewer modules / components than those shown in the diagram, or may have different component configurations. Furthermore, the various components / modules shown in the diagrams may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0162] The encoding and decoding methods involved in this application can be applied to any computing power sharing scenario or any GPU computing power offloading scenario (also known as GPU computing power offloading). The following explanation uses a computing power sharing scenario or a GPU rendering computing power offloading scenario as examples.

[0163] Figure 1A is a schematic diagram of a rendering computing power sharing scenario 100 according to an embodiment of this application. Figure 1A shows a scenario where a first device shares the rendering computing power of a second device, and Figure 1A also shows a scenario where all the rendering computing power of the GPU in the first device is unloaded.

[0164] In Figure 1A, the first device 11 can send all the rendering API data used to generate the rendered image to the second device 12. The GPU of the second device 12 then performs a rendering operation based on the rendering API data, generates the rendered image, and sends the rendered image back to the first device 11. The first device 11 can then display the rendered image. In this way, all the rendering computing power of the GPU in the first device 11 can be offloaded to the second device 12.

[0165] It should be noted that the GPU in the second device 12 in Figure 1A has better performance than the GPU in the first device 11.

[0166] Figure 1B is a schematic diagram of another rendering computing power sharing scenario 110 according to an embodiment of this application. Figure 1B shows a scenario where a first device shares the rendering computing power of a second device, and Figure 1B also shows a scenario where the GPU portion of the rendering computing power in the first device is unloaded.

[0167] In Figure 1B, the first device 11 can send partial rendering API data (i.e., rendering API data_1) used to generate a rendered image to the second device 12. The GPU of the second device 12 performs rendering operation 1 based on the rendering API data_1 to generate a rendered image_1, which is then sent to the first device 11. Additionally, the GPU in the first device 11 performs rendering operation 2 based on another portion of the rendering API data (i.e., rendering API data_2) used to generate the rendered image, generating a rendered image_2. Afterwards, the first device 11 can merge the rendered image_1 and the rendered image_2 to obtain and display the final rendered image. In this way, some of the rendering computing power of the GPU in the first device 11 can be offloaded to the second device 12.

[0168] It should be noted that the GPU performance in the second device 12 in Figure 1B is better than that in the first device 11, or the GPU performance in the second device 12 is comparable to that in the first device 11.

[0169] It should be noted that rendered image_1 and rendered image_2 can be complete images or parts of complete images; the rendered image obtained by fusion for final display is a complete image.

[0170] Figure 1C is a schematic diagram of another rendering computing power sharing scenario 120 according to an embodiment of this application. Figure 1C shows a scenario where a first device shares the rendering computing power of multiple second devices, and Figure 1C also shows a scenario where all the rendering computing power of the GPU in the first device is unloaded.

[0171] In Figure 1C, the first device 11 can send a portion of the rendering API data (i.e., rendering API data_1) used to generate the rendered image to the second device 1 (121), send a portion of the rendering API data (i.e., rendering API data_2) used to generate the rendered image to the second device 2 (122), ..., and send the remaining portion of the rendering API data (i.e., rendering API data_N (N is an integer greater than 1)) used to generate the rendered image to the second device 2 (12N). Then, the GPU in the second device 1 (121) performs rendering operation 1 based on the rendering API data_1, generates the rendered image_1, and sends the rendered image_1 to the first device. The GPU in the second device 2 (122) performs rendering operation 2 based on the rendering API data_2, generates the rendered image_2, and sends the rendered image_2 to the first device. Similarly, the GPU in the second device 2 (12N) performs rendering operation N based on the rendering API data_N, generates the rendered image_N, and sends the rendered image_N to the first device. Next, the first device 11 can fuse the rendered image_1, rendered image_2, ..., rendered image_N to obtain a rendered image for final display. In this way, all the rendering computing power of the GPU in the first device 11 can be offloaded to multiple second devices 12.

[0172] It should be noted that the GPU performance in the second device in Figure 1C is better than that in the first device 11, or the GPU performance in the second device is comparable to that in the first device 11.

[0173] It should be noted that the rendered image_1, rendered image_2, ..., rendered image_N can be a complete image or a part of a complete image; the rendered image obtained by fusion for the final display is a complete image.

[0174] Figure 1D is a schematic diagram of another rendering computing power sharing scenario 130 according to an embodiment of this application. Figure 1D shows a scenario where a first device shares the rendering computing power of multiple second devices, and Figure 1D also shows a scenario where the GPU portion of the rendering computing power in the first device is unloaded.

[0175] In Figure 1D, the first device 11 can send a portion of the rendering API data (i.e., rendering API data_1) used to generate a rendered image to the second device 1 (121), send a portion of the rendering API data (i.e., rendering API data_2) used to generate a rendered image to the second device 2 (122), ..., and send a portion of the rendering API data (i.e., rendering API data_N (N is an integer greater than 1)) used to generate a rendered image to the second device 2 (12N). Then, the GPU in the second device 1 (121) performs rendering operation 1 based on the rendering API data_1, generates a rendered image_1, and sends the rendered image_1 to the first device. The GPU in the second device 2 (122) performs rendering operation 2 based on the rendering API data_2, generates a rendered image_2, and sends the rendered image_2 to the first device. Similarly, the GPU in the second device 2 (12N) performs rendering operation N based on the rendering API data_N, generates a rendered image_N, and sends the rendered image_N to the first device. And the GPU in the first device 11 performs rendering operation _N+1 based on the remaining portion of rendering API data (i.e. rendering API data _N+1) used to generate the rendered image _N+1.

[0176] Next, the first device 11 can fuse rendered image_1, rendered image_2, ..., rendered image_N, and rendered image_N+1 to obtain a rendered image for final display. In this way, some of the rendering computing power of the GPU in the first device 11 can be offloaded to multiple second devices 12.

[0177] It should be noted that the GPU performance in the second device in Figure 1D is better than that in the first device 11, or the GPU performance in the second device is comparable to that in the first device 11.

[0178] It should be noted that the rendered image_1, rendered image_2, ..., rendered image_N, and rendered image_N+1 can be complete images or parts of complete images; the rendered image obtained by fusion for final display is a complete image.

[0179] It should be noted that the second device in Figures 1A to 1D can also be called a computing center device, such as a personal computer, a home computing center device, a dedicated device, a server, etc. The first device in Figures 1A to 1D can be a terminal device, such as a mobile phone, wearable device, tablet computer, personal computer, etc.

[0180] It should be noted that the rendering API data involved in this application may include operation instructions, algorithms and data for controlling the GPU to perform rendering, such as the standard OpenGL API; the GPU needs to process several rendering API data to render one frame of image.

[0181] The following uses Figure 1A as an example to introduce specific scenarios that require sharing or unloading GPU rendering computing power.

[0182] Figure 2A is a schematic diagram of a game scene 200 according to an embodiment of this application. In Figure 2A, the mobile phone 21 is an example of the first device 11 in Figure 1A, and the personal computer 22 in Figure 2A is an example of the second device 12 in Figure 1A.

[0183] In Figure 2A, mobile phone 21 can generate rendering API data for generating game images (also known as game screens) and encode this rendering API data to obtain bitstream 1 (including the encoded rendering API data). Then, mobile phone 21 can send bitstream 1 to personal computer 22 via the network. Subsequently, personal computer 22 can decode bitstream 1 to obtain reconstructed data from the rendering API data; next, personal computer 22 can render the 3D game scene based on the reconstructed data from the rendering API data (i.e., perform a rendering operation) to obtain a game image (i.e., a rendered image); then, personal computer 22 can encode this game image to obtain bitstream 2 (including the encoded game image) and send bitstream 2 to mobile phone 21 via the network. Mobile phone 21 can decode bitstream 2 to obtain and display the reconstructed image of the game image. In this way, during the game played on mobile phone 21, all the computing power of the GPU in mobile phone 21 can be offloaded to the GPU of personal computer 22.

[0184] Figure 2B is a schematic diagram of a meeting scenario 210 according to an embodiment of this application. Figure 2B shows a VR / AR meeting scenario using Virtual Reality (VR) / Augmented Reality (AR) glasses; each participant corresponds to a digital human (which can be obtained through rendering). The VR / AR glasses 23 in Figure 2B are an example of the first device 11 in Figure 1A, and the server 24 in Figure 2B is an example of the second device 12 in Figure 1A.

[0185] In Figure 2B, VR / AR glasses 23 can generate rendering API data for generating meeting images (also known as meeting screens) and encode this rendering API data to obtain bitstream 3 (including the encoded rendering API data). Then, VR / AR glasses 23 can send bitstream 3 to server 24 via the network. Subsequently, server 24 can decode bitstream 3 to obtain reconstructed data from the rendering API data; next, server 24 can render the 3D meeting scene based on the reconstructed data from the rendering API data to obtain a meeting image (i.e., a rendered image); then, server 24 can encode this meeting image to obtain bitstream 4 (including the encoded meeting image) and send bitstream 4 to VR / AR glasses 23 via the network. VR / AR glasses 23 can decode bitstream 4 to obtain and display the reconstructed image of the meeting image. In this way, during VR / AR meetings using VR / AR glasses 23, all the computing power of the GPU in VR / AR glasses 23 can be offloaded to the GPU of server 24.

[0186] Figure 2C is a schematic diagram of a virtual concert scene 220 according to an embodiment of this application. In Figure 2C, the personal computer 25 is an example of the first device 11 in Figure 1A, and the server cluster 26 in Figure 2C is an example of the second device 12 in Figure 1A.

[0187] In Figure 2C, personal computer 25 can generate rendering API data for generating concert images (also known as concert footage) and encode this rendering API data to obtain bitstream 5 (including the encoded rendering API data). Then, personal computer 25 can send bitstream 5 to server cluster 26 via a network. Subsequently, server cluster 26 can decode bitstream 5 to obtain reconstructed data from the rendering API data; next, server cluster 26 can render the 3D concert scene based on the reconstructed data from the rendering API data to obtain concert images (i.e., rendered images); then, server cluster 26 can encode the concert images to obtain bitstream 6 (including the encoded concert images) and send bitstream 6 to personal computer 25 via a network. Personal computer 25 can decode bitstream 6 to obtain and display the reconstructed concert images. In this way, during the process of watching a concert using personal computer 25, all the computing power of the GPU in personal computer 25 can be offloaded to the GPU of server cluster 26.

[0188] Figure 2D is a schematic diagram of a digital exhibition scene 230 according to an embodiment of this application. The digital exhibition scene in Figure 2D includes examples such as a digital exhibition in a museum or a digital exhibition in a science and technology museum. The tablet computer 27 in Figure 2D is an example of the first device 11 in Figure 1A, and the desktop computer 28 in Figure 2D is an example of the second device 12 in Figure 1A.

[0189] In Figure 2D, tablet computer 27 can generate rendering API data for creating digital exhibition images (also known as digital exhibition screens) and encode this rendering API data to obtain bitstream 7 (including the encoded rendering API data). Tablet computer 27 can then send bitstream 7 to desktop computer 28 via a network. Desktop computer 28 can then decode bitstream 7 to obtain reconstructed data from the rendering API data. Next, desktop computer 28 can render the 3D exhibition scene based on the reconstructed data from the rendering API data to obtain digital exhibition images (i.e., rendered images). Desktop computer 28 can then encode these digital exhibition images to obtain bitstream 8 (including the encoded digital exhibition images) and send bitstream 8 to tablet computer 27 via a network. Tablet computer 27 can decode bitstream 8 to obtain and display the reconstructed image of the digital exhibition image. In this way, during the process of viewing a digital exhibition using tablet computer 27, all the computing power of the GPU in tablet computer 27 can be offloaded to the GPU in desktop computer 28.

[0190] It should be understood that this application may also include other specific scenarios that require sharing or offloading GPU rendering computing power, and this application does not limit this.

[0191] Figure 3A is a schematic diagram of the structure of a rendering system 300 according to an embodiment of this application. The rendering system 300 in Figure 3A can be used to implement any rendering computing power sharing scenario or any scenario where GPU rendering computing power is offloaded.

[0192] In Figure 3A, the rendering system 300 may include a first device 21 and a second device 22.

[0193] The first device 11 may include: an application program 111, a central processing unit (CPU) 112, a GPU driver 113, a GPU (114), a rendering API data encoding module 115, a communication module 116, and a video decoding module 117.

[0194] The second device 12 may include: a communication module 121, a rendering API data decoding module 122, a GPU (123), and a video encoding module 124.

[0195] Referring again to Figure 3A, the process of sharing or unloading GPU rendering computing power can be as follows: After the user inputs an operation in application 111, application 111 can respond to the user's operation by generating a corresponding execution instruction 31 and outputting the execution instruction 31 to the CPU (112). Subsequently, the CPU (112) can generate rendering API data based on the execution instruction 31 and output the rendering API data 32 to the GPU driver 113. In one possible approach, the GPU driver 113 can intercept all rendering API data 33 and output all rendering API data 33 to the rendering API data encoding module 115 (as shown in the application scenarios of Figures 1A and 1C). In another possible approach, the GPU driver 113 can intercept a portion of the rendering API data and output this portion of the rendering API data 33 to the rendering API data encoding module 115 (as shown in the application scenarios of Figures 1B and 1D); and output another portion of the rendering API data to the GPU (114).

[0196] Next, after receiving the rendering API data 33, the rendering API data encoding module 115 can encode the rendering API data 33 to obtain the rendering API data bitstream 34 and output the rendering API data bitstream 34 to the communication module 116. Subsequently, the communication module 116 can send the rendering API data bitstream 34 to the second device 12.

[0197] Subsequently, after receiving the bitstream 34 of the rendering API data, the communication module 121 of the second device 12 can output the bitstream 34 of the rendering API data to the rendering API data decoding module 122. The rendering API data decoding module 122 can decode the bitstream 34 of the rendering API data to obtain the reconstructed data 35 of the rendering API data and output the reconstructed data 35 of the rendering API data to the GPU (123). The GPU (123) can render the 3D scene based on the reconstructed data 35 of the rendering API data, generate a rendered image 36, and output the rendered image 36 to the video encoding module 124. The video encoding module 124 can encode the rendered image 36 to obtain the bitstream 37 of the rendered image and output the bitstream 37 of the rendered image to the communication module 121; the communication module 121 can send the bitstream 37 of the rendered image to the first device 11.

[0198] After receiving the rendered image bitstream 37, the communication module 116 of the first device 11 can output the rendered image bitstream 37 to the video decoding module 117. The video decoding module 117 can decode the rendered image bitstream 37 to obtain the reconstructed image 38 of the rendered image.

[0199] When the GPU driver intercepts all rendering API data and outputs it to the rendering API data encoding module 115, the video decoding module 117 can output the reconstructed image 38 of the rendered image to the application 111. Then, the application 111 can display the reconstructed image 38 of the rendered image. When the GPU driver only intercepts part of the rendering API data and outputs it to the rendering API data encoding module 115, the video decoding module 117 can output the reconstructed image 38 of the rendered image to the GPU (114); the GPU (114) can fuse the rendered image (one or more images) generated by its own rendering operation and the reconstructed image (one or more images) received from the video decoding module 117 to obtain the rendered image for final display; subsequently, the GPU (114) outputs the rendered image for final display to the application 111. Then, the application 111 can display the rendered image for final display.

[0200] It should be understood that the rendering API data encoding module 115 and the rendering API data decoding module 122 can be implemented in software, in which case they can be deployed in the CPU. Alternatively, they can be implemented in hardware, for example, using a dedicated chip. Similarly, the video encoding module 124 and the video decoding module 117 can be implemented in software, in which case they can be deployed in the CPU. Again, the video encoding and decoding modules can be implemented in hardware, for example, using a dedicated chip.

[0201] Figure 3B is a schematic diagram of the structure of a rendering API data encoding module 115 according to an embodiment of this application.

[0202] The rendering API data encoding module 115 in Figure 3B may include: a classification module 1151, a data structure decomposition module 1152, a lossy / lossless evaluation module 1153, a lossy processing module 1154, a general compression module 1155, and a dedicated compression module 1156; among which, the lossy / lossless evaluation module 1153, the lossy processing module 1154, and the dedicated encoding module 1156 are optional modules.

[0203] For example, the classification module 1151 can be used to classify the rendering API data. The data structure decomposition module 1152 can be used to decompose the rendering API data into data structures. The lossy / lossless evaluation module 1153 can be used to determine the sub-data of the rendering API data that requires lossy processing and the sub-data that does not require lossy processing. The lossy processing module 1154 can be used to perform lossy processing on the rendering API data. The general compression module 1155 can be used to compress the rendering API data using a general compression algorithm, and the dedicated compression module 1156 can be used to compress the rendering API data using a dedicated compression algorithm.

[0204] Figure 3C is a schematic diagram of the structure of a rendering API data decoding module 122 according to an embodiment of this application.

[0205] The rendering API data encoding module 122 in Figure 3C may include: a general decompression module 1221, a dedicated decoding module 1222, a lossy inverse processing module 1223, and a data reconstruction module 1224; wherein the lossy inverse processing module 1223 and the dedicated decompression module 1222 are optional modules.

[0206] For example, the general decompression module 1221 can be used to decompress the encoded rendering API data using a decompression algorithm corresponding to a general compression algorithm (which can be called a general decompression algorithm). The dedicated decompression module 1222 can be used to decompress the rendering API data using a decompression algorithm corresponding to a dedicated compression algorithm (which can be called a dedicated decompression algorithm). The lossy processing inverse processing module 1223 can be used to perform lossy processing inverse processing on the reconstructed data. The data reconstruction module 1224 can be used to reconstruct the data output by the general decompression module 1221, the dedicated decompression module 1222, and the lossy processing inverse processing module 1223 to generate reconstructed rendering API data for final rendering.

[0207] The encoding and decoding methods involved in this application can refer to encoding and decoding methods for rendering API data, with the encoding method executed by the rendering API data encoding module 115 and the decoding method executed by the rendering API data decoding module 122. The following describes the encoding and decoding process of multiple rendering API data required for GPU to render one frame of image (i.e., to generate one frame of rendered image for final display).

[0208] Figure 4 is a schematic diagram of an encoding process 400 according to an embodiment of this application. The encoding process 400 can be executed by the rendering API data encoding module 115 of the first device.

[0209] S401, obtain multiple rendering application programming interface (API) data, which are used to generate rendered images.

[0210] For example, the rendering API data encoding module 115 of the first device obtains multiple rendering API data from the GPU driver of the first device, which may be all or part of the rendering API data required to generate the rendered image for final display.

[0211] For example, a rendering API data can be an operation instruction, a piece of data, or a set of algorithms. For example, the data included in the rendering API data can be such as texture data, mesh data, etc.; the algorithms included in the rendering API data (also known as rendering operation logic) can be such as geometry processing, shading, lighting calculation, etc.; the operation instructions included in the rendering API data can be such as translation operations, rotation operations, etc.

[0212] Mesh data: A data structure used in computer graphics to model various irregular objects. It can be used to represent the shape and structure of three-dimensional objects. Mesh data can include vertex coordinates, normals, topology, etc.

[0213] Texture data: Texture data can be used to describe the details and texture of an image or object's surface (or, in other words, texture data can be used to describe the texture of an image or object's surface). Texture data can be viewed as an image, which can be used in computer graphics to attach to the surface of a 3D model, increasing the realism and detail of the graphics. For example, texture data can include the color information of vertices on a 3D model (such as RGB, YUV, etc.); texture data can also include the material information of the 3D model, such as roughness, normal information, ambient light information, etc.

[0214] Rendering Computation Logic: Algorithms that control and guide various calculations and operations during the rendering process, determining how a 3D scene is converted into a 2D image.

[0215] S402, retrieves the type of each rendering API data from multiple rendering API data sets.

[0216] For example, for each of the multiple rendering API data sets, the rendering API data encoding module 115 can perform semantic analysis to determine the semantics of each rendering API data set. For instance, through semantic analysis, it can be determined that the semantics of rendering API data 1 is a translation operation, the semantics of rendering API data 2 is a rotation operation, the semantics of rendering API data 3 is roughness, the semantics of rendering API data 4 is ambient light information, the semantics of rendering API data 5 is the mesh data of object A, the semantics of rendering API data 6 is the mesh data of object B, and so on.

[0217] Next, the rendering API data encoding module 115 can determine the type of each rendering API data based on its semantics. For example, rendering API data with the same or similar semantics can be grouped into one category; the specific process of determining the type of each rendering API data among multiple rendering API data will be explained later.

[0218] S403: Based on the type of each rendering API data in multiple rendering API data sets, obtain the encoding method corresponding to each rendering API data in multiple rendering API data sets.

[0219] In one possible implementation, the encoding method corresponding to each type can be pre-set; in this way, the encoding method corresponding to each rendering API data can be determined according to the pre-set encoding method corresponding to each type and the type of each rendering API data.

[0220] In one possible implementation, the impact of each rendering API data on the quality of the rendered image and the data volume of each rendering API data can be analyzed based on the type of each rendering API data. Then, based on the impact of each rendering API data on the quality of the rendered image and the data volume of each rendering API data, the encoding method corresponding to each rendering API data can be determined.

[0221] For example, the rendering API data encoding module 115 can obtain the encoding method corresponding to each rendering API data in the total data of these multiple rendering API data sets. Alternatively, the rendering API data encoding module 115 can obtain the encoding method corresponding to each rendering API data in the partial data of these multiple rendering API data sets.

[0222] For example, two rendering API data of the same type can have the same encoding method, while two rendering API data of different types can have different encoding methods.

[0223] For example, the encoding method may include the compression algorithm used for compression. Optionally, the encoding method may also include whether to refer to data in the cache queue for compression. Optionally, the encoding method may also include preprocessing methods before compression and / or post-processing methods after compression.

[0224] For example, the encoding methods involved in this application may include lossy encoding methods and lossless encoding methods, wherein the encoding method corresponding to at least one type of rendering API data is a lossy encoding method.

[0225] S404 encodes each rendering API data in multiple rendering API data based on the encoding method corresponding to each rendering API data in multiple rendering API data to obtain a bitstream.

[0226] Next, the rendering API data encoding module 115 can encode each piece of rendering API data using the encoding method corresponding to each piece of rendering API data to obtain a bitstream. This bitstream can refer to bitstream 34, bitstream 1, bitstream 3, bitstream 5, or bitstream 7 of the aforementioned rendering API data.

[0227] Each rendering API data point is encoded to obtain encoded rendering API data (also known as encoded data of each rendering API data point). Then, multiple encoded rendering API data points (also known as encoded data of multiple rendering API data points) are used to generate the bitstream. Furthermore, bitstream description information can be written (or packaged) into the bitstream. This description information may include, but is not limited to: information describing the rendering API data (such as the type of rendering API data, the number of rendering API data points, etc.), information describing the compression algorithm, information describing the preprocessing method, information describing the post-processing method, and information describing whether data in the cache queue is referenced for compression.

[0228] In other words, this application takes into account the types of rendering API data and encodes the rendering API data based on its type during the encoding process. Specifically, this application uses corresponding encoding methods for different types of rendering API data. Since data of the same type has a high degree of structural consistency, the encoding method of this application can reduce redundancy in different types of rendering API data, thereby improving the compression rate of the rendering API data.

[0229] In addition, at least one type of rendering API data in this application is encoded using a lossy encoding method; this can reduce the compression rate of rendering API data with a lossy encoding method.

[0230] For example, after the first device generates a code stream according to the above encoding process 400, it can send the code stream to a system for transmitting the code stream. The system for transmitting the code stream may include a transceiver module and a storage module; wherein the storage module is used to store the code stream; and the transceiver module is used to transmit the code stream.

[0231] For example, after the first device generates a code stream according to the above encoding process 400, it can send the code stream to a system for transmitting the code stream. The system for transmitting the code stream may include a transceiver module and a storage module; wherein, the storage module is used to store the code stream; and the transceiver module is used to send the code stream.

[0232] In this way, the second device can receive the bitstream transmitted or sent by the system through the transmitted bitstream. After receiving the bitstream, the second device can decode the bitstream, and the decoding process can refer to the following decoding process 500.

[0233] Figure 5 is a schematic diagram of a decoding process 500 according to an embodiment of this application. The decoding process 500 in Figure 5 corresponds to the encoding process 400 in Figure 4, and the decoding process 500 can be executed by the rendering API data decoding module 122.

[0234] S501, receiving the bitstream.

[0235] For example, the rendering API data decoding module 122 can receive a bitstream. This bitstream may include multiple encoded rendering API data entries and bitstream description information. The bitstream description information may include a first identifier indicating the type of each of the multiple rendering API data entries. It should be understood that the bitstream description information may also include other bitstream description information, which will be explained later.

[0236] S502, parse the bitstream to obtain multiple encoded rendering API data and a first identifier, which describes the type of each rendering API data in the multiple rendering API data.

[0237] For example, after receiving the bitstream, the rendering API data decoding module 122 can parse the bitstream to obtain multiple encoded rendering API data and a first identifier. It should be understood that parsing the bitstream can also obtain other bitstream description information, which will be explained later.

[0238] S503, based on the first identifier, determines the decoding method corresponding to each encoded rendering API data.

[0239] For example, the first device and the second device can pre-synchronize the encoding method (or decoding method) corresponding to each type; in this way, the rendering API data decoding module 122 can determine the decoding method corresponding to each encoded rendering API data based on the first identifier.

[0240] For example, the decoding method corresponds to the encoding method. The decoding method may include the decompression algorithm used for decompression. Optionally, the decoding method may also include whether to refer to data in the cache queue for decompression. Optionally, the decoding method may also include a preprocessing method before decompression (corresponding to post-processing after compression, which can be seen as the inverse of post-processing after compression) and / or a post-processing method after decompression (corresponding to preprocessing before compression, which can be seen as the inverse of preprocessing before compression).

[0241] S504, based on the decoding method corresponding to each encoded rendering API data, decode each encoded rendering API data to obtain the reconstructed data of each rendering API data. The reconstructed data of multiple rendering API data are used to generate a rendered image. There is a difference between the reconstructed data of at least one type of rendering API data and the reconstructed data of at least one type of rendering API data.

[0242] Subsequently, for each encoded rendering API data, the rendering API data decoding module 122 can decode the encoded rendering API data based on the decoding method corresponding to that rendering API data to obtain the reconstructed data of that rendering API data; after decoding all the encoded rendering API data, the reconstructed data of multiple rendering API data can be obtained; the reconstructed data of these multiple rendering API data can be used to generate a rendered image.

[0243] It should be understood that during the encoding process of the first device, the encoding method corresponding to at least one type of rendering API data is a lossy encoding method; correspondingly, the reconstructed data of at least one type of rendering API data obtained by the decoding end differs from the rendering API data of that at least one type.

[0244] It should be understood that when the bitstream sent by the first device encodes only the portion of the rendering API data required to generate the rendered image for final display, the reconstructed data from the multiple rendering API data points decoded by the second device can generate a portion of the rendered image for final display. Conversely, when the bitstream sent by the first device encodes all the rendering API data required to generate the rendered image for final display, the reconstructed data from the multiple rendering API data points decoded by the second device can generate the rendered image for final display.

[0245] Subsequently, the rendering API data decoding module 122 of the second device can output the reconstructed data of the multiple rendering API data to the GPU of the second device; the GPU of the second device renders the three-dimensional scene based on the reconstructed data of the multiple rendering API data (i.e., performs a rendering operation) to obtain a rendered image and sends the rendered image to the first device, which displays the rendered image (or the first device merges the rendered image received from the second device with the rendered image generated by the local GPU and displays it).

[0246] The following example, shown in Figure 1A, illustrates the scenario where the first device shares the rendering power of a second device, and where all the rendering power of the GPU in the first device is unloaded.

[0247] Figure 6 is a schematic diagram of another encoding process 600 according to an embodiment of this application. The encoding process 600 in Figure 6 describes the process of encoding all the rendering API data required to generate the rendered image for final display; the encoding process 600 can be executed by the rendering API data encoding module 115 of the first device.

[0248] S601 retrieves multiple rendering application programming interface (API) data, which are used to generate rendered images.

[0249] For example, in the encoding process 600, the rendering API data encoding module 115 of the first device obtains multiple rendering API data from the GPU driver of the first device, which are all the rendering API data required to generate the rendered image for final display.

[0250] S602, based on the semantics of each rendering API data in the multiple rendering API data, classify the multiple rendering API data to obtain the type of each rendering API data in the multiple rendering API data.

[0251] For example, data classification rules can be pre-defined. For example, the data classification rules could be to classify multiple rendering API data based on their semantic types. For instance, the data classification rules could be: classifying rendering API data whose semantics are function names into one category, called function type; classifying rendering API data whose semantics are rendering operation logic into one category, called operation logic type; classifying rendering API data whose semantics are texture data of the rendered object into one category, called texture data type; classifying rendering API data whose semantics are mesh data of the rendered object into one category, called mesh data type; and classifying rendering API data whose semantics are other (semantics other than function names, rendering operation logic, texture data of the rendered object, and mesh data of the rendered object) into another category, called other types.

[0252] It should be understood that multiple rendering API data can also be divided into other types or more granular types, and this application does not impose any restrictions on this.

[0253] Thus, for each rendering API data, the rendering API data encoding module 115 can determine the type of the rendering API data based on its semantics and data classification rules. Then, the rendering API data encoding module 115 can obtain the encoding method corresponding to each of the multiple rendering API data based on its type. The rendering API data encoding module 115 can obtain the encoding method corresponding to each of the multiple rendering API data based on its type and preset conditions. Specifically, different preset conditions can be pre-set with corresponding encoding methods, thus determining the encoding method corresponding to each rendering API data based on the preset conditions it meets; see S603-S604 below:

[0254] S603, based on the type of each rendering API data in the multiple rendering API data and the first preset type, determine the first rendering API data in the multiple rendering API data.

[0255] S604, based on the type of each rendering API data in multiple rendering API data and the second preset type, determine the second rendering API data in the multiple rendering API data.

[0256] For example, the aforementioned preset conditions may include a first preset type and a second preset type. The encoding method corresponding to the first preset type may be set to the first encoding method, and the encoding method corresponding to the second preset type may be set to the second encoding method.

[0257] For example, the rendering API data encoding module 115 can identify rendering API data of type 1 preset type from multiple rendering API data as first rendering API data; and identify rendering API data of type 2 preset type from multiple rendering API data as second rendering API data.

[0258] In one possible approach, the first preset type includes an arithmetic logic type, a texture data type, or a mesh data type; the second preset type includes a function type or other types. In this way, rendering API data of type arithmetic logic type, texture data type, or mesh data type from multiple rendering API data sets can be identified as the first rendering API data; and rendering API data of type function type or other types from multiple rendering API data sets can be identified as the second rendering API data.

[0259] In one possible approach, the first preset type includes a texture data type or a mesh data type; the second preset type includes a function type, an arithmetic logic type, or other types. In this way, rendering API data of type texture data type or mesh data type from among multiple rendering API data can be identified as the first rendering API data; and rendering API data of type arithmetic logic type, function type, or other types from among multiple rendering API data can be identified as the second rendering API data.

[0260] The first rendering API data consists of one or more records, and the second rendering API data consists of one or more records. In other words, the multiple rendering API data records acquired by S601 include one or more records of the first rendering API data and one or more records of the second rendering API data.

[0261] In this way, the rendering API data encoding module 115 can determine that the encoding method corresponding to the first rendering API data is the first encoding method, which includes data structure decomposition and the second encoding method; and can determine that the encoding method corresponding to the second rendering API data is the third encoding method.

[0262] For example, data structure decomposition can also be called data structure parsing, which is used to obtain data structure information such as fields, semantics of fields, data length of fields, and data characteristics of fields for each rendered API data.

[0263] For example, the second encoding method and the third encoding method are different.

[0264] S605, based on the first encoding method corresponding to each first rendering API data, encode each first rendering API data to obtain each encoded first rendering API data.

[0265] For example, for each first rendering API data, the rendering API data encoding module 115 can use the first encoding method corresponding to the first rendering API data to encode the first rendering API data to obtain the encoded first rendering API data; in this way, multiple encoded first rendering API data can be obtained.

[0266] S606, based on the third encoding method corresponding to each second rendering API data, encodes each second rendering API data to obtain each encoded second rendering API data.

[0267] For example, for each piece of second rendering API data, the rendering API data encoding module 115 can use the third encoding method corresponding to that piece of second rendering API data to encode the piece of second rendering API data to obtain the encoded piece of second rendering API data; in this way, multiple encoded pieces of second rendering API data can be obtained.

[0268] S607 generates a bitstream based on multiple encoded first rendering API data, multiple encoded second rendering API data, and a first identifier.

[0269] For example, multiple encoded first rendering API data, multiple encoded second rendering API data, and a first identifier can be packaged into a bitstream, where the first identifier indicates the type of each rendering API data.

[0270] In other words, in the process of encoding rendering API data, this application uses the same encoding method to encode rendering API data of the same type based on the semantics of the rendering API data. Since the structural consistency of data of the same type is high, the encoding method of this application can reduce the redundancy of different types of rendering API data, thereby improving the compression rate of different types of rendering API data and thus improving the overall data compression rate.

[0271] Figure 7 is a schematic diagram of another decoding process 700 according to an embodiment of this application. The decoding process 700 in Figure 7 corresponds to the encoding process 600 in Figure 6, and the decoding process 700 can be executed by the rendering API data decoding module 122.

[0272] S701, receiving the bitstream.

[0273] For example, the bitstream may include multiple encoded rendering API data and bitstream description information, the bitstream description information may include a first identifier, which specifically indicates the type of each rendering API data.

[0274] S702, parse the bitstream to obtain multiple encoded rendering API data and a first identifier, the first identifier specifically indicating the type of each rendering API data.

[0275] S703, based on the first identifier and the first preset type, determines the encoded first rendering API data.

[0276] For example, the first device and the second device can pre-synchronize data classification rules, a first preset type, and a second preset type. In this way, the rendering API data decoding module 122 can determine the encoded first rendering API data from multiple encoded rendering API data based on the data classification rules, the first identifier, and the first preset type.

[0277] S704, based on the first identifier and the second preset type, determines the encoded second rendering API data.

[0278] For example, the rendering API data decoding module 122 can determine the second rendering API data from multiple encoded rendering API data based on data classification rules, a first identifier, and a second preset type.

[0279] For example, the first device and the second device can pre-synchronize the encoding method corresponding to the first rendering API data and the encoding method corresponding to the second rendering API data. In this way, the rendering API data decoding module 122 can determine that the decoding method corresponding to the first rendering API data is the first decoding method, and determine that the decoding method corresponding to the second rendering API data is the second decoding method.

[0280] For example, the first decoding method corresponds to the second encoding method described above; that is, the first decoding method is the decoding method corresponding to the second encoding method described above.

[0281] For example, the second decoding method corresponds to the third encoding method described above; that is, the second decoding method is the decoding method corresponding to the third encoding method described above.

[0282] S705, based on the first decoding method corresponding to each encoded first rendering API data, decodes each encoded first rendering API data to obtain the reconstructed data of each first rendering API data.

[0283] For example, for each encoded first rendering API data, the rendering API data decoding module 122 can decode the encoded first rendering API data based on the first decoding method corresponding to the first rendering API data to obtain the reconstructed data of the first rendering API data; in this way, the reconstructed data of multiple first rendering API data can be obtained.

[0284] S706, based on the second decoding method corresponding to each encoded second rendering API data, decodes each encoded second rendering API data to obtain the reconstructed data of each second rendering API data.

[0285] For example, for each encoded second rendering API data, the rendering API data decoding module 122 can decode the encoded second rendering API data based on the second decoding method corresponding to the second rendering API data to obtain the reconstructed data of the second rendering API data; in this way, the reconstructed data of multiple second rendering API data can be obtained.

[0286] The reconstructed data from multiple first rendering API data sets and multiple reconstructed data sets from multiple second rendering API data sets can be combined to form the reconstructed data of multiple rendering API data sets. Then, the GPU of the second device renders the 3D scene based on this reconstructed data of multiple rendering API data sets (i.e., performs a rendering operation) to obtain a rendered image, which is then sent to the first device for display.

[0287] For example, the second encoding method includes at least one of the following: lossy processing and compression using a general compression algorithm; compression using a dedicated compression algorithm; and compression based on reconstructed data from encoded rendering API data in a cache queue. The dedicated compression algorithm can refer to a lossy or lossless compression algorithm that can only be used to compress specific types of data. For example, dedicated compression algorithms can include various types, such as JPEG compression algorithm (for compressing images), H.264 compression algorithm (for compressing images), H.265 compression algorithm (for compressing images), Draco dedicated compression algorithm (for compressing mesh data), and V-mesh dedicated compression algorithm (for compressing mesh data). In other words, the second encoding method can include multiple types.

[0288] For example, lossy processing can be used to reduce data volume, data complexity, and data entropy to improve data consistency; lossy processing can include, but is not limited to, quantization and clustering. It should be understood that lossy processing methods can include a variety of approaches.

[0289] For example, the third encoding method includes compression using a general compression algorithm. The general compression algorithm can refer to a lossless compression algorithm that can be used to compress all types of data (including other types besides the rendering API data types involved in this application). For example, the general compression algorithm can include multiple algorithms, such as LZ77 compression, Deflate compression, Zstd compression, etc. That is, the third encoding method can include multiple methods, and one third encoding method can correspond to one general compression algorithm.

[0290] It should be noted that the first encoding methods corresponding to any two first rendering API data can be the same or different; the third encoding methods corresponding to any two second rendering API data can be the same or different.

[0291] For example, the first decoding method may include at least one of the following: inverse processing of lossy processing and decompression using a decompression algorithm corresponding to a general compression algorithm; decompression using a decompression algorithm corresponding to a special compression algorithm; and decompression based on reconstructed data of the decoded rendering API data in the cache queue.

[0292] For example, the second decompression method may include decompression using a decompression algorithm corresponding to a general compression algorithm.

[0293] For example, the inverse processing of lossy processing is used to restore the data; whereby lossy processing may include, but is not limited to, inverse quantization processing, inverse clustering processing, etc. Decompression algorithms corresponding to specialized compression algorithms may include, but are not limited to, the decompression algorithms corresponding to JPEG compression, H.264 compression, H.265 compression, Draco compression, and V-mesh compression. Decompression algorithms corresponding to general compression algorithms may include, for example, the decompression algorithms corresponding to LZ77 compression, Deflate compression, and Zstd compression.

[0294] Figure 8A is a schematic diagram of the compression, decompression, and rendering process of rendering API data according to an embodiment of this application. In Figure 8A, the second encoding method is lossy processing and compression using a general compression algorithm, and the first decoding method is decompression using the decompression algorithm corresponding to the general algorithm.

[0295] Referring to Figure 8A, the encoding process of rendering API data can be as follows: After classifying the rendering API data, the first device can directly perform general compression on a portion of the data to obtain the encoded portion of the data; the other portion of the data undergoes data structure decomposition and loss processing before being subjected to general compression to obtain the encoded portion of the data; then the two portions of encoded data, the classification information involved in the classification process, and the compression information involved in the compression process are packaged into a bitstream and sent.

[0296] The decoding process of the rendering API data can be as follows: The second device receives and parses the bitstream, obtaining encoded data, compressed information, and classification information. Next, the encoded data is decompressed using the compressed and classification information. Then, a portion of the decompressed data undergoes inverse loss processing based on the compressed information. Finally, the inversely processed data and the remaining decompressed data are reconstructed using the classification information to obtain the reconstructed rendering API data. Subsequently, the reconstructed rendering API data can be used to render the 3D scene, resulting in a rendered image.

[0297] Based on Figure 8A, the encoding and decoding process of the first rendering API data will be explained in more detail below.

[0298] Figure 8B is a schematic diagram of another encoding process 800 according to an embodiment of this application. In the encoding process 800 in Figure 8B, the first encoding method is lossy processing and compression using a general compression algorithm; the encoding process 800 can be executed by the rendering API data encoding module 115 of the first device.

[0299] S801 retrieves multiple rendering application programming interface (API) data, which are used to generate rendered images.

[0300] S802, based on the semantics of each rendering API data in the multiple rendering API data, classifies the multiple rendering API data to obtain the type of each rendering API data in the multiple rendering API data.

[0301] S803, based on the type of each rendering API data in the multiple rendering API data and the first preset type, determine the first rendering API data in the multiple rendering API data.

[0302] S804, based on the type of each rendering API data in multiple rendering API data and the second preset type, determines the second rendering API data in the multiple rendering API data.

[0303] It should be noted that the first preset type involved in the encoding process 800 includes arithmetic logic type, texture data type or mesh data type, and the second preset type includes function type or other types.

[0304] For example, S801 to S804 can be referred to the description of S601 to S604 above, and will not be repeated here.

[0305] Subsequently, the rendering API data encoding module 115 can determine that the encoding method corresponding to the first rendering API data is lossy processing and compression using a general compression algorithm, and determine that the encoding method corresponding to the second rendering API data is compression using a general compression algorithm.

[0306] The process of encoding the first rendering API data (i.e., rendering API data of type arithmetic logic type, texture data type, or mesh data type) can be referred to as follows: S805~S809:

[0307] S805 performs data structure decomposition on each first rendering API data to obtain the data structure information of each first rendering API data.

[0308] For example, the format of a certain first rendering API data is shown in Table 1, and the data description information is shown in Table 2:

[0309] Table 1

[0310] Table 2

[0311] In Table 1, pos corresponds to in_POSITION0 in Table 2; in Table 1, normal corresponds to in_NORMAL0 in Table 2; in Table 1, color corresponds to in_COLOR0 in Table 2; in Table 1, tex corresponds to in_TEXCOORD0 in Table 2; and in Table 1, tan corresponds to in_TANGENT0 in Table 2.

[0312] The rendering API data encoding module 115 performs data structure decomposition on the first rendering API data. The resulting data structure information of the first rendering API data can be as follows: The first rendering API data includes 5 variables (or 5 fields or 5 sub-data). These 5 variables occupy a total of 28 bytes. Bytes 0-7 belong to POSITION (pos), bytes 8-15 belong to NORMAL (normal), bytes 16-19 belong to COLOR (color), bytes 20-23 belong to TEXCOORD (tex), and bytes 24-27 belong to TANGENT (tan). The data of POSITION, NORMAL, and TEXCOORD are float type data.

[0313] S806, based on the data structure information of each first rendering API data, determine the sub-data in each first rendering API data that meets the first preset condition.

[0314] In one possible approach, the first preset condition can be set in advance according to the amount of data and / or the degree of impact on the quality of the rendered image.

[0315] For example, the first preset condition is float type data. Thus, for the first rendering API data in Table 1, the rendering API data encoding module 115 can determine that the sub-data that satisfies the first preset condition is the data included in the three fields POSITION, NORMAL, and TEXCOORD.

[0316] For example, the first preset condition is a specific field; where different types of first rendering API data correspond to different specific fields; for example, the specific field corresponding to the first rendering API data in Table 1 can be POSITION. In this way, the rendering API data encoding module 115 can determine the sub-data that satisfies the first preset condition for the first rendering API data in Table 1 as the data included in the POSITION field.

[0317] In one possible approach, the first preset condition could be that the data volume is greater than a first threshold, and / or that the impact on the quality of the rendered image is less than a second threshold. The impact on the quality of the rendered image can be represented by the distortion determined between a rendered image obtained using this data and a rendered image obtained without using this data. The first and second thresholds can be set as needed, and this application does not impose any restrictions on them.

[0318] For example, regarding the first rendering API data in Table 1, if the rendering API data encoding module 115 determines that the impact of TEXCOORD on the quality of the rendered image is less than the second threshold, then the rendering API data encoding module 115 can determine that the sub-data satisfying the first preset condition is the data included in the TEXCOORD field. As another example, regarding the first rendering API data in Table 1, if the rendering API data encoding module 115 determines that the amount of POSITION data is greater than the first threshold, then the rendering API data encoding module 115 can determine that the sub-data satisfying the first preset condition is the data included in the POSITION field.

[0319] S807, perform lossy processing on each sub-data in the first rendering API data that meets the first preset condition to obtain lossy sub-data. The lossy processing includes quantization processing or clustering processing.

[0320] For example, the lossy processing performed by the rendering API data encoding module 115 on different sub-data that meet the first preset condition in the same first rendering API data can be the same or different. For example, the sub-data that meet the first preset condition in the first rendering API data in Table 1 includes the data contained in the three fields POSITION, NORMAL, and TEXCOORD. The lossy processing performed on the data contained in the POSITION field can be clustering, and the lossy processing performed on the data contained in the NORMAL and TEXCOORD fields can be quantization (such as converting float data to integer data). Alternatively, the lossy processing performed on the data contained in the three fields POSITION, NORMAL, and TEXCOORD can all be either quantization or clustering.

[0321] For example, the lossy processing performed by the rendering API data encoding module 115 on sub-data that meet the first preset condition in different types of first rendering API data can be the same or different. For instance, the lossy processing performed on sub-data that meet the first preset condition in first rendering API data of texture type is clustering, and the lossy processing performed on sub-data that meet the first preset condition in first rendering API data of operation logic type is quantization. As another example, the lossy processing performed on sub-data that meet the first preset condition in first rendering API data of texture type and sub-data that meet the first preset condition in first rendering API data of operation logic type are both clustering or quantization.

[0322] In this way, by performing lossy processing on the first rendering API data, the consistency of the first rendering API data can be increased.

[0323] S808 uses a general compression algorithm to compress lossy sub-data to obtain encoded sub-data.

[0324] For example, the rendering API data encoding module 115 may use the same or different general compression algorithms to compress different lossy sub-data within the same first rendering API data. For instance, if the sub-data in the first rendering API data in Table 1 that meets the first preset condition includes data from the fields POSITION, NORMAL, and TEXCOORD, then after lossy processing of the data in the POSITION field, the LZ77 compression algorithm can be used; after lossy processing of the data in the NORMAL field, the Deflate compression algorithm can be used; and after lossy processing of the data in the TEXCOORD field, the Zstd compression algorithm can be used. Alternatively, after lossy processing of the data in the POSITION, NORMAL, and TEXCOORD fields, the LZ77 compression algorithm can be used for all of them.

[0325] For example, the rendering API data encoding module 115 may use the same or different general compression algorithms for compressing lossy sub-data in different types of first rendering API data. For instance, after lossy processing of sub-data in texture data first rendering API data that meets the first preset condition, the LZ77 compression algorithm may be used; after lossy processing of sub-data in operational logic type first rendering API data that meets the first preset condition, the Deflate compression algorithm may be used. As another example, after lossy processing of sub-data in texture data first rendering API data that meets the first preset condition, and after lossy processing of sub-data in operational logic type first rendering API data that meets the first preset condition, the LZ77 compression algorithm may be used for both.

[0326] The encoded sub-data obtained after executing S808 is essentially encoded sub-data that satisfies the first preset condition (or encoded data of sub-data that satisfies the first preset condition).

[0327] S809, a general compression algorithm is used to compress the sub-data in each first rendering API data that does not meet the first preset condition, so as to obtain the encoded sub-data.

[0328] For example, for each sub-data in the first rendering API data that does not meet the first preset condition, the rendering API data encoding module 115 may not perform lossy processing on these sub-data, but instead directly use a general compression algorithm to compress these sub-data.

[0329] It should be understood that the general compression algorithm used by the rendering API data encoding module 115 to compress different sub-data that do not meet the first preset condition in the same first rendering API data can be the same or different. Similarly, the general compression algorithm used by the rendering API data encoding module 115 to compress sub-data that do not meet the first preset condition in different types of first rendering API data can be the same or different.

[0330] The encoded sub-data obtained after executing S809 is essentially encoded sub-data that does not meet the first preset condition (or encoded data of sub-data that does not meet the first preset condition).

[0331] S810 uses a general compression algorithm corresponding to each second rendering API data to encode each second rendering API data to obtain each encoded second rendering API data.

[0332] For example, for each piece of second rendering API data, the rendering API data encoding module 115 may not perform lossy processing on the second rendering API data, but instead directly use a general compression algorithm to compress the second rendering API data.

[0333] It should be understood that the general compression algorithm used by the rendering API data encoding module 115 to compress different types of second rendering API data may be the same or different.

[0334] S811 generates a bitstream based on multiple encoded first rendering API data, multiple encoded second rendering API data, classification information, and compression information.

[0335] For example, after executing S802, the rendering API data encoding module 115 can generate classification information for the rendering API data. This classification information may include, but is not limited to: frame identifier (frame ID), total number of rendering API data entries, total number of rendering API data types, a first identifier (used to indicate the type of rendering API data, also known as a classification enumeration value, with values ​​ranging from 0 to 4, representing the aforementioned five types of rendering API data), and the data length of each type of rendering API data.

[0336] It should be noted that a single rendering API data entry can correspond to a single first identifier, or multiple rendering API data entries of each type can correspond to a single first identifier. The value of the first identifier differs for different types of rendering API data.

[0337] For example, after executing S808 to S810, the rendering API data encoding module 115 can generate compression information corresponding to the rendering API data. This compression information may include: a second identifier (used to indicate the compression algorithm, also known as an enumeration value of the compression method, with values ​​ranging from 0 to 2, representing the LZ77 compression algorithm, Deflate compression algorithm, and Zstd compression algorithm respectively), a third identifier (used to indicate whether the rendering API data has undergone lossy processing; this third identifier can also be called a lossy identifier, with values ​​of 0 or 1, where 0 indicates no lossy processing and 1 indicates lossy processing), and lossy processing parameters (such as the quantization level and clustering threshold). It should be understood that when the encoding process 800 involves more general compression algorithms, the second identifier will have more values ​​(or, in other words, the enumeration value of the compression method will have more values).

[0338] It should be noted that each sub-data (data contained in a field) in a rendering API data corresponds to a second identifier, or a rendering API data corresponds to a second identifier (all sub-data contained in that rendering API data corresponds to a second identifier), or multiple rendering API data included in each type of rendering API data correspond to a second identifier.

[0339] It should be noted that each sub-data (data contained in a field) in a rendering API data corresponds to a third identifier, or a rendering API data corresponds to a third identifier (all sub-data contained in that rendering API data corresponds to a third identifier), or multiple rendering API data included in each type of rendering API data correspond to a third identifier.

[0340] It should be noted that each sub-data (data contained in a field) in a rendering API data corresponds to a set of lossy processing parameters, or a rendering API data corresponds to a set of lossy processing parameters (all sub-data contained in this rendering API data correspond to a set of lossy processing parameters), or multiple rendering API data included in each type of rendering API data correspond to a set of lossy processing parameters.

[0341] Subsequently, the rendering API data encoding module 115 can package the encoded multiple first rendering API data, the encoded multiple second rendering API data, classification information, and compression information into a bitstream.

[0342] For example, the encoded first rendering API data and the encoded second rendering API data output by the rendering API data encoding module 115 can form a raw stream, also known as a payload. In one possible approach, the rendering API data encoding module 115 can write classification information and compression information into the extended fields (such as Supplemental Enhancement Information, SEI) of the standard bitstream. In another possible approach, the rendering API data encoding module 115 can write classification information and compression information into the header or trailer of the raw stream. In yet another possible approach, the communication module can also carry the classification information and compression information in a transport protocol (such as a Real-time Transport Protocol, RTP). For example, the classification information and compression information can be encapsulated in the RTP extended header, specifying that the extended type is classification information and compression information. Yet another example is that the classification information and compression information can be encapsulated in the RTP payload, with the RTP version number or syntax information identifying that the RTP payload carries classification information and compression information. For example, classification and compression information can be carried in file formats (such as the file format specified by standard ISO14496-12, or the MPEG2TS file format).

[0343] For example, classification information can be added to the header of each type of encoded rendering API data, or to the header of the encoded whole-frame rendering API data (i.e., the rendering API data required by the GPU to render a frame of image); compression information can be added to the header of each type of encoded rendering API data, or to the header of the whole-frame rendering API data, or to the header of each encoded piece of rendering API data.

[0344] Figure 8C is a schematic diagram of a bitstream structure according to an embodiment of this application.

[0345] In Figure 8C, the bitstream of the full-frame rendering API data includes AF_header and AF_payload; AF_header is the bitstream header, where AF stands for Application Programming Interface Forwarding. AF_header is the bitstream header of the full-frame rendering API data, and AF_payload is the payload in the bitstream of the full-frame rendering API data.

[0346] The AF_header contains the following syntax elements: FrameID (frame identifier), API_num (total number of API data entries), Cat_num (total number of API data types), Cat_type (first identifier or category enumeration value) (not shown in Figure 8C), and Buf_len (data length of each type of API data) (not shown in Figure 8C).

[0347] Among them, AF_payload includes Cmd_load (the payload of rendering API data of function type), Cal_load (the payload of rendering API data of operation logic type), Tex_load (the payload of rendering API data of texture data type), Mesh_load (the payload of rendering API data of mesh data type) (not shown in Figure 8C), and Other_load (the payload of rendering API data of other types) (not shown in Figure 8C).

[0348] Cmd_load includes Enc_type (second identifier or compression method enumeration value), Demage (lossy flag or third identifier), QL (quantization level) (not shown in Figure 8C), ClusOff (clustering threshold) (not shown in Figure 8C), and enc_load (raw stream or standard compressed bitstream, i.e., encoded function type rendering API data).

[0349] For example, the syntax of the bitstream of the full-frame rendering API data can be illustrated in Table 3 below:

[0350] Table 3

[0351] In Table 3, frame_id is equivalent to FrameID in Figure 8C, api_total_num is equivalent to API_num in Figure 8C, category_num is equivalent to Cat_num in Figure 8C, category_type is equivalent to Cat_type, and buffer_length is equivalent to Buf_len.

[0352] In Table 3, enc_type is equivalent to Enc_type in Figure 8C, demage_flag is equivalent to Demage in Figure 8C, quantization_level is equivalent to QL, and clustering_offset is equivalent to ClusOff.

[0353] In this way, by performing lossy processing on some types of rendering API data, the consistency of this part of the rendering API data can be improved, thereby increasing the compression rate of this part of the rendering API data, which in turn improves the compression rate of the entire frame rendering API data and reduces the amount of data transmitted.

[0354] [Corrected according to Rule 91 10.09.2025] Figure 9 is a schematic diagram of another decoding process 900 according to an embodiment of this application. The decoding process 900 in Figure 9 corresponds to the encoding process 800 in Figure 8B. The first decoding method in the decoding process 900 is to decompress using the decompression algorithm corresponding to the general algorithm. The decoding process 900 can be executed by the rendering API data decoding module 122.

[0355] S901, receiving the bitstream.

[0356] For example, when the rendering API data encoding module 115 of the first device encodes multiple rendering API data according to the above encoding process 800 to obtain a bitstream, the bitstream received by the rendering API data decoding module 122 of the second device may include the encoded multiple rendering API data, classification information and compression information.

[0357] S902, parse the bitstream to obtain multiple encoded rendering API data, classification information, and compression information; wherein, the classification information includes a first identifier, and the compression information includes a second identifier, a third identifier, and lossy processing parameters.

[0358] For example, the rendering API data decoding module 122 parses the bitstream and can obtain multiple encoded rendering API data, classification information and compression information; wherein, the classification information may include a first identifier, and the compression information may include a second identifier, a third identifier and lossy processing parameters.

[0359] S903, based on the first identifier and the first preset type, determines the encoded first rendering API data.

[0360] For example, S903 can be described with reference to the above description of S703, and will not be repeated here.

[0361] Subsequently, the rendering API data encoding module 115 can determine that the encoding method corresponding to the first rendering API data is the first encoding method; wherein, the determination of the specific first encoding method corresponding to the first rendering API data and the decoding process can be referred to as follows S904 to S907:

[0362] S904, based on the second identifier, determines the general decompression algorithm corresponding to the general compression algorithm for each first rendering API data.

[0363] S905, based on the general decompression algorithm corresponding to the general compression algorithm corresponding to each first rendering API data, decompresses each encoded first rendering API data to obtain the reconstructed data of each first rendering API data.

[0364] For example, after the rendering API data decoding module 122 determines the first encoded rendering API data among the multiple encoded rendering API data based on the first identifier, it can determine that the decoding method corresponding to the first encoded rendering API data is the first decoding method; then, the rendering API data decoding module 122 can further decompress each encoded first rendering API data according to the second identifier.

[0365] For example, for each piece of first rendering API data, the rendering API data decoding module 122 can determine the general compression algorithm corresponding to each sub-data in the first rendering API data based on the second identifier corresponding to the first rendering API data (or the data type to which the first rendering API data belongs, or the sub-data in the first rendering API data). Then, for each sub-data in the first rendering API data, the rendering API data decoding module 122 can use the decompression algorithm corresponding to the general compression algorithm corresponding to each sub-data (such as the decompression algorithm corresponding to LZ77 compression, Deflate compression, or Zstd compression) to decompress the sub-data to obtain the reconstructed data of the sub-data; after decompressing each sub-data in the encoded first rendering API data, the reconstructed data of the first rendering API data can be obtained. After decompressing multiple encoded first rendering API data, the reconstructed data of multiple first rendering API data can be obtained.

[0366] S906, based on the third identifier, determines whether the sub-data in each first rendering API data has undergone lossy processing.

[0367] S907, for the reconstructed data of the sub-data that has undergone lossy processing in each first rendering API data, performs inverse lossy processing based on the lossy processing parameters.

[0368] For example, after obtaining the reconstructed data of the first rendering API data, the rendering API data decoding module 122 can also determine whether each sub-data in each first rendering API data has undergone lossy processing based on the third identifier corresponding to each first rendering API data (or corresponding to the data type to which the first rendering API data belongs, or corresponding to each sub-data in the first rendering API data). For each sub-data in the first rendering API data, if the rendering API data decoding module 122 determines that the sub-data has undergone lossy processing based on the third identifier, it can perform inverse lossy processing on the reconstructed data of the sub-data based on the lossy processing parameters corresponding to the lossy sub-data (or corresponding to the first rendering API data to which the lossy sub-data belongs, or corresponding to the data type to which the lossy sub-data belongs), to obtain the reconstructed data of the sub-data used for rendering. If the rendering API data decoding module 122 determines that the sub-data has not undergone lossy processing based on the third identifier, then there is no need to perform inverse lossy processing on the reconstructed data of the sub-data, and the reconstructed data of the sub-data obtained in S905 is used for rendering.

[0369] S908, based on the first identifier and the second preset type, determines the encoded second rendering API data.

[0370] For example, S908 can be described with reference to the above description of S704, and will not be repeated here.

[0371] Subsequently, the rendering API data encoding module 115 can determine that the encoding method corresponding to the second rendering API data is the third encoding method; wherein, the determination of the specific third encoding method corresponding to the second rendering API data and the decoding process can be referred to as follows S909~S910:

[0372] S909, based on the second identifier, determines the general decompression algorithm corresponding to the general compression algorithm for each second rendering API data.

[0373] S910, based on the general decompression algorithm corresponding to the general compression algorithm corresponding to each second rendering API data, decompresses each encoded first rendering API data to obtain the reconstructed data of each second rendering API data.

[0374] For example, after the rendering API data decoding module 122 determines the encoded second rendering API data among the multiple encoded rendering API data based on the first identifier, it can determine that the decoding method corresponding to the encoded second rendering API data is the second decoding method; then, the rendering API data decoding module 122 can further decompress each encoded second rendering API data according to the second identifier.

[0375] For example, for each piece of second rendering API data, the rendering API data decoding module 122 can determine the general compression algorithm corresponding to that piece of second rendering API data (the compression algorithm corresponding to all sub-data in that piece of second rendering API data is the same) based on the second identifier corresponding to each piece of second rendering API data (or the data type to which the first rendering API data belongs). Then, for that piece of second rendering API data, the decompression algorithm corresponding to the general compression algorithm corresponding to that piece of second rendering API data (such as the decompression algorithm corresponding to LZ77 compression algorithm, the decompression algorithm corresponding to Deflate compression algorithm, or the decompression algorithm corresponding to Zstd compression algorithm) can be used to decompress the piece of second rendering API data to obtain the reconstructed data of that piece of second rendering API data. After completing the decompression of multiple encoded pieces of second rendering API data, the reconstructed data of multiple pieces of second rendering API data can be obtained.

[0376] For example, the rendering API data decoding module 122 can reconstruct the reconstructed data of multiple first rendering API data and multiple second rendering API data based on other information in the classification information to obtain the reconstructed data of multiple rendering API data; the reconstructed data of the multiple rendering API data is used by the GPU to perform rendering operations to generate a rendered image.

[0377] It should be understood that the data structures of different types of first-rendering API data differ, therefore, the lossy characteristics of the reconstructed data differ depending on the type of first-rendering API data.

[0378] Figure 10A is a schematic diagram of another rendering API data compression, decompression, and rendering process according to an embodiment of this application. In Figure 10A, the second encoding method uses a dedicated compression algorithm for compression, and the first decoding method uses the decompression algorithm corresponding to the dedicated compression algorithm for decompression.

[0379] Referring to Figure 10A, the encoding process of rendering API data can be as follows: After classifying the rendering API data, the first device can directly perform general compression on a part of the data to obtain the encoded part of the data; and directly perform special compression on another part of the data to obtain the encoded part of the data; then, the two parts of encoded data, the classification information involved in the classification process, and the compression information involved in the compression process are packaged into a bitstream and sent.

[0380] The decoding process of the rendering API data can be as follows: The second device receives and parses the bitstream, obtaining encoded data, compression information, and classification information. Next, based on the compression and classification information, a portion of the encoded data undergoes general decompression, while another portion undergoes specific decompression. Then, based on the classification information, the data obtained from the specific decompression and the general decompression are reconstructed to obtain the reconstructed rendering API data. Subsequently, the 3D scene can be rendered based on the reconstructed rendering API data to obtain a rendered image.

[0381] Based on Figure 10A, the encoding and decoding process of the second rendering API data will be explained in more detail below.

[0382] Figure 10B is a schematic diagram of another encoding process 1000 according to an embodiment of this application. In the encoding process 1000 in Figure 10B, the second encoding method is to use a dedicated compression algorithm for compression; the encoding process 1000 can be executed by the rendering API data encoding module 115 of the first device.

[0383] S1001: Obtain multiple rendering application programming interface (API) data. These API data are used to generate rendered images.

[0384] S1002, based on the semantics of each rendering API data in the multiple rendering API data, classify the multiple rendering API data to obtain the type of each rendering API data in the multiple rendering API data.

[0385] S1003, based on the type of each rendering API data in the multiple rendering API data and the first preset type, determine the first rendering API data in the multiple rendering API data.

[0386] S1004, based on the type of each rendering API data in the multiple rendering API data and the second preset type, determine the second rendering API data in the multiple rendering API data.

[0387] It should be noted that the first preset type involved in the encoding process 1000 includes texture data type or mesh data type, and the second preset type includes function type, operation logic type or other types.

[0388] For example, S1001 to S1004 can be referred to the description of S601 to S604 above, and will not be repeated here.

[0389] Subsequently, the rendering API data encoding module 115 can determine that the encoding method corresponding to the first rendering API data is to use a dedicated compression algorithm for compression, and determine that the encoding method corresponding to the second rendering API data is to use a general compression algorithm for compression.

[0390] The encoding process for the first rendering API data (i.e., rendering API data of type texture data type or mesh data type) can be referred to as follows: S1005~S1007:

[0391] S1005, perform data structure decomposition on each first rendering API data to obtain the data structure information of each first rendering API data.

[0392] For example, S1007 can be described with reference to the above description of S805, and will not be repeated here.

[0393] For example, the data structure information obtained by the rendering API data encoding module 115 in S1005 can be format information. For example, the format information of the first rendering API data of texture data type can include, but is not limited to: texture image width and height, texture format (such as RGBA, ASTC), etc. For example, the format information of the first rendering API data of mesh data type can include, but is not limited to: vertex connectivity, normal parameters, etc.

[0394] S1006, Based on the data structure information of each first rendering API data, determine the dedicated compression algorithm corresponding to each first rendering API data.

[0395] For example, for each piece of first rendering API data, the rendering API data encoding module 115 can determine the dedicated compression algorithm corresponding to that piece of first rendering API data based on the format information of that piece of first rendering API data. It should be understood that the principle for selecting the dedicated compression algorithm corresponding to the first rendering API data from multiple dedicated compression algorithms can be to select the dedicated compression algorithm that can achieve the optimal balance between compression ratio and rendered image quality.

[0396] For example, for first rendering API data of texture data type, the corresponding dedicated compression algorithm can be determined to be, but is not limited to, traditional image and video compression algorithms such as JPEG compression algorithm and H26x (e.g., H264, H465) compression algorithm. For example, assuming there are two first rendering APIs (of texture data type), one first rendering API data has a texture format of RGBA in its format information, and the other first rendering API data has a texture format of ASTC in its format information; then for the first rendering API data with a texture format of RGBA in its format information, the corresponding dedicated compression algorithm is determined to be the JPEG compression algorithm, and for the first rendering API data with a texture format of ASTC in its format information, the corresponding dedicated compression algorithm is determined to be the H26x compression algorithm.

[0397] For example, for first rendering API data of mesh data type, the corresponding dedicated compression algorithm can be determined to be, but is not limited to, mesh data compression algorithms such as Draco compression algorithm and V-mesh compression algorithm. For example, assuming there are two first rendering APIs (of mesh data type), based on the format information of one first rendering API data, it is determined that this first rendering API data is the mesh data of object A, and based on the format information of the other first rendering API data, it is determined that this first rendering API data is the mesh data of object B; then, for the mesh data of object A, the corresponding Draco compression algorithm is determined, and for the mesh data of object B, the corresponding dedicated compression algorithm is determined to be the V-mesh compression algorithm.

[0398] In other words, the dedicated compression algorithms corresponding to two first rendering API data of the same type can be the same or different; and the dedicated compression algorithms corresponding to two first rendering API data of different types can be the same or different.

[0399] S1007 uses a dedicated compression algorithm corresponding to each first rendering API data to compress each first rendering API data to obtain encoded first rendering API data.

[0400] For example, the rendering API data encoding module 115 can use the JPEG compression algorithm to compress the first rendering API data with RGBA texture format in the format information to obtain the encoded first rendering API data. The rendering API data encoding module 115 can also use the H26x compression algorithm to compress the first rendering API data with ASTC texture format in the format information to obtain the encoded first rendering API data.

[0401] For example, the rendering API data encoding module 115 can use the Draco compression algorithm to compress the mesh data of object A to obtain encoded mesh data of object A (i.e., an encoded first rendering API data). The rendering API data encoding module 115 can use the V-mesh compression algorithm to compress the mesh data of object B to obtain encoded mesh data of object B (i.e., an encoded first rendering API data).

[0402] S1008 uses a general compression algorithm corresponding to each second rendering API data to encode each second rendering API data to obtain each encoded second rendering API data.

[0403] S1009 generates a bitstream based on multiple encoded first rendering API data, multiple encoded second rendering API data, classification information, and compression information.

[0404] For example, S1008 to S1009 can be described with reference to the above description of S810 to S811, and will not be repeated here.

[0405] The difference between the bitstream generated by encoding process 1000 and the bitstream generated by encoding process 800 is as follows:

[0406] The bitstream generated by encoding process 1000 includes a second identifier whose value ranges from 0 to 2, representing the LZ77 compression algorithm, Deflate compression algorithm, and Zstd compression algorithm, respectively; and a value range of 10 to 11, representing the JPEG compression algorithm and H.264x compression algorithm, respectively; furthermore, a value range of 20 to 21, representing the Draco compression algorithm and V-mesh compression algorithm, respectively. It should be understood that when encoding process 1000 involves more specialized compression algorithms, the second identifier has more possible values ​​(or more enumerated values ​​for compression methods).

[0407] In one possible implementation, the compressed information included in the bitstream generated by the encoding process 1000 may not include the third identifier and lossy processing parameters.

[0408] In one possible implementation, the compressed information included in the bitstream generated by the encoding process 1000 may include a third identifier, but not lossy processing parameters; in this case, the value of the third identifier can be 0.

[0409] In one possible implementation, the compressed information included in the bitstream generated by the encoding process 1000 may include a third identifier and lossy processing parameters; in this case, the value of the third identifier can be 0, and the lossy processing parameters are preset values.

[0410] Compared to using a general compression algorithm to compress the first rendering API data, using a dedicated compression algorithm to compress the first rendering API data can improve the compression ratio of the first rendering API data, thereby improving the overall data compression ratio.

[0411] Figure 11 is a schematic diagram of another decoding process 1100 according to an embodiment of this application. The decoding process 1100 in Figure 11 corresponds to the encoding process 1000 in Figure 10B. In the decoding process 1100, the first decoding method is to decompress using the decompression algorithm corresponding to the dedicated compression algorithm. The decoding process 1100 can be executed by the rendering API data decoding module 122.

[0412] S1101, receive bitstream.

[0413] S1102, parse the bitstream to obtain multiple encoded rendering API data, classification information, and compression information; wherein, the classification information includes a first identifier, and the compression information includes a second identifier.

[0414] S1103, based on the first identifier and the first preset type, determine the encoded first rendering API data.

[0415] For example, S1103 can be described with reference to the above description of S703, and will not be repeated here.

[0416] Subsequently, the rendering API data encoding module 115 can determine that the encoding method corresponding to the first rendering API data is the first encoding method; wherein, the determination of the specific first encoding method corresponding to the first rendering API data and the decoding process can be referred to as follows S1104~S1105:

[0417] S1104, Based on the second identifier, determine the dedicated decompression algorithm corresponding to the dedicated compression algorithm for each piece of first rendering API data.

[0418] S1105, based on the dedicated decompression algorithm corresponding to the dedicated compression algorithm for each first rendering API data, decompress each encoded first rendering API data to obtain the reconstructed data of each first rendering API data.

[0419] For example, if the rendering API data decoding module 122 determines that the decoding method corresponding to the encoded first rendering API data is the first decoding method based on the first identifier, the rendering API data decoding module 122 can further decompress each encoded first rendering API data according to the second identifier.

[0420] For example, for each piece of first rendering API data, the rendering API data decoding module 122 can determine the dedicated compression algorithm corresponding to that piece of first rendering API data based on the second identifier corresponding to each piece of first rendering API data. Then, for that piece of first rendering API data, the decompression algorithm corresponding to the dedicated compression algorithm corresponding to that piece of first rendering API data can be used (such as the decompression algorithm corresponding to the JPEG compression algorithm, the decompression algorithm corresponding to the H26x (such as H264, H465) compression algorithm, the decompression algorithm corresponding to the Draco compression algorithm, and the decompression algorithm corresponding to the V-mesh compression algorithm) to decompress the piece of first rendering API data to obtain the reconstructed data of that piece of first rendering API data. After completing the decompression of multiple encoded pieces of first rendering API data, the reconstructed data of multiple pieces of first rendering API data can be obtained.

[0421] S1106, based on the first identifier and the second preset type, determine the encoded second rendering API data.

[0422] For example, S1106 can be described with reference to the above description of S704, and will not be repeated here.

[0423] Subsequently, the rendering API data encoding module 115 can determine that the encoding method corresponding to the second rendering API data is the third encoding method; wherein, the determination of the specific third encoding method corresponding to the second rendering API data and the decoding process can be referred to as follows S1107~S1108:

[0424] S1107, Based on the second identifier, determine the general decompression algorithm corresponding to the general compression algorithm for each second rendering API data.

[0425] S1108, based on the general decompression algorithm corresponding to the general compression algorithm corresponding to each second rendering API data, decompress each encoded first rendering API data to obtain the reconstructed data of each second rendering API data.

[0426] For example, S1106 to S1108 can be described with reference to the above description of S908 to S910, and will not be repeated here.

[0427] It should be understood that when different types of first-rendering API data use different dedicated compression methods, the lossy characteristics of the reconstructed data will differ depending on the type of first-rendering API data.

[0428] Figure 12A is a schematic diagram of another rendering API data compression, decompression, and rendering process according to an embodiment of this application. In Figure 12A, the second encoding method is to compress the reconstructed data based on the encoded rendering API data in the cache queue, and the first decoding method is to decompress the reconstructed data based on the decoded rendering API data in the cache queue.

[0429] Referring to Figure 12A, the encoding process of the rendering API data can be as follows: After classifying the rendering API data, the first device can directly perform general / specialized compression on a portion of the data to obtain the encoded portion; for the other portion, it can determine whether reconstructed data for that portion exists in the cache queue; if not, general / specialized compression is directly performed on the non-existent data to obtain the encoded portion; if it exists, the identifier value of the reconstructed data for that portion is obtained and used as compression information. Then, the encoded data, the classification information involved in the classification process, and the compression information involved in the compression process are packaged into a bitstream and sent.

[0430] The decoding process of the rendering API data can be as follows: The second device receives and parses the bitstream, obtaining encoded data, compression information, and classification information. Next, based on the compression and classification information, it checks the cache queue for reconstructed data for a portion of the data. If it exists, the reconstructed data for that portion is read from the cache queue. If it does not exist, the encoded data for that portion is decompressed using a general / specific method. Then, based on the compression and classification information, another portion of the encoded data is decompressed using a general / specific method. Afterward, the data obtained from the general / specific decompression and the data read from the cache queue are reassembled based on the classification information to obtain the reconstructed rendering API data. Subsequently, the 3D scene can be rendered based on the reconstructed rendering API data to obtain a rendered image.

[0431] Based on Figure 12A, the encoding and decoding process of the first rendering API data will be explained in more detail below.

[0432] Figure 12B is a schematic diagram of another encoding process 1200 according to an embodiment of this application. In the encoding process 1200 in Figure 12B, the second encoding method is to compress the reconstructed data based on the encoded rendering API data in the cache queue; the encoding process 1200 can be executed by the rendering API data encoding module 115 of the first device.

[0433] S1201: Obtain multiple rendering application programming interface (API) data. These API data are used to generate the rendered image.

[0434] S1202, based on the semantics of each rendering API data in the multiple rendering API data, classify the multiple rendering API data to obtain the type of each rendering API data in the multiple rendering API data.

[0435] S1203, based on the type of each rendering API data in the multiple rendering API data and the first preset type, determine the first rendering API data in the multiple rendering API data.

[0436] S1204, based on the type of each rendering API data in the multiple rendering API data and the second preset type, determine the second rendering API data in the multiple rendering API data.

[0437] It should be noted that the first preset type involved in the encoding process 1200 includes texture data type, operation logic type or grid data type, and the second preset type includes function type or other types.

[0438] For example, S1201 to S1204 can be referred to the description of S601 to S604 above, and will not be repeated here.

[0439] Subsequently, the rendering API data encoding module 115 can determine that the encoding method corresponding to the first rendering API data is to compress the reconstructed data based on the already encoded rendering API data in the cache queue, and determine that the encoding method corresponding to the second rendering API data is to compress it using a general compression algorithm.

[0440] The process of encoding the first rendering API data (i.e., rendering API data of type texture data type, operation logic type, or mesh data type) can be referred to as follows: S1205~S1209:

[0441] S1205, perform data structure decomposition on each first rendering API data to obtain the data structure information of each first rendering API data.

[0442] For example, S1205 can be described with reference to the above description of S805, and will not be repeated here.

[0443] S1206, Based on the data structure information of each first rendering API data, determine the sub-data in each first rendering API data that meets the second preset condition.

[0444] For example, different second preset conditions can be pre-set for different types of first rendering API data. Specifically, for one type of first rendering API data, it can be pre-checked whether there is any identical sub-data among multiple first rendering API data of the same type corresponding to any two frames; when there is identical sub-data among multiple first rendering API data of the same type in two adjacent frames, the second preset condition can be set to that sub-data (or that field or that parameter).

[0445] For example, the second preset condition for the first rendering API data setting for the operation logic type can be set to: shader information.

[0446] For example, the second preset condition for the first rendering API data setting for texture data types could be: texture information (which could refer to a texture image).

[0447] For example, the second preset condition for the first rendering API data setting for texture data types could be: mesh information (which could refer to vertex coordinates).

[0448] S1207, Determine whether there is reconstructed data of sub-data that meets the second preset condition in the cache queue.

[0449] For example, the cache queue can be located inside or outside the rendering API data encoding module 115, and this application does not impose any restrictions on this. After encoding a sub-data of each piece of first rendering API data, if the reconstructed data and the identifier value of the reconstructed data of the sub-data do not exist in the cache queue, the rendering API data encoding module 115 can store the reconstructed data and the identifier value (ID value) of the reconstructed data of the sub-data into the cache queue; in this way, other sub-data to be encoded subsequently can be compressed by referring to the sub-data already encoded in the cache queue.

[0450] For example, algorithms such as hashing and cyclic redundancy check (CRC) can be used to calculate the feature value of the reconstructed data for each sub-data; then, the feature value of the reconstructed data for each sub-data is used as the identifier (ID value) of each sub-data.

[0451] For example, the process of compressing the reconstructed data of the encoded sub-data in the cache queue can be referred to S1208 to S1211. First, for each sub-data that meets the second preset condition, the rendering API data encoding module 115 can determine whether the reconstructed data of the sub-data exists in the cache queue (which can be understood as determining whether there is encoded sub-data in the cache queue that matches the sub-data). Specifically, the rendering API data encoding module 115 can calculate the ID value of the reconstructed data of the sub-data; then, it compares the ID value of the reconstructed data of the sub-data with the ID value stored in the cache queue to determine whether there is reconstructed data of the sub-data in the cache queue. When the rendering API data encoding module 115 finds an ID value in the cache queue that is the same as the ID value of the reconstructed data of the sub-data, it can determine that there is reconstructed data of the sub-data in the cache queue, and S1208 can be executed. When the rendering API data encoding module 115 does not find an ID value in the cache queue that is the same as the ID value of the reconstructed data of the sub-data, it can determine that there is no reconstructed data of the sub-data in the cache queue, and S1209 can be executed.

[0452] S1208, when there is reconstructed data of sub-data that meets the second preset condition in the cache queue, generate an identifier value for the reconstructed data of sub-data that meets the second preset condition.

[0453] For example, when it is determined that there is encoded sub-data in the cache queue that matches sub-data that meets the second preset condition, the rendering API data encoding module 115 can write the identifier value of the reconstructed data of the sub-data that meets the second preset condition into the bitstream. In this way, during the decoding process of the second device, the sub-data that meets the second preset condition can be found from the cache queue according to the decoded identifier value.

[0454] It should be understood that for each sub-data that meets the second preset condition, the same compression algorithm as the encoded sub-data that matches the sub-data can be used to compress the sub-data; and the corresponding decompression algorithm can be used to decompress the encoded sub-data to obtain the reconstructed data of the sub-data.

[0455] S1209, when there is no reconstructed data of sub-data that meets the second preset condition in the cache queue, the sub-data that meets the second preset condition is compressed to obtain encoded sub-data.

[0456] For example, when it is determined that there is no encoded sub-data in the cache queue that matches the sub-data that meets the second preset condition, the rendering API data encoding module 115 can use a general compression algorithm or a special compression algorithm to compress the sub-data that meets the second preset condition to obtain encoded sub-data (the sub-data that meets the second preset condition).

[0457] It should be understood that within the same first rendering API data set, different sub-data points that meet the second preset condition may use the same or different general compression algorithms; similarly, within the same first rendering API data set, different sub-data points that meet the second preset condition may use the same or different dedicated compression algorithms. Furthermore, within different types of first rendering API data sets, sub-data points that meet the second preset condition may use the same or different general compression algorithms; and similarly, within different types of first rendering API data sets, sub-data points that meet the second preset condition may use the same or different dedicated compression algorithms.

[0458] S1210, when there is no reconstructed data of sub-data that meets the second preset condition in the cache queue, the reconstructed data of the sub-data that meets the second preset condition and the identifier value of the reconstructed data of the sub-data that meets the second preset condition are stored in the cache queue.

[0459] S1211, compress the sub-data in each first rendering API data that does not meet the second preset condition to obtain encoded sub-data that does not meet the second preset condition.

[0460] For example, for each sub-data that does not meet the second preset condition, the rendering API data encoding module 115 may use a general compression algorithm or a special compression algorithm to compress the sub-data that does not meet the second preset condition to obtain the encoded sub-data (the sub-data that does not meet the second preset condition).

[0461] It should be understood that within the same first rendering API data set, different sub-data points that do not meet the second preset condition may use the same or different general compression algorithms; similarly, within the same first rendering API data set, different sub-data points that do not meet the second preset condition may use the same or different dedicated compression algorithms. Similarly, within different types of first rendering API data sets, sub-data points that do not meet the second preset condition may use the same or different general compression algorithms; and similarly, within different types of first rendering API data sets, sub-data points that do not meet the second preset condition may use the same or different dedicated compression algorithms.

[0462] S1212, using a general compression algorithm corresponding to each second rendering API data, encodes each second rendering API data to obtain encoded second rendering API data.

[0463] S1213 generates a bitstream based on multiple encoded first rendering API data, multiple encoded second rendering API data, classification information, and compression information.

[0464] For example, S1212 to S1213 can be described with reference to the above description of S810 to S811, and will not be repeated here.

[0465] For example, the difference between the compressed information in S1215 and the compressed information in S812 is as follows:

[0466] The compression information generated during the encoding process 1200 may further include a fourth identifier (match_flag) and a fifth identifier (match_id). For example, the fourth identifier indicates whether there is already encoded rendering API data in the cache queue that matches the rendering API data to be encoded. The value of the fourth identifier can be 0 or 1; 0 indicates that there is no reconstructed data of the rendering API data to be encoded in the cache queue, and 1 indicates that there is reconstructed data of the rendering API data to be encoded in the cache queue. For example, the fifth identifier indicates the reconstructed data of the rendering API data to be encoded in the cache queue; the value of the fifth identifier can be the identifier value of the reconstructed data of the rendering API data to be encoded in the cache queue.

[0467] It should be noted that each sub-data (data contained in a field) in a rendering API data corresponds to a fourth identifier, or a rendering API data corresponds to a fourth identifier (all sub-data contained in that rendering API data corresponds to a fourth identifier), or multiple rendering API data included in each type of rendering API data correspond to a fourth identifier.

[0468] It should be noted that each sub-data (data contained in a field) in a rendering API data set corresponds to a fifth identifier.

[0469] The bitstream generated by encoding process 1200 includes a second identifier whose value ranges from 0 to 2, representing the LZ77 compression algorithm, Deflate compression algorithm, and Zstd compression algorithm, respectively; and a value range of 10 to 11, representing the JPEG compression algorithm and H.264x compression algorithm, respectively; furthermore, a value range of 20 to 21, representing the Draco compression algorithm and V-mesh compression algorithm, respectively. It should be understood that when encoding process 1000 involves more specialized compression algorithms, the second identifier has more possible values ​​(or more enumerated values ​​for compression methods).

[0470] In one possible implementation, the compressed information included in the bitstream generated by the encoding process 1000 may include a third identifier and lossy processing parameters; in this case, the value of the third identifier can be 0, and the lossy processing parameters are preset values.

[0471] For example, the syntax of the bitstream generated by S1215 can be illustrated in Table 4:

[0472] Table 4

[0473] Compared to Table 3, the new syntax elements added in Table 4 include: match_flag and match_id.

[0474] In one possible implementation, the compressed information included in the bitstream generated by the encoding process 1000 may not include the third identifier and lossy processing parameters.

[0475] In one possible implementation, the compressed information included in the bitstream generated by the encoding process 1000 may include a third identifier, but not lossy processing parameters; in this case, the value of the third identifier can be 0.

[0476] Compared to encoding all the data of a certain type of rendering API data, the embodiments of this application only encode the data that does not exist in the cache queue, that is, only encode a part of the data of this type of rendering API data. In this way, fewer bits are required, which can further improve the compression rate of this type of data, thereby improving the overall data compression rate.

[0477] Figure 13 is a schematic diagram of another decoding process 1300 according to an embodiment of this application. The decoding process 1300 in Figure 13 corresponds to the encoding process 1200 in Figure 12B. The first decoding method in the decoding process 1300 is to decompress the reconstructed data based on the decoded rendering API data in the cache queue. The decoding process 1300 can be executed by the rendering API data decoding module 122.

[0478] S1301, receive bitstream.

[0479] S1302, parse the bitstream to obtain multiple encoded rendering API data, classification information, and compression information; wherein, the classification information includes a first identifier, and the compression information includes a second identifier, a third identifier, a fourth identifier, and a fifth identifier.

[0480] S1303, based on the first identifier and the first preset type, determine the encoded first rendering API data.

[0481] For example, S1303 can be described with reference to the above description of S703, and will not be repeated here.

[0482] Subsequently, the rendering API data encoding module 115 can determine that the encoding method corresponding to the first rendering API data is the first encoding method; wherein, the determination of the specific first encoding method corresponding to the first rendering API data and the decoding process can be referred to as follows S1304~S1308:

[0483] S1304, based on the fourth identifier, determine whether the cache queue contains reconstructed data of sub-data in the first rendering API data.

[0484] For example, determining whether the cache queue contains reconstructed data of sub-data in the first rendering API data can be understood as determining whether the cache queue contains decoded sub-data that matches the sub-data in the first rendering API data.

[0485] For example, for each sub-data in the first rendering API data, the rendering API data decoding module 122 can determine whether the value of the fourth identifier corresponding to the sub-data is 0 or 1. When the value of the fourth identifier corresponding to the sub-data is 1, it can be determined that the cache queue contains reconstructed data of the sub-data in the first rendering API data, and S1305 can be executed. When the value of the fourth identifier corresponding to the sub-data is 0, it can be determined that the cache queue does not contain reconstructed data of the sub-data in the first rendering API data, and S1306 can be executed.

[0486] S1305, when it is determined that the cache queue contains reconstructed data of sub-data in the first rendering API data, the reconstructed data of sub-data in the first rendering API data is read from the cache queue based on the fifth identifier.

[0487] For example, for each sub-data in the first rendering API data, when it is determined that the reconstructed data of the sub-data exists in the cache queue, the rendering API data decoding module 122 can find the identifier value that is the same as the value of the fifth identifier corresponding to the sub-data in the cache queue, and use the reconstructed data corresponding to the identifier value in the cache queue as the reconstructed data of the sub-data.

[0488] S1306, when it is determined that there is no reconstructed data of the first rendering API data in the cache queue, the compression algorithm corresponding to the sub-data in the first rendering API data is determined based on the second identifier.

[0489] For example, for each sub-data in the first rendering API data, when it is determined that the reconstructed data for that sub-data does not exist in the cache queue, the rendering API data decoding module 122 can determine the compression algorithm corresponding to that sub-data based on the second identifier corresponding to that sub-data. It should be understood that the compression algorithm corresponding to that sub-data can be a general compression algorithm or a special compression algorithm.

[0490] S1307, based on the decompression algorithm corresponding to the compression algorithm of the sub-data in the first rendering API data, decompress the encoded sub-data in the first rendering API data to obtain the reconstructed data of the sub-data in the first rendering API data.

[0491] Subsequently, for each sub-data in the first rendering API data, the rendering API data decoding module 122 can decompress the encoded sub-data based on the decompression algorithm corresponding to the compression algorithm of the sub-data to obtain the reconstructed data of the sub-data.

[0492] S1308, add the reconstructed data of the sub-data in the first rendering API data and the identifier value of the reconstructed data of the sub-data in the first rendering API data to the cache queue.

[0493] Furthermore, for each sub-data in the first rendering API data, when it is determined that the reconstructed data of the sub-data does not exist in the cache queue, the rendering API data decoding module 122 can add the reconstructed data of the sub-data and the identifier value of the reconstructed data of the sub-data to the cache queue.

[0494] S1309, based on the first identifier and the second preset type, determine the encoded second rendering API data.

[0495] For example, S1309 can be described with reference to the above description of S704, and will not be repeated here.

[0496] Subsequently, the rendering API data encoding module 115 can determine that the encoding method corresponding to the second rendering API data is the third encoding method; wherein, the determination of the specific third encoding method corresponding to the second rendering API data and the decoding process can be referred to as follows S1310~S1311:

[0497] S1310, based on the second identifier, determine the general compression algorithm corresponding to each second rendering API data.

[0498] S1311, based on the decompression algorithm corresponding to the general compression algorithm corresponding to each second rendering API data, decompress each encoded first rendering API data to obtain the reconstructed data of each second rendering API data.

[0499] For example, S1310 to S1311 can be described with reference to the above description of S909 to S910, and will not be repeated here.

[0500] It should be understood that the sub-data that meet the second preset condition differs in different types of first rendering API data. Therefore, the lossy characteristics of the reconstructed data of different types of first rendering API data are different.

[0501] It should be understood that any two or more of the three second encoding methods in the above encoding processes 800, 1000 and 1200 can be combined; and any two or more of the three first decoding methods in the above decoding processes 900, 1100 and 1300 can be combined, which will not be elaborated here.

[0502] Referring again to Figure 3A, for the scenario where the rendering computing power of the first device's GPU is partially offloaded:

[0503] In one possible approach, the GPU driver 111 of the first device 11 can intercept the second rendering API data and output it to the rendering API data encoding module 115. The rendering API data encoding module 115 can then encode the second rendering API data according to the aforementioned encoding processes 400, 600, 800, 1000, or 1200 to obtain encoded second rendering API data. Subsequently, based on the encoded second rendering API data, classification information, and compression information, a bitstream is generated.

[0504] In one possible approach, the GPU driver 111 of the first device 11 determines which part of the rendering API data to output to the rendering API data encoding module 115 based on the rendering complexity and / or data volume. Then, the rendering API data encoding module 115 can encode this part of the rendering API data according to the above-described encoding process 400, encoding process 600, encoding process 800, encoding process 1000 or encoding process 1200 to obtain the encoded part of the rendering API data. After that, a bitstream is generated based on the encoded part of the rendering API data, classification information and compression information.

[0505] In one possible approach, the GPU driver 111 of the first device 11 outputs operations, data, and algorithms corresponding to the same type of operation to the rendering API data encoding module 115, based on the operation type. For example, multiple operations included in `glcompressedtexmage2D` are of the same type; similarly, multiple operations included in `gldrawelementsbasevertex` are of the same type. Then, the rendering API data encoding module 115 can encode the operations, data, and algorithms corresponding to the same type of operation according to the aforementioned encoding processes 400, 600, 800, 1000, or 1200 to obtain encoded data; subsequently, a bitstream is generated based on the encoded data, classification information, and compression information.

[0506] It should be understood that when all first rendering API data is encoded using the same first encoding method and all second rendering API data is encoded using the same third encoding method, the bitstream may not include compression information. In this case, the rendering API data decoding module can determine the first decoding method of all encoded first rendering API data and the second decoding method of all encoded second rendering API data based solely on the first identifier in the classification information.

[0507] In one example, FIG14 shows a schematic block diagram of an apparatus 1400 according to an embodiment of the present application. The apparatus 1400 may include a processor 1401 and a transceiver 1402, and optionally, a memory 1403.

[0508] The various components of device 1400 are coupled together via bus 1404, which includes a data bus, a power bus, a control bus, and a status signal bus. However, for clarity, all buses are referred to as bus 1404 in the figure.

[0509] Optionally, the memory 1403 can be used to store instructions from the foregoing method embodiments. The processor 1401 can be used to execute the instructions in the memory 1403, control the transceiver 1402 to receive signals, and control the transceiver 1402 to transmit signals.

[0510] The device 1400 may be an electronic device or a chip of an electronic device in the above method embodiments. Exemplarily, the electronic device may be a first device or a second device.

[0511] All relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0512] This application also provides a chip, including one or more interface circuits and one or more processors; the one or more processors receive or transmit data through the one or more interface circuits, and when the one or more processors execute computer instructions, the steps of the above-described related method steps that implement the method in the above embodiments are executed. The interface circuit is a transceiver 1302.

[0513] This application also provides a computer-readable storage medium storing computer instructions. When these computer instructions are executed on an electronic device, the electronic device performs the aforementioned method steps to implement the methods described in the above embodiments. Exemplarily, the computer-readable storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0514] This application also provides a computer program product comprising computer instructions that, when executed by a computer or processor, cause the computer to perform the aforementioned steps to implement the methods described in the above embodiments. Exemplarily, the computer program product may be stored in random access memory (RAM), flash memory, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, read-only optical discs (CD-ROMs), or any other form of storage medium known in the art.

[0515] In this embodiment, the electronic device, computer-readable storage medium, computer program product or chip are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0516] Through the above description of the embodiments, those skilled in the art will understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0517] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0518] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0519] Any content in the various embodiments of this application, as well as any content in the same embodiment, can be freely combined. Any combination of the above content is within the scope of this application.

[0520] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An encoding method, characterized in that, The method includes: Obtain multiple rendering application programming interface (API) data, which are used to generate a rendered image; Obtain the type of each rendering API data in the multiple rendering API data; Based on the type of each of the multiple rendering API data, the encoding method corresponding to each of the multiple rendering API data is obtained, and the encoding method corresponding to at least one type of rendering API data is a lossy encoding method; Based on the encoding method corresponding to each of the multiple rendering API data, each of the multiple rendering API data is encoded to obtain a bitstream.

2. The method according to claim 1, characterized in that, The process of obtaining the type of each rendering API data in the multiple rendering API data includes: Based on the semantics of each of the multiple rendering API data, the multiple rendering API data are classified to obtain the type of each of the multiple rendering API data.

3. The method according to claim 1 or 2, characterized in that, The step of obtaining the encoding method corresponding to each rendering API data in the multiple rendering API data based on the type of each rendering API data in the multiple rendering API data includes: Based on the type of each rendering API data in the multiple rendering API data and the first preset type, determine the first rendering API data in the multiple rendering API data; The encoding method corresponding to the first rendering API data is determined to be the first encoding method. The first encoding method includes data structure decomposition and the second encoding method. The first rendering API data is one or more pieces. The step of encoding each rendering API data point in the plurality of rendering API data points according to the encoding method corresponding to each rendering API data point in the plurality of rendering API data points to obtain a bitstream includes: Based on the first encoding method corresponding to each first rendering API data, each first rendering API data is encoded to obtain each encoded first rendering API data. The bitstream includes one or more encoded first rendering API data.

4. The method according to claim 3, characterized in that, The second encoding method includes at least one of the following: Lossy compression and compression using a general compression algorithm; Compression is performed using a dedicated compression algorithm; Compress the reconstructed data based on the coded rendering API data in the cache queue.

5. The method according to any one of claims 1 to 4, characterized in that, The step of obtaining the encoding method corresponding to each rendering API data in the multiple rendering API data based on the type of each rendering API data in the multiple rendering API data also includes: Based on the type of each rendering API data in the multiple rendering API data and the second preset type, determine the second rendering API data in the multiple rendering API data; The encoding method corresponding to the second rendering API data is determined to be the third encoding method, and the second rendering API data consists of one or more pieces; The step of encoding each rendering API data point in the plurality of rendering API data points according to the encoding method corresponding to each rendering API data point in the plurality of rendering API data points to obtain a bitstream further includes: Based on the third encoding method corresponding to each second rendering API data, each second rendering API data is encoded to obtain each encoded second rendering API data. The bitstream also includes one or more encoded second rendering API data.

6. The method according to claim 5, characterized in that, The third encoding method includes compression using a general compression algorithm.

7. The method according to any one of claims 1 to 6, characterized in that, The rendering API data type includes at least one of the following: function type, operation logic type, texture data type, mesh data type, or other types.

8. The method according to claim 5 or 6, characterized in that, The first preset type includes arithmetic logic type, texture data type, or mesh data type; the second preset type includes function type or other types. or, The first preset type includes texture data type or mesh data type, and the second preset type includes function type, operation logic type or other type.

9. The method according to any one of claims 3 to 8, characterized in that, The process of encoding each first rendering API data point based on a first encoding method corresponding to each first rendering API data point to obtain encoded first rendering API data points includes: The data structure of each first rendering API data is decomposed to obtain the data structure information of each first rendering API data. Based on the data structure information of each first rendering API data and the second encoding method corresponding to each first rendering API data, each first rendering API data is encoded to obtain each encoded first rendering API data.

10. The method according to claim 9, characterized in that, The second encoding method involves lossy processing and compression using a general compression algorithm; the encoding of each first rendering API data based on the data structure information of each first rendering API data and the corresponding second encoding method to obtain encoded first rendering API data includes: Based on the data structure information of each first rendering API data, determine the sub-data in each first rendering API data that meets the first preset condition; For each piece of first rendering API data, the sub-data that meets the first preset condition is subjected to lossy processing to obtain lossy sub-data. The lossy processing includes quantization processing or clustering processing. The lossy sub-data is compressed using a general compression algorithm to obtain encoded sub-data.

11. The method according to claim 9, characterized in that, The second encoding method uses a dedicated compression algorithm for compression; the encoding of each first rendering API data based on the data structure information of each first rendering API data and the corresponding second encoding method to obtain each encoded first rendering API data includes: Based on the data structure information of each first rendering API data, determine the dedicated compression algorithm corresponding to each first rendering API data; Each first rendering API data is compressed using a dedicated compression algorithm corresponding to each first rendering API data to obtain an encoded first rendering API data.

12. The method according to claim 9, characterized in that, The second encoding method is to compress reconstructed data based on the encoded rendering API data in the cache queue; the encoding of each first rendering API data based on the data structure information of each first rendering API data and the second encoding method corresponding to each first rendering API data to obtain each encoded first rendering API data includes: Based on the data structure information of each first rendering API data, determine the sub-data in each first rendering API data that meets the second preset condition; Determine whether the reconstructed data of the sub-data that satisfies the second preset condition exists in the cache queue; When the reconstructed data of the sub-data that meets the second preset condition exists in the cache queue, the identifier value of the reconstructed data of the sub-data that meets the second preset condition is written into the bitstream; When there is no reconstructed data of the sub-data that meets the second preset condition in the cache queue, the sub-data that meets the second preset condition is compressed to obtain encoded sub-data.

13. The method according to claim 12, characterized in that, The method further includes: When the reconstructed data of the sub-data that meets the second preset condition does not exist in the cache queue, the reconstructed data of the sub-data that meets the second preset condition and the identifier value of the reconstructed data of the sub-data that meets the second preset condition are added to the cache queue.

14. A bitstream, characterized in that, The bitstream includes: encoded rendering application programming interface (API) data and a first identifier, the first identifier indicating the type of the rendering API data, which is used to generate a rendered image.

15. The bitstream according to claim 14, characterized in that, The bitstream further includes a second identifier, which indicates the compression algorithm corresponding to the rendering API data.

16. The bitstream according to claim 14 or 15, characterized in that, The bitstream also includes a third identifier, which indicates whether the rendering API data has undergone lossy processing.

17. The bitstream according to any one of claims 14 to 16, characterized in that, The bitstream also includes a fourth identifier, which indicates whether reconstructed data of the rendering API data exists in the cache queue.

18. The method according to any one of claims 14 to 17, characterized in that, The bitstream also includes a fifth identifier, which indicates the reconstructed data of the rendering API data in the cache queue.

19. The bitstream according to any one of claims 14 to 18, characterized in that, The bitstream also includes at least one of the following: frame identifier, total number of rendering API data entries, total number of types of rendering API data, and data length or lossy processing parameters for each type of rendering API data.

20. A decoding method, characterized in that, The method includes: Receive bitstream; The bitstream is parsed to obtain multiple encoded rendering application programming interface (API) data and a first identifier, wherein the first identifier indicates the type of each rendering API data in the multiple rendering API data; Based on the first identifier, determine the decoding method corresponding to each encoded rendering API data; Based on the decoding method corresponding to each encoded rendering API data, each encoded rendering API data is decoded to obtain the reconstructed data of each rendering API data. The reconstructed data of multiple rendering API data are used to generate a rendered image. There is a difference between the reconstructed data of at least one type of rendering API data and the reconstructed data of the at least one type of rendering API data.

21. The method according to claim 20, characterized in that, The step of determining the decoding method corresponding to each encoded rendering API data based on the first identifier includes: Based on the first identifier, the encoded first rendering API data is determined, and the decoding method corresponding to the encoded first rendering API data is determined to be the first decoding method; The decoding of each encoded rendering API data point, based on the decoding method corresponding to each encoded rendering API data point, to obtain the reconstructed data of each rendering API data point, includes: Based on the first decoding method corresponding to each encoded first rendering API data, each encoded first rendering API data is decoded to obtain the reconstructed data of each first rendering API data.

22. The method according to claim 21, characterized in that, The first decoding method includes at least one of the following: The lossy compression is reversed and decompressed using the decompression algorithm corresponding to the general compression algorithm; Decompression is performed using the decompression algorithm corresponding to the dedicated compression algorithm; Decompress the reconstructed data based on the decoded rendering API data in the cache queue.

23. The method according to claim 21 or 22, characterized in that, The step of determining the decoding method corresponding to each encoded rendering API data based on the first identifier further includes: Based on the first identifier, the encoded second rendering API data is determined, and the decoding method corresponding to the encoded second rendering API data is determined to be the second decoding method; The method of decoding each encoded rendering API data according to the decoding method corresponding to each encoded rendering API data to obtain the reconstructed data of each rendering API data also includes: Based on the second decoding method corresponding to each encoded second rendering API data, each encoded second rendering API data is decoded to obtain the reconstructed data of each second rendering API data.

24. The method according to claim 23, characterized in that, The second decoding method includes decompression using a decompression algorithm corresponding to a general compression algorithm.

25. The method according to claim 23 or 24, characterized in that, The first rendering API data is of type arithmetic logic type, texture data type, or mesh data type, and the second rendering API data is of type a function type or other types; or, The first rendering API data is of texture data type or mesh data type, and the second rendering API data is of operation logic type, function type or other type.

26. The method according to claim 25, characterized in that, The reconstructed data from different types of first-rendering API data have different lossy characteristics.

27. The method according to any one of claims 21 to 26, characterized in that, The first decoding method corresponding to each encoded first rendering API data is used to decode each encoded first rendering API data to obtain reconstructed data for each first rendering API data, including: Based on the second identifier parsed from the bitstream, the compression algorithm corresponding to each first rendering API data is determined; Based on the decompression algorithm corresponding to the compression algorithm of each first rendering API data, each encoded first rendering API data is decompressed to obtain the reconstructed data of each first rendering API data.

28. The method according to claim 27, characterized in that, The method of decoding each encoded first rendering API data according to the first decoding method corresponding to each encoded first rendering API data to obtain the reconstructed data of each first rendering API data further includes: Based on the third identifier parsed from the bitstream, determine whether the sub-data in each first rendering API data has undergone lossy processing; For the reconstructed data of each lossy sub-data in the first rendering API data, the lossy processing is reversed based on the lossy processing parameters parsed from the bitstream. The reverse processing includes dequantization or inverse clustering.

29. The method according to any one of claims 21 to 26, characterized in that, The first decoding method corresponding to each encoded first rendering API data is used to decode each encoded first rendering API data to obtain reconstructed data for each first rendering API data, including: Based on the fourth identifier parsed from the bitstream, determine whether the cache queue contains reconstructed data of the sub-data in the first rendering API data; When it is determined that the cache queue contains reconstructed data of the sub-data in the first rendering API data, the reconstructed data of the sub-data in the first rendering API data is read from the cache queue based on the fifth identifier parsed from the bitstream.

30. The method according to claim 29, characterized in that, The method of decoding each encoded first rendering API data according to the first decoding method corresponding to each encoded first rendering API data to obtain the reconstructed data of each first rendering API data further includes: When it is determined that the cache queue does not contain reconstructed data of the sub-data in the first rendering API data, the compression algorithm corresponding to the sub-data in the first rendering API data is determined based on the second identifier parsed from the bitstream. Based on the decompression algorithm corresponding to the compression algorithm of the sub-data in the first rendering API data, the encoded sub-data in the first rendering API data is decompressed to obtain the reconstructed data of the sub-data in the first rendering API data.

31. The method according to claim 30, characterized in that, The method further includes: When it is determined that the cache queue does not contain reconstructed data of the sub-data in the first rendering API data, the reconstructed data of the sub-data in the first rendering API data and the identifier value of the reconstructed data of the sub-data in the first rendering API data are added to the cache queue.

32. A rendering system, characterized in that, The rendering system includes a first device and a second device, wherein: The first device is configured to: acquire multiple rendering application programming interface (API) data; acquire the type of each rendering API data; based on the type of each rendering API data, acquire the encoding method corresponding to each rendering API data, wherein at least one type of rendering API data corresponds to a lossy encoding method; encode each rendering API data based on the encoding method corresponding to each rendering API data to obtain a bitstream; write a first identifier into the bitstream, the first identifier indicating the type of each rendering API data; and send the bitstream. The second device is configured to receive a bitstream; parse the bitstream to obtain multiple encoded rendering API data and the first identifier; determine the decoding method corresponding to each encoded rendering API data based on the first identifier; decode each encoded rendering API data based on the decoding method corresponding to each encoded rendering API data to obtain reconstructed data for each rendering API data, wherein the reconstructed data of at least one type of rendering API data differs from the reconstructed data of the at least one type of rendering API data; render a 3D scene based on the reconstructed data of the multiple rendering API data to obtain a rendered image; and send the rendered image. The first device is also used to display the rendered image.

33. The rendering system according to claim 32, characterized in that, The first device is specifically used to classify the multiple rendering API data based on the semantics of each rendering API data in the multiple rendering API data, so as to obtain the type of each rendering API data in the multiple rendering API data; Based on the type of each rendering API data in the multiple rendering API data and the first preset type, determine the first rendering API data in the multiple rendering API data; Based on the type and second preset type of each of the multiple rendering API data, the second rendering API data in the multiple rendering API data is determined; based on the first encoding method corresponding to each first rendering API data, each first rendering API data is encoded to obtain each encoded first rendering API data. Based on the third encoding method corresponding to each second rendering API data, each second rendering API data is encoded to obtain each encoded second rendering API data; the bitstream includes one or more encoded first rendering API data and one or more encoded second rendering API data.

34. The rendering system according to claim 33, characterized in that, The first preset type includes arithmetic logic type, texture data type, or mesh data type; the second preset type includes function type or other types. or, The first preset type includes texture data type or mesh data type, and the second preset type includes function type, operation logic type or other type.

35. A bitstream, characterized in that, The bitstream is generated by the method according to any one of claims 1 to 13.

36. An encoder, characterized in that, The encoder is used to perform the method as described in any one of claims 1 to 13.

37. A decoder, characterized in that, The decoder is used to perform the method as described in any one of claims 20 to 31.

38. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed on a computer or processor, causes the computer or processor to perform the method as described in any one of claims 1 to 13; or causes the computer or processor to perform the method as described in any one of claims 20 to 31.

39. A computer program product, characterized in that, The computer program product includes computer instructions that, when executed by a computer or processor, cause the steps of the method as described in any one of claims 1 to 13 to be performed; or cause the steps of the method as described in any one of claims 20 to 31 to be performed.

Images