Image processing method and apparatus

Abstract

The present application provides an image processing method and apparatus. The method comprises: acquiring a first region related to a scene change in an image to be processed, and performing graphical calculation on the first region, to obtain a first intermediate result of the image to be processed; and encoding the first intermediate result, to obtain an encoded bitstream. The present method performs graphical calculation on a region related to a scene change in an image to be processed, so as to obtain a first intermediate result of the first region in the image to be processed, thereby greatly reducing the amount of calculation and the calculation cost of a server performing graphical calculation on the image to be processed, to obtain the intermediate result of the image to be processed, reducing the amount of encoding, shortening the length of a bitstream, and further reducing the proportion of the bitstream in a transmission bandwidth.

Description

Image processing methods and apparatus

[0001] This application claims priority to Chinese Patent Application No. 202411598631.8, filed on November 8, 2024, entitled "Image Processing Method and Apparatus", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of image processing technology, and in particular to an image processing method and apparatus. Background Technology

[0003] Many edge-cloud collaborative scenarios (such as cloud gaming, virtual reality (VR) / augmented reality (AR), 3D cloud conferencing, and collaborative editing and production of 3D scenes) require graphics processing to generate visuals, making the images more realistic and improving the user experience. Rendering, physical simulation, and animation require significant computing power, but edge devices are often limited by physical constraints such as device size and power consumption, resulting in computing power far weaker than cloud servers. Therefore, graphics processing algorithms such as rendering, physical simulation, and animation are typically deployed on cloud servers. The cloud servers perform these graphics processing tasks, and then compress and send the resulting images / videos to the edge devices for display.

[0004] To avoid wasting the computing power of edge devices by solely utilizing cloud computing capabilities for graphics processing, cloud servers can perform only computationally expensive graphics processing tasks. The intermediate results of this processing (e.g., rendering intermediate results in the case of rendering) are encoded and transmitted to the edge devices. The edge devices can then perform simple graphics processing on the decoded intermediate results to obtain high-quality visuals. Compared to directly transmitting the graphics-generated images from the cloud, edge-cloud collaborative graphics processing can improve the utilization of computing power on both the cloud and edge devices while saving bandwidth, achieving similar display effects.

[0005] However, in the edge-cloud collaborative solutions of related technologies, the cloud-side server has a high computational cost for intermediate results, and transmitting intermediate results requires a high bandwidth. Summary of the Invention

[0006] This application provides an image processing method and apparatus that can reduce the computational cost of intermediate results on cloud-side servers and reduce the amount of encoded data, thereby reducing the bandwidth occupied by transmitting intermediate results.

[0007] In a first aspect, this application provides an image processing method, comprising: acquiring a first region in an image to be processed, the first region being a region related to scene changes; performing graphic calculations on the first region to obtain a first intermediate result of the image to be processed; and encoding the obtained first intermediate result to obtain an encoded bitstream.

[0008] The first region is the region related to changes in the scene (such as a 3D scene). Factors that cause changes in the scene may include, but are not limited to, at least one of the following: changes in the position of objects in the scene, changes in time, changes in perspective, etc.

[0009] For example, a change in the position of an object in a 3D scene can cause a change in the scene corresponding to the image being processed: when a character in a cloud game enters a 3D scene, the lighting around the character needs to change, so the area around the character in the 3D scene can be an area related to the scene change.

[0010] For example, changes in time can cause changes in the scene corresponding to the image to be processed: for instance, the game time in cloud gaming changes, and based on the game logic of cloud gaming, some scenes in the game screen at the next moment are updated. The two-dimensional image area converted into the updated part of the scene is the area in the image to be processed that is related to the scene change.

[0011] For example, a change in perspective can cause a change in the scene corresponding to the image to be processed: for instance, a change in perspective in cloud gaming can cause changes in objects and lighting in certain areas of the game scene (an example of a 3D scene). The area in the image to be processed that causes changes in objects and lighting in the game scene due to the change in perspective is the area related to the scene change.

[0012] It should be understood that the first region defined in this application is the region related to scene changes. Therefore, the scene changes are not limited to changes in lighting within the scene, but can also include changes in objects within the scene (such as characters, objects, etc.). This is not a limitation.

[0013] For example, graphics computing may include, but is not limited to, one or more of the following: rendering processing, physical simulation, and animation processing.

[0014] For example, this method can be executed by the server. The following embodiments will illustrate the method by example.

[0015] In this embodiment of the application, graphical calculations are performed on the regions in the image to be processed that are related to scene changes to obtain the first intermediate result of the first region in the image to be processed. This significantly reduces the amount of computation and computational cost of the server performing graphical calculations on the image to be processed to obtain the intermediate result of the image to be processed. In addition, the first intermediate result of the first region is encoded during encoding, thereby reducing the amount of encoding and shortening the length of the bitstream, thereby reducing the proportion of the bitstream to the transmission bandwidth.

[0016] In one embodiment, the method further includes: encoding first information into a bitstream, the first information being used to indicate the location of a first region in the image to be processed.

[0017] In this way, the bitstream not only includes the encoded data of the first intermediate result of the first region, but also includes first information indicating the position of the first region in the image to be processed, so that the decoding end (e.g., the client) can use the decoded bitstream to reconstruct the reconstructed data of the intermediate result of the image to be processed (e.g., the rendering intermediate result of the image to be processed), without the end side (e.g., the client) having to calculate which region of the image to be processed is the first intermediate result, thereby reducing the amount of computation on the end side.

[0018] In one embodiment, obtaining the first region in the image to be processed includes: obtaining second information, which is used to divide the image to be processed into regions; and obtaining the first region in the image to be processed based on the second information.

[0019] In this embodiment of the application, second information for dividing the image to be processed into regions can be obtained, thereby dividing the image to be processed into regions to obtain a first region in the image to be processed that is related to scene changes, thus ensuring the accuracy of the first region.

[0020] In one embodiment, the method is performed by a first device, and the second information includes at least one of the following: scene information corresponding to the image to be processed; one or more of the following: network status information, computing power, power consumption, and thermal threshold of the first device and / or the second device; and communication link information between the first device and the second device, wherein the second device communicates with the first device.

[0021] In this embodiment, the server can use scene information corresponding to the image to be processed to divide the image into regions, so that the first region obtained is closely related to the scene of the image to be processed. Thus, if the scene corresponding to the image to be processed changes, the first region determined by the method of this embodiment based on the scene information can also flexibly change with the scene. Furthermore, the server can also use at least one of the following information—network status, computing power, battery level, and heat threshold—of the service and / or the second device communicating with the server (e.g., client 200)—to divide the image into regions. In this way, the first region divided by the server can be adaptively adjusted based on the network status, computing power, battery level, and heat threshold of both ends, avoiding unnecessary calculations, saving computational load on the first intermediate result, and reducing the transmission bandwidth of the bitstream encoded from the first intermediate result. Additionally, the server can adaptively determine the first region in the image to be processed based on the communication link information between the two ends, so that the size of the determined first region matches the status of the communication link. For example, this avoids situations where the communication link capacity is too small to transmit the bitstream, reduces network congestion, and reduces end-to-cloud interaction latency.

[0022] In one embodiment, the above-mentioned encoding of the first intermediate result to obtain an encoded bitstream includes: obtaining the encoding method of the first intermediate result; and encoding the first intermediate result based on the encoding method to obtain an encoded bitstream.

[0023] In this embodiment of the application, the server can flexibly determine the encoding method of the first intermediate result, thereby enabling flexible encoding of the first intermediate result.

[0024] In one embodiment, the method further includes: obtaining a reconstruction method of the image to be processed, the reconstruction method corresponding to the encoding method described above; and encoding the reconstruction method into a bitstream.

[0025] In this embodiment, the server can obtain a reconstruction method corresponding to the encoding method of the first intermediate result of the first region and encode the reconstruction method into the bitstream. Thus, the decoding side (e.g., the client) can reconstruct the intermediate result of the image to be processed according to the reconstruction method indicated by the server in the bitstream, and then perform further graphics calculations on the intermediate result of the image to be processed to obtain the graphics calculation result (e.g., the rendering result) of the image to be processed. The rendering result of the image to be processed is the rendered image. Therefore, the client does not need to determine the reconstruction method of the image to be processed itself; it can directly decode the reconstruction method from the bitstream, reducing the client's decoding computation load. Furthermore, since the reconstruction method corresponds to the encoding method of the first intermediate result, the reconstruction data (also called the reconstruction intermediate result) of the intermediate result of the image to be processed reconstructed using this reconstruction method can include the accurate reconstruction data of the first intermediate result.

[0026] In one embodiment, the encoding method includes: acquiring a reference frame, the reference frame being an intermediate result of a reference image corresponding to the image to be processed; and encoding the first intermediate result based on the reference frame.

[0027] The reference image corresponding to the image to be processed can be any one or more frames that precede the image to be processed in either the display order or the encoding order. Taking a cloud gaming scenario as an example, the reference image can be the previous frame of the game screen that was displayed before the image to be processed.

[0028] The reference frame in each embodiment of this application does not refer to the reference image corresponding to the image to be processed, but rather to the intermediate result corresponding to the reference image corresponding to the image to be processed. For example, taking graphics computation as rendering processing, the intermediate result corresponding to the reference image can be the rendering intermediate result obtained by rendering the complete area (also known as the whole scene) of the reference image, or the rendering intermediate result of the first area obtained by rendering the first area of ​​the reference image.

[0029] For example, the encoder in the server can obtain a reference frame and encode the first intermediate result row of the first region of the image to be processed based on the reference frame by means of intra-frame prediction or inter-frame prediction.

[0030] In this embodiment, the intermediate result of the reference image corresponding to the image to be processed (the intermediate result of the complete area of ​​the reference image or the intermediate result of the first area of ​​the reference image) is used as the reference frame for the first intermediate result of the first area in the image to be processed, and the first intermediate result is encoded. In this way, when the scene changes of the first area in two or more consecutive frames are small (or the scene changes are not obvious), such as when the lighting changes are small, the intermediate result of the previous frame or several previous frames can be used as the reference frame to encode the first intermediate result of the first area of ​​the current frame (i.e., the image to be processed). This reduces the amount of data encoded from the time dimension, shortens the length of the bitstream, and further reduces the bitstream's bandwidth usage.

[0031] In one embodiment, the encoding method includes: acquiring preset data corresponding to the image to be processed, the preset data including a first preset intermediate result corresponding to a first region; and encoding the first intermediate result according to the preset data.

[0032] In this embodiment, the server can encode a first intermediate result of a first region in the image to be processed based on preset data corresponding to the image to be processed. This preset data includes a first preset intermediate result corresponding to the first region. This allows the encoding of the first intermediate result of the first region to be achieved by referencing the preset data (specifically, the aforementioned first preset intermediate result) used for graphic calculations on the image to be processed within the spatial dimension of the image to be processed. Therefore, when a scene switch occurs from the previous frame to the current frame (specifically, the image to be processed), this embodiment, by referencing the preset data corresponding to the current frame to encode the first intermediate result of the first region in the current frame, can reduce the amount of data encoded (e.g., determining the difference data input to the encoder based on the preset data and the first intermediate result), thus improving encoding efficiency.

[0033] In one embodiment, the preset data is a preset intermediate result of the image to be processed, or the preset data is a preset intermediate result of a reference image corresponding to the image to be processed.

[0034] In this embodiment, the server 100 can encode the first intermediate result of the first region in the image to be processed based on the preset intermediate result of the image to be processed. In this way, when the image to be processed has a preset intermediate result pre-calculated, the preset intermediate result of the image to be processed can be directly reused to quickly encode the first intermediate result of the first region in the image to be processed, thereby improving the encoding efficiency.

[0035] Alternatively, in some embodiments, the preset data corresponding to the image to be processed is a preset intermediate result of a reference image corresponding to the image to be processed. Then, the first preset intermediate result corresponding to the first region is the preset data corresponding to the first region in the reference image corresponding to the image to be processed (i.e., the intermediate result obtained by pre-performing graphic calculations). The "reference image" mentioned in this embodiment is different from the "reference image" mentioned in the "reference frame" above. In this embodiment, the reference image corresponding to the image to be processed is a "nearby frame" of the image to be processed.

[0036] For example, if the server performs lighting calculations on each scene to obtain the preset data corresponding to each scene, then the preset data here is the preset data obtained by the server performing lighting calculations on the "nearest frames" (e.g., the entire scene) of the image to be processed.

[0037] In some scenarios, the server may not have pre-calculated the preset intermediate results for the image to be processed. However, it may have pre-calculated the preset intermediate results for the "nearest frames" of the image to be processed. The server can then use these preset intermediate results to obtain (e.g., through interpolation) the preset intermediate results for the current frame. In this way, the server can refer to the obtained preset intermediate results for the current frame to encode the first intermediate result for the first region in the image to be processed. This reduces the amount of encoded data and improves encoding efficiency when the scene changes within the current frame.

[0038] In one embodiment, encoding a first intermediate result based on preset data includes: obtaining a first preset intermediate result based on the preset data; obtaining first difference data based on the first intermediate result and the first preset intermediate result; and encoding the first difference data.

[0039] In this embodiment, a preset intermediate result for a first region in the image to be processed can be obtained based on preset data of the image to be processed. A difference data is then calculated between the first intermediate result and the preset intermediate result for the first region, and this difference data is encoded to achieve the encoding of the first intermediate result for the first region. In this scheme, the difference data between the preset data for the first region and the first intermediate result for the first region in the image to be processed can be calculated, and this difference data can be encoded (e.g., inputting the difference data into an encoder for encoding), thereby reducing the amount of encoded data, improving encoding efficiency, shortening the bitstream length, and reducing bandwidth usage.

[0040] In one embodiment, the first intermediate result includes a second intermediate result obtained by graphical calculation of a portion of the first region and a third intermediate result obtained by graphical calculation of another portion of the first region. Based on preset data, the preset intermediate result of the other portion (region) of the first region in the image to be processed is obtained. Based on the third intermediate result and the preset intermediate result of the other portion (region), a second difference data is obtained. The second difference data, the second intermediate result, and the third information are encoded, and the third information is used to indicate the position of a portion and / or another portion of the first region in the image to be processed.

[0041] The first region can be divided into at least two parts, taking region c and region d as examples. The first intermediate result of the first region can include intermediate result B1 of region c (an example of a third intermediate result) and intermediate result B2 of region d (an example of a second intermediate result). Similarly, the preset intermediate result A of the first region can also be divided into preset intermediate result A1 of region c and preset intermediate result A2 of region d. When the server encodes the first intermediate result (specifically intermediate result B1 and intermediate result B2) of the first region 11, it can calculate the difference C2 between intermediate result B1 of region c and preset intermediate result A1 of region c. Then, it can encode the difference C2 of region c, the intermediate result B2 of region d in the first intermediate result B, and the third information to achieve the encoding of the first intermediate result of the first region (composed of intermediate result B1 and intermediate result B2), obtaining the aforementioned bitstream.

[0042] The third piece of information may be information used to indicate the location of at least one of regions c and d in the first region.

[0043] In this embodiment, when encoding the first intermediate result of the first region in the image to be processed, the intermediate result of a portion of the first intermediate result can be directly encoded, and the intermediate result of another portion of the first intermediate result can be used to perform a difference calculation with reference to the preset data of that other portion of the first intermediate result, and the difference is encoded to achieve the encoding of the other portion of the intermediate result in the first intermediate result. Thus, when the preset data of the image to be processed may be incomplete, for example, the preset data may not include the preset intermediate results of the aforementioned portion of the first region, the server can flexibly apply a "difference encoding" scheme (specifically, calculating the difference between the intermediate result of the same region and the preset intermediate result, and encoding the difference data) to the regions in the image to be processed that have preset intermediate results on the client side, based on the regions corresponding to the preset data on the client side. For regions that do not have preset intermediate results on the client side, the aforementioned "direct encoding" scheme is used to flexibly and completely encode the first region of the image to be processed.

[0044] In one embodiment, the method further includes: obtaining indication information corresponding to the image to be processed, the indication information being used to indicate preset data corresponding to the image to be processed; and encoding the indication information into a bitstream.

[0045] In this embodiment, since the server performs graphic calculations on the first region in the current frame (specifically, the image to be processed) and encodes the first intermediate result of the first region obtained from the graphic calculation into the bitstream, the decoding end (e.g., the client) can reuse preset data from the current frame or adjacent frames and combine it with the first intermediate result to reconstruct the intermediate result of the entire scene reconstruction of the current frame. Therefore, the server can encode indication information for indicating the preset data corresponding to the image to be processed into the bitstream and synchronize it to the client, thereby facilitating the client to use the indication information to find the preset data corresponding to the image to be processed, and combine it with the first intermediate result to reconstruct the intermediate result of the image to be processed.

[0046] In addition, the server can encode the indication information used to indicate the preset data corresponding to the image to be processed into the bitstream. In this way, the decoding end (e.g., the client) can use the indication information obtained from the bitstream to directly obtain the preset data that needs to be referenced, so as to combine it with the first intermediate result of the first region obtained by decoding to reconstruct the intermediate result of the image to be processed, thereby improving the reconstruction efficiency of the image to be processed.

[0047] In one embodiment, the above-mentioned indication information includes the address for obtaining preset data corresponding to the image to be processed.

[0048] In this embodiment of the application, the server can encode the acquisition address of the preset data corresponding to the image to be processed into the bitstream and synchronize it to the client, so that the client can use the acquisition address to obtain the preset data corresponding to the image to be processed, and use it to decode and obtain the intermediate reconstruction result of the image to be processed.

[0049] In one embodiment, the method further includes: obtaining a second region in the image to be processed based on the second information, wherein the scene change degree of the second region is lower than that of the first region; and encoding the fourth information into a bitstream, wherein the fourth information is used to indicate the position of the second region in the image to be processed.

[0050] In this embodiment, the server encodes the fourth information indicating the position of the second region in the image to be processed into the bitstream. This allows the client to determine the second region in the image to be processed according to the fourth information. The client then uses the second region and the first intermediate result corresponding to the first region to reconstruct the intermediate result of the image to be processed, thereby reducing the amount of computation required for the client to determine the second region.

[0051] Secondly, this application provides an image processing method, comprising: decoding a bitstream to obtain decoded data corresponding to a first intermediate result of an image to be processed, wherein the first intermediate result is obtained by performing a first graphic calculation on a first region in the image to be processed, and the first region is a region related to scene changes; and obtaining a graphic calculation result of the image to be processed based on the decoded data corresponding to the first intermediate result.

[0052] Taking the client-side execution of this method as an example, the client can decode the bitstream using a standard video decoder to obtain the decoded data corresponding to the first intermediate result, such as the first intermediate result or the difference data corresponding to the first intermediate result. This standard video decoder can be implemented according to industry video compression standards, such as ITU-T H.264, H.265 / HEVC, H.266 / VVC, AVS2, AVS3 standards, or extensions of such standards. The technology in this application is not limited to any specific decoding standard.

[0053] For a detailed introduction to the first region, please refer to the introduction of the first region in the first aspect above, which will not be repeated here.

[0054] In this embodiment, the client can decode the bitstream to obtain decoded data corresponding to the first intermediate result of the image to be processed. This first intermediate result is obtained by performing graphics calculations on a first region of the image to be processed that is related to scene changes. Compared to decoding the bitstream to obtain decoded data for the entire scene of the image to be processed, this embodiment can decode the bitstream to obtain decoded data corresponding to the first intermediate result of a local region of the image to be processed. This reduces the amount of decoded data, improves decoding efficiency, reduces the bitstream's bandwidth usage, and thus reduces latency. Based on the decoded data corresponding to the first intermediate result, the client can obtain the graphics calculation result of the image to be processed. Therefore, even when decoding the decoded data corresponding to the intermediate result of a partial region of the image to be processed, the client can still reconstruct the graphics calculation result of the entire scene of the image to be processed, ensuring the accuracy of the graphics calculation result.

[0055] In one embodiment, obtaining the graphic calculation result of the image to be processed based on the decoded data corresponding to the first intermediate result includes: acquiring first information, which is used to indicate the position of the first region in the image to be processed; and obtaining the graphic calculation result of the image to be processed based on the first information and the decoded data corresponding to the first intermediate result.

[0056] In this embodiment, the client can obtain first information indicating the position of a first region related to scene changes in the image to be processed. This first information allows the client to determine the location of the first region in the image to be processed that corresponds to the first intermediate result. Thus, the client can use the position of the first region in the image to be processed corresponding to the first intermediate result, along with the decoded data corresponding to the first intermediate result, to obtain the graphics calculation result of the image to be processed, ensuring the accuracy of the graphics calculation result obtained by the client.

[0057] In one embodiment, obtaining the graphic calculation result of the image to be processed based on the decoded data corresponding to the first information and the first intermediate result includes: obtaining the reconstruction method of the image to be processed; and obtaining the graphic calculation result of the image to be processed according to the reconstruction method based on the first information and the decoded data.

[0058] In this embodiment, the client 200 can obtain the reconstruction method of the image to be processed, and obtain the graphics calculation result of the image to be processed according to the reconstruction method based on the first information and the decoded data. Thus, the client 200 can reconstruct the intermediate result of the image to be processed according to the reconstruction method, and then use the reconstructed intermediate result to obtain the graphics calculation result of the image to be processed, thereby improving the quality of the graphics calculation result of the image to be processed.

[0059] In one embodiment, the reconstruction method of the image to be processed is obtained based on the first information.

[0060] In this embodiment, the client can determine the reconstruction method of the image to be processed independently based on the position of the first region related to scene changes indicated by the first information in the image to be processed. This allows for flexible determination of the reconstruction method of the intermediate results of the image to be processed, without relying on the reconstruction method decided by the server, thus improving the client's flexibility in reconstructing the intermediate results of the image to be processed. For example, the client can flexibly decide which part of the intermediate result calculated by the server should be used to reconstruct the intermediate results of the image to be processed based on the first information. Furthermore, since the client can determine the reconstruction method of the intermediate results of the image to be processed independently based on the first information, the server does not need to send the reconstruction method to the client through the bitstream, thereby reducing the data in the bitstream and reducing the bandwidth consumption of the bitstream.

[0061] In one embodiment, the reconstruction method of the image to be processed is obtained from the bitstream.

[0062] In this embodiment, the client can decode the reconstruction method of the image to be processed from the bitstream, and then obtain the intermediate reconstruction result of the image to be processed based on the first intermediate result according to the reconstruction method, since the reconstruction method corresponds to the encoding method of the first intermediate result. In this way, the reconstruction quality of the intermediate reconstruction result of the image to be processed obtained according to the reconstruction method is higher, thereby improving the quality of the obtained graphics calculation result.

[0063] In one embodiment, obtaining the graphic calculation result of the image to be processed based on the decoded data corresponding to the first information and the first intermediate result includes: acquiring preset data corresponding to the image to be processed, the preset data including the first preset intermediate result corresponding to the first region in the image to be processed; and obtaining the graphic calculation result of the image to be processed based on the preset data, the first information and the decoded data corresponding to the first intermediate result.

[0064] In this embodiment, the client can obtain preset data corresponding to the image to be processed, and obtain the graphics calculation result of the image to be processed based on the preset data, first information indicating the position of the first region in the image to be processed, and decoding data corresponding to the first intermediate result of the first region. The preset data includes the first preset intermediate result corresponding to the first region, so that the intermediate result (e.g., reconstruction intermediate result) of the image to be processed can be reconstructed by referring to the preset data (specifically the aforementioned first preset intermediate result) for graphics calculation of the image to be processed in the spatial dimension of the image to be processed. Therefore, when a scene switch occurs from the previous frame to the current frame (specifically the image to be processed), this embodiment can reconstruct the reconstruction intermediate result of the current frame by referring to the preset data corresponding to the current frame and the decoding data corresponding to the first intermediate result of the first region of the current frame, and then obtain the graphics calculation result of the current frame, which can reduce the amount of data to be decoded and improve decoding accuracy and decoding efficiency.

[0065] In one embodiment, the preset data is a preset intermediate result of the image to be processed, or the preset data is a preset intermediate result of a reference image corresponding to the image to be processed.

[0066] In this embodiment, the client can reconstruct the intermediate reconstruction result of the image to be processed based on the preset intermediate result of the image to be processed, the decoding data corresponding to the first intermediate result of the first region in the image to be processed, and the first information. This allows for the decoding of the intermediate reconstruction result of the image to be processed in the spatial dimension of the image to be processed, referencing the preset data used for graphic calculations on the image to be processed (specifically, the preset intermediate result of the image to be processed). Therefore, when a scene switch occurs from the previous frame to the current frame (specifically, the image to be processed), compared to intra-frame decoding of the first intermediate result of the current frame or using the intermediate reconstruction result of the previous frame as a reference frame to decode the intermediate reconstruction result of the current frame, this embodiment obtains the intermediate reconstruction result of the current frame by referencing the preset intermediate result of the current frame and the decoding data corresponding to the first intermediate result of the first region in the current frame. This reduces the amount of decoding data and improves decoding efficiency.

[0067] Alternatively, in some embodiments, the preset data corresponding to the image to be processed is: a preset intermediate result of a reference image corresponding to the image to be processed.

[0068] In this embodiment of the application, the reference image corresponding to the image to be processed is the "nearby frame" of the image to be processed.

[0069] For example, if the server pre-calculates the lighting for each scene, then the preset data here is the preset data obtained by the server in pre-calculating the lighting for the "nearest frames" (e.g., the entire scene) of the image to be processed.

[0070] In some scenarios, the server may not have pre-calculated the preset intermediate results for the image to be processed. However, it may have pre-calculated the preset intermediate results for the "nearby frames" of the image to be processed. The client can then use the preset intermediate results of the "nearby frames" to obtain (e.g., through interpolation) the preset intermediate results for the current frame. In this way, the client can refer to the decoded data corresponding to the obtained preset intermediate results for the current frame and the first intermediate results for the first region in the image to be processed to reconstruct the reconstructed intermediate results for the image to be processed. This reduces the amount of decoded data and improves decoding efficiency when the scene changes within the current frame.

[0071] In one embodiment, the decoded data is used to indicate first difference data, and the graphic calculation result of the image to be processed is obtained based on preset data, first information, and the first difference data. For example, the first difference data is the difference between a first intermediate result of the image to be processed and a preset intermediate result of a first region in the image to be processed.

[0072] In this embodiment, the client decodes the bitstream to obtain the first difference data corresponding to the first intermediate result. Compared to the data volume of the first intermediate result, the difference data has a smaller data volume, thereby improving decoding efficiency. Furthermore, when the decoded data obtained from decoding the bitstream indicates the first difference data corresponding to the first intermediate result, the client can obtain a preset intermediate result (e.g., preset data for the entire scene) of the image to be processed based on preset data of the image to be processed. This allows the client to reuse the preset intermediate result of the entire scene of the image to be processed and the decoded first difference data to reconstruct the reconstruction intermediate result of the entire scene of the image to be processed. In this process, the client does not need to perform graphic calculations on regions other than the first region (e.g., the second region) in the image to be processed. Instead, it directly reuses the preset data of the image to be processed, combining it with the first intermediate result of the first region calculated in real time by the server, to reconstruct the reconstruction intermediate result of the image to be processed. This avoids redundant graphic calculations on the regions (e.g., the second region) where the scene has not changed, both on the client side and the cloud side (e.g., the server), improving the reconstruction efficiency of the reconstruction intermediate result of the image to be processed and the efficiency of obtaining the graphic calculation results of the image to be processed.

[0073] In one embodiment, the decoded data is used to indicate a first intermediate result. Based on the preset data and the first information, a preset intermediate result of a second region in the image to be processed is obtained. The second region is a region in the image to be processed that is different from the first region. Based on the first intermediate result and the preset intermediate result of the second region, the graphic calculation result of the image to be processed is obtained.

[0074] In this embodiment of the application, when the client obtains the graphics calculation result of the image to be processed, it can obtain the preset intermediate result of the region other than the first region (e.g., the second region) in the preset data of the image to be processed, and based on the first intermediate result of the first region obtained by decoding and the preset intermediate result of the second region (referred to as "stitching"), it can obtain, for example, the reconstruction data of the intermediate result of the complete region of the image to be processed including the first region and the second region, and then obtain the graphics calculation result of the image to be processed, thereby improving the flexibility of the client in obtaining the graphics calculation result of the image to be processed.

[0075] In one embodiment, the decoded data includes third information, a second intermediate result obtained by first graphic calculation of a portion (region) of the first region, and second difference data corresponding to another portion (region) of the first region. The third information is used to indicate the position of a portion (region) and / or another portion (region) of the first region in the image to be processed. The second difference data is the difference between the third intermediate result obtained by first graphic calculation of the other portion (region) and a preset intermediate result of the other portion (region). Then, based on the preset data and the third information, a preset intermediate result of a third region in the image to be processed, excluding the aforementioned portion (region), is obtained. Based on the preset intermediate result of the third region, the second difference data, and the second intermediate result, the graphic calculation result of the image to be processed is obtained.

[0076] For example, the third region may include the second region, as well as the other portion (region) of the first region described above.

[0077] In this embodiment, the client decodes the bitstream to obtain the difference of another part in the first region, an intermediate result of a part of the first region (an intermediate result obtained by the server through the first graphic calculation), and information indicating the position of the aforementioned part or the other part in the first region. This allows for the reconstruction of the intermediate results of the entire scene of the image to be processed using the decoded data. This increases the reconstruction complexity while ensuring secure data decoding. Furthermore, the reconstruction method used by the client to reconstruct the intermediate results of the image to be processed can be dynamically adjusted based on the preset intermediate results of the image to be processed and the first intermediate results of the first region of the image to be processed. This is equivalent to transforming the data of the first intermediate results based on the preset data corresponding to the image to be processed, making the transformed first intermediate results more suitable for encoding and decoding, thus improving decoding efficiency.

[0078] In this embodiment, considering that the client side may not have a preset intermediate result for a portion of the first region of the image to be processed, the client can directly decode from the bitstream to obtain the second intermediate result obtained by the server performing the first graphic calculation on the aforementioned portion. If the client side has a preset intermediate result for another portion of the first region of the image to be processed, the first intermediate result of the first region can be decoded by superimposing the second difference data corresponding to that other portion with the preset intermediate result of the other portion. Thus, when the preset data of the image to be processed may be incomplete, the client side can flexibly decode the bitstream from the server side (for example, "directly decode" the aforementioned portion using reconstruction method 1, and "superimpose" decode the aforementioned other portion using reconstruction method 2) to flexibly and completely encode the first intermediate result of the first region of the image to be processed.

[0079] In one embodiment, indication information corresponding to the image to be processed can be obtained from the bitstream. This indication information is used to indicate preset data corresponding to the image to be processed. Then, the preset data corresponding to the image to be processed can be obtained based on the indication information.

[0080] In this embodiment, since the client can only decode the first intermediate result of the first region in the bitstream, and cannot decode the intermediate result of the region other than the first region in the image to be processed obtained by the server's graphics calculation, the decoding end (e.g., client 200) can reuse the preset data of the current frame or adjacent frames and combine it with the first intermediate result to reconstruct the reconstruction intermediate result of the entire scene of the current frame. Therefore, the client can decode the indication information from the bitstream to indicate the preset data corresponding to the image to be processed, so that the client can use the indication information to find the preset data corresponding to the image to be processed, and combine it with the first intermediate result to reconstruct the reconstruction intermediate result of the image to be processed. In addition, when reconstructing the reconstruction intermediate result of the current frame, if the preset data referenced is different from the preset data referenced by the previously decoded frame (e.g., the previous frame), the client can decode the indication information from the bitstream to obtain the preset data that can reconstruct the reconstruction intermediate result of the current frame, ensuring the client's accurate decoding of the reconstruction intermediate result of the current frame.

[0081] In one embodiment, the above-mentioned indication information includes the address for obtaining preset data corresponding to the image to be processed.

[0082] In this embodiment of the application, if the client does not store the preset data corresponding to the image to be processed locally, the client can decode the acquisition address of the preset data corresponding to the image to be processed from the bitstream, and access the preset data according to the acquisition address, so as to decode the intermediate reconstruction result of the image to be processed.

[0083] In one embodiment, this initial information can be obtained from the bitstream.

[0084] In this embodiment of the application, the client can decode the first information indicating the position of the first region in the image to be processed from the bitstream, so as to reduce the amount of computation required for the client to calculate the first information. Even in scenarios where the client's computing performance is poor, the reconstruction of the graphic calculation result of the image to be processed can still be achieved.

[0085] In one embodiment, one way to obtain the above-mentioned decoded data includes: obtaining a reference frame, which is reconstructed data of an intermediate result of a reference image corresponding to the image to be processed; and decoding the bitstream based on the reference frame to obtain decoded data corresponding to the first intermediate result of the image to be processed.

[0086] The reference image corresponding to the image to be processed can be any one or more frames that precede the image to be processed in either the display order or the encoding order. Taking a cloud gaming scenario as an example, the reference image can be the previous frame of the game screen that was displayed before the image to be processed.

[0087] The reference frame in the various embodiments of this application does not refer to the reference image corresponding to the image to be processed, but rather to the reconstruction data of the intermediate result corresponding to the reference image corresponding to the image to be processed. For example, taking graphics computing as a rendering process, the reconstruction data of the intermediate result corresponding to the reference image can be the reconstruction data of the rendering intermediate result obtained by rendering the complete area (also known as the whole scene) of the reference image, or the reconstruction data of the rendering intermediate result of the first area obtained by rendering the first area of ​​the reference image.

[0088] In this embodiment, a reference frame can be obtained, wherein the reference frame is the reconstructed data of the intermediate result of the reference image corresponding to the image to be processed. The bitstream can be decoded based on the reference frame (e.g., through inter-frame prediction or intra-frame prediction, which is not limited here) to obtain the decoded data corresponding to the first intermediate result of the image to be processed. In this way, when the scene changes of the first region in two or more consecutive frames are small (or the scene changes are not obvious), such as when the lighting changes are small, the reconstructed data of the intermediate result corresponding to the previous frame or several previous frames can be used as the reference frame. The bitstream can be decoded using the reference frame from the time dimension, thereby obtaining the decoded data corresponding to the first intermediate result of the first region of the current frame (i.e., the image to be processed). This can reduce the amount of data to be decoded and improve decoding efficiency.

[0089] In one embodiment, obtaining the graphic calculation result of the image to be processed based on the decoded data corresponding to the first intermediate result includes: obtaining the reconstructed data of the intermediate result of the image to be processed based on the decoded data corresponding to the first intermediate result; and performing a second graphic calculation on the reconstructed data to obtain the graphic calculation result.

[0090] In this embodiment of the application, the client can use the decoding data corresponding to the first intermediate result of the first region of the image to be processed obtained by decoding the bitstream to obtain the reconstruction data of the intermediate result of the image to be processed. In this way, the client does not need to perform graphics calculations on the first region related to scene changes, thereby reducing the pressure of graphics calculations on the end side and improving the efficiency of obtaining the graphics calculation results of the image to be processed.

[0091] In one embodiment, the method further includes saving the reconstructed data to a local or remote server.

[0092] In this embodiment of the application, by saving the intermediate reconstruction result of the current frame on a local or remote server, the reconstruction result can be used as the preset data for the subsequent frames of the current frame, so as to reconstruct the intermediate reconstruction result of the subsequent frames. This can realize the updating of the preset data saved by the client, and also facilitate the sharing of preset data between the client and other terminals, thereby reducing the computing overhead of the cloud.

[0093] In one embodiment, before decoding the bitstream, the method further includes: obtaining second information used to divide the image to be processed into regions; and then sending the second information. For example, the second information can be sent to a server.

[0094] In this embodiment of the application, the client can send second information for dividing the image to be processed into regions (e.g., dividing the first region mentioned above), so that the encoding side (e.g., the server) can obtain the first region related to scene changes based on the received second information, which can improve the processing speed of the server in the overall process of graphics calculation of the image to be processed.

[0095] In one embodiment, the method is executed by a second device, and the second information includes at least one of the following: scene information corresponding to the image to be processed; one or more of the following: network status information, computing power, battery level, and thermal threshold of the first device and / or the second device; and communication link information between the first device and the second device, wherein the second device communicates with the first device. For example, the second device is a client device, and the first device is a server device.

[0096] In this embodiment, the client can obtain scene information corresponding to the image to be processed, which can be used by the server or client to divide the image to be processed into regions, so that the divided first region is closely related to the scene of the image to be processed. In addition, the client can also obtain at least one of the network status information, computing power, power consumption, and heat threshold of the server and / or the client communicating with the server, which can be used by the server or client to divide the image to be processed into regions, to ensure that at least one of the divided first region and second region can be adaptively adjusted based on the network status, computing power, power consumption, heat threshold, etc. of both ends. In addition, the client can also obtain the communication link information between the two ends for the purpose of dividing the image to be processed into regions, so that the divided first region can match the status of the communication link.

[0097] In one embodiment, the method further includes sending first information before decoding the bitstream.

[0098] In some embodiments, the client may send the first information to the server before receiving the bitstream. In this way, the server does not need to use the second information for region division to divide the image to be processed into regions in order to obtain the first information. The server can directly receive the first information from the client, which can reduce the amount of computation on the server.

[0099] Thirdly, this application provides an image processing apparatus, the apparatus comprising: an acquisition module for acquiring a first region in an image to be processed, the first region being a region related to scene changes; a graphics calculation module for performing graphics calculations on the first region to obtain a first intermediate result of the image to be processed; and an encoding module for encoding the first intermediate result to obtain an encoded bitstream.

[0100] In one embodiment, the encoding module is further configured to encode first information into the bitstream, the first information being used to indicate the position of the first region in the image to be processed.

[0101] In one embodiment, the acquisition module is specifically used to: acquire second information, the second information being used to divide the image to be processed into regions; and based on the second information, obtain the first region in the image to be processed.

[0102] In one embodiment, the image processing apparatus operates on a first device, and the second information includes at least one of the following: scene information corresponding to the image to be processed; one or more of the following: network status information, computing power, power consumption, and thermal threshold of the first device and / or the second device; and communication link information between the first device and the second device. The second device communicates with the first device.

[0103] In one embodiment, the encoding module is specifically used to: obtain the encoding method of the first intermediate result; and encode the first intermediate result based on the encoding method to obtain the bitstream.

[0104] In one embodiment, the acquisition module is further configured to acquire the reconstruction method of the image to be processed, the reconstruction method corresponding to the encoding method; the encoding module is further configured to encode the reconstruction method into the bitstream.

[0105] In one embodiment, the encoding module is specifically configured to: acquire a reference frame, wherein the reference frame is an intermediate result of a reference image corresponding to the image to be processed; and encode the first intermediate result according to the reference frame.

[0106] In one embodiment, the encoding module is specifically configured to: acquire preset data corresponding to the image to be processed, the preset data including a first preset intermediate result corresponding to the first region; and encode the first intermediate result according to the preset data.

[0107] In one embodiment, the preset data is a preset intermediate result of the image to be processed, or the preset data is a preset intermediate result of a reference image corresponding to the image to be processed.

[0108] In one embodiment, the encoding module is specifically used to: obtain a first preset intermediate result based on the preset data; obtain a first difference data based on the first intermediate result and the first preset intermediate result; and encode the first difference data.

[0109] In one embodiment, the first intermediate result includes a second intermediate result obtained by the graphic calculation of a portion of the first region and a third intermediate result obtained by the graphic calculation of another portion of the first region. The encoding module is specifically configured to: obtain a preset intermediate result of the other portion of the first region in the image to be processed based on the preset data; obtain second difference data based on the third intermediate result and the preset intermediate result of the other portion; and encode the second difference data, the second intermediate result, and the third information, wherein the third information is used to indicate the position of a portion of the first region and / or the other portion in the image to be processed.

[0110] In one embodiment, the acquisition module is further configured to acquire indication information corresponding to the image to be processed, the indication information being used to indicate preset data corresponding to the image to be processed; the encoding module is further configured to encode the indication information into the bitstream.

[0111] In one embodiment, the indication information includes the address for obtaining preset data corresponding to the image to be processed.

[0112] In one embodiment, the acquisition module is further configured to obtain a second region in the image to be processed based on the second information, wherein the scene change degree of the second region is lower than that of the first region; the encoding module is further configured to encode the fourth information into the bitstream, wherein the fourth information is used to indicate the position of the second region in the image to be processed.

[0113] Fourthly, this application provides an image processing apparatus, comprising: a decoding module for decoding a bitstream to obtain decoded data corresponding to a first intermediate result of an image to be processed, wherein the first intermediate result is obtained by performing a first graphic calculation on a first region in the image to be processed, and the first region is a region related to scene changes; and an acquisition module for obtaining a graphic calculation result of the image to be processed based on the decoded data corresponding to the first intermediate result.

[0114] In one embodiment, the acquisition module is specifically configured to: acquire first information, the first information indicating the position of the first region in the image to be processed; and obtain the graphic calculation result of the image to be processed based on the first information and the decoded data corresponding to the first intermediate result.

[0115] In one embodiment, the acquisition module is specifically used to: acquire the reconstruction method of the image to be processed; and obtain the graphic calculation result of the image to be processed according to the reconstruction method based on the first information and the decoded data.

[0116] In one embodiment, the acquisition module is specifically used to acquire the reconstruction method of the image to be processed based on the first information.

[0117] In one embodiment, the decoding module is further configured to obtain the reconstruction method of the image to be processed from the bitstream.

[0118] In one embodiment, the acquisition module is specifically used to: acquire preset data corresponding to the image to be processed, the preset data including a first preset intermediate result corresponding to the first region in the image to be processed; and obtain the graphic calculation result of the image to be processed based on the preset data, the first information and the decoding data corresponding to the first intermediate result.

[0119] In one embodiment, the preset data is a preset intermediate result of the image to be processed, or the preset data is a preset intermediate result of a reference image corresponding to the image to be processed.

[0120] In one embodiment, the decoded data is used to indicate first difference data, and the acquisition module is specifically used to: obtain the graphic calculation result of the image to be processed based on the preset data, the first information and the first difference data; the first difference data is the difference between the first intermediate result of the image to be processed and the preset intermediate result of the first region in the image to be processed.

[0121] In one embodiment, the decoded data is used to indicate the first intermediate result, and the acquisition module is specifically used to: obtain a preset intermediate result of a second region in the image to be processed based on the preset data and the first information, wherein the second region is a region in the image to be processed that is different from the first region; and obtain the graphic calculation result of the image to be processed based on the first intermediate result and the preset intermediate result of the second region.

[0122] In one embodiment, the decoded data includes third information, a second intermediate result obtained by the first graphic calculation of a portion of the first region, and second difference data corresponding to another portion of the first region. The third information is used to indicate the position of a portion and / or another portion of the first region in the image to be processed. The second difference data is the difference between the third intermediate result obtained by the first graphic calculation of the other portion and the preset intermediate result of the other portion. The acquisition module is specifically used to: obtain a preset intermediate result of a third region in the image to be processed, excluding the portion, based on the preset data and the third information; and obtain the graphic calculation result of the image to be processed based on the preset intermediate result of the third region, the second difference data, and the second intermediate result.

[0123] In one embodiment, the decoding module is further configured to obtain indication information corresponding to the image to be processed from the bitstream, the indication information being used to indicate preset data corresponding to the image to be processed; the acquisition module is specifically configured to obtain the preset data corresponding to the image to be processed based on the indication information.

[0124] In one embodiment, the indication information includes the address for obtaining preset data corresponding to the image to be processed.

[0125] In one embodiment, the decoding module is further configured to obtain the first information from the bitstream.

[0126] In one embodiment, the decoding module is specifically configured to: acquire a reference frame, wherein the reference frame is reconstructed data of an intermediate result of a reference image corresponding to the image to be processed; and decode the bitstream based on the reference frame to obtain decoded data corresponding to a first intermediate result of the image to be processed.

[0127] In one embodiment, the acquisition module is specifically used to: obtain reconstructed data of the intermediate result of the image to be processed based on the decoded data corresponding to the first intermediate result; and perform a second graphic calculation on the reconstructed data to obtain the graphic calculation result.

[0128] In one embodiment, the apparatus further includes a storage module for storing the reconstructed data to a local or remote server.

[0129] In one embodiment, the acquisition module is further configured to acquire second information, the second information being used to divide the image to be processed into regions; the device further includes: a first sending module, configured to send the second information.

[0130] In one embodiment, the image processing device operates on a second device, and the second information includes at least one of the following: scene information corresponding to the image to be processed; one or more of the following: network status information, computing power, battery level, and thermal threshold of the first device and / or the second device; and communication link information between the first device and the second device. The second device communicates with the first device.

[0131] In one embodiment, the apparatus further includes a second transmitting module for transmitting the first information.

[0132] Fifthly, this application provides an end-to-end cloud collaboration system, comprising: a client and a server, wherein the server is configured to execute the method as described in the first aspect or any embodiment of the first aspect above, and the client is configured to execute the method as described in the second aspect or any embodiment of the second aspect above.

[0133] Sixthly, this application provides an edge-cloud collaborative system, including a client and a server. The client is used to send a graphics calculation request for an image to be processed to the server. The server is used to respond to the graphics calculation request by obtaining a first region in the image to be processed, wherein the first region is a region related to scene changes; performing graphics calculation on the first region to obtain a first intermediate result of the image to be processed; encoding the first intermediate result to obtain an encoded bitstream; and sending the bitstream to the client. The client is also used to receive the bitstream sent by the server, decode the bitstream to obtain decoded data corresponding to the first intermediate result of the image to be processed; and the client is also used to obtain a graphics calculation result of the image to be processed based on the decoded data corresponding to the first intermediate result.

[0134] In a seventh aspect, this application provides an apparatus comprising: one or more processors; a memory for storing one or more programs; wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in the first aspect or any embodiment of the first aspect above.

[0135] Eighthly, this application provides an apparatus comprising: one or more processors; a memory for storing one or more programs; wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in the second aspect or any embodiment of the second aspect above.

[0136] Ninthly, this application provides a computer-readable storage medium including a computer program that, when executed on a device, causes the device to perform the method described in the first aspect or any embodiment of the first aspect.

[0137] In a tenth aspect, this application provides a computer-readable storage medium including a computer program that, when executed on a device, causes the device to perform the method described in the second aspect or any embodiment of the second aspect.

[0138] In one aspect, this application provides a computer program that, when executed by a device, is used to perform the method described in the first aspect or any embodiment of the first aspect, or to perform the method described in the second aspect or any embodiment of the second aspect.

[0139] In a twelfth aspect, this application provides a computer program product comprising computer program code that, when executed on a device, causes the device to perform the method described in the first aspect or any embodiment of the first aspect, or to perform the method described in the second aspect or any embodiment of the second aspect.

[0140] In a thirteenth aspect, this application provides a bitstream generated according to the method described in the first aspect or any embodiment of the first aspect.

[0141] In a fourteenth aspect, this application provides a computer-readable storage medium storing a bitstream generated according to the method in any of the embodiments of the first aspect above.

[0142] In a fifteenth aspect, an apparatus for storing a bitstream is provided, the apparatus comprising: a transceiver unit and a storage unit, the transceiver unit being configured to receive a bitstream generated according to the method described in accordance with the first aspect or any embodiment of the first aspect, and the storage unit being configured to store the bitstream.

[0143] In a sixteenth aspect, an apparatus for transmitting a bitstream is provided, the apparatus comprising: a storage unit and a transceiver unit, the storage unit being configured to store a bitstream generated according to the method described in accordance with the first aspect or any embodiment thereof, and the transceiver unit being configured to transmit the bitstream.

[0144] In a seventeenth aspect, a system for distributing bitstreams is provided, the system comprising: at least one storage medium for storing a bitstream generated according to the method described in accordance with the first aspect or any embodiment of the first aspect; and a streaming media device for acquiring the bitstream from the at least one storage medium and transmitting the bitstream, wherein the streaming media device includes a content server or a content distribution server.

[0145] Eighteenthly, a transcoding system is provided, the transcoding system comprising: at least one storage medium for storing a bitstream generated according to the method described in the first aspect or any embodiment of the first aspect; and a transcoding device for acquiring the bitstream from the at least one storage medium and transcoding the bitstream.

[0146] In one embodiment, the transcoding device can convert the bitstream into MPEG-4 Part 4 (MP4), Matroska Video (MKV), Audio Video Interleave (AVI), Digital Audio Video (DAV), etc., without limitation. Attached Figure Description

[0147] Figure 1a is a schematic diagram of an application scenario provided by an embodiment of this application;

[0148] Figure 1b is a rendering diagram of a three-dimensional scene provided in an embodiment of this application;

[0149] Figure 1c is a schematic diagram of an end-to-cloud collaborative system provided in an embodiment of this application;

[0150] Figure 2a is a schematic diagram of an image processing procedure provided in an embodiment of this application;

[0151] Figure 2b is a schematic diagram of an encoding framework provided in an embodiment of this application;

[0152] Figure 3a is a schematic diagram of an image processing procedure provided in an embodiment of this application;

[0153] Figure 3b is a schematic diagram of a decoding framework provided in an embodiment of this application;

[0154] Figure 4 is a flowchart of process 400 of an image processing method provided in an embodiment of this application;

[0155] Figure 5a is a schematic diagram of an application scenario provided by an embodiment of this application;

[0156] Figure 5b is a schematic diagram of an application scenario of an image processing method provided in an embodiment of this application;

[0157] Figure 5c is a schematic diagram of an application scenario of an image processing method provided in an embodiment of this application;

[0158] Figure 6 is a flowchart of process 600 of an image processing method provided in an embodiment of this application;

[0159] Figure 7a is a schematic diagram of an application scenario of an image processing method provided in an embodiment of this application;

[0160] Figure 7b is a schematic diagram of an application scenario of an image processing method provided in an embodiment of this application;

[0161] Figure 7c is a schematic diagram of an application scenario of an image processing method provided in an embodiment of this application;

[0162] Figure 7d is a schematic diagram of an application scenario of an image processing method provided in an embodiment of this application;

[0163] Figure 8 is a schematic diagram of the architecture of an edge-cloud system provided in an embodiment of this application;

[0164] Figure 9a is a schematic diagram of rendering intermediate data provided in an embodiment of this application;

[0165] Figure 9b is a schematic diagram of pre-calculated intermediate rendering data provided in an embodiment of this application;

[0166] Figure 10 is a schematic diagram of the architecture of an edge-cloud system provided in an embodiment of this application;

[0167] Figure 11 is a schematic diagram of the architecture of an edge-cloud system provided in an embodiment of this application;

[0168] Figure 12 is a schematic diagram of the structure of an image processing device 700 provided in an embodiment of this application;

[0169] Figure 13 is a schematic diagram of the structure of an image processing device 800 provided in an embodiment of this application. Detailed Implementation

[0170] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0171] The terms "first," "second," etc., used in the specification, embodiments, claims, and drawings of this application are for distinguishing purposes only and should not be construed as indicating or implying relative importance or order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, such as including a series of steps or units. A method, system, product, or apparatus is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0172] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0173] Before introducing the technical solutions of the embodiments of this application, the technical names involved in this application will first be explained and described:

[0174] 1) Graphics computation: This refers to the processing method of the image to be processed. Graphics computation can be any of the following: rendering, physical simulation or animation processing.

[0175] 2) Graphical calculation results: These represent the data generated after performing graphical calculations on the image to be processed.

[0176] Specifically, when the graphics calculation is for rendering, the graphics calculation result is the rendering result of the three-dimensional scene corresponding to the image to be processed; when the graphics calculation is for physical simulation, the graphics calculation result is the result of the change law of the physical properties of the simulated object in the above three-dimensional scene; when the graphics calculation is for animation processing, the graphics calculation result is the animation description result of the object to be processed in the three-dimensional scene.

[0177] 3) Intermediate results: These represent the intermediate data generated during the process of generating the graphic calculation results from the image (or local area) to be processed.

[0178] 4) Rendering result: When graphics calculation is in the rendering process, the graphics rendering engine renders the image to be processed, and the resulting image is the rendering result of the image to be processed, such as the image or video to be displayed.

[0179] 5) Rendering Intermediate Results: This is a type of intermediate result, representing the intermediate data generated by the graphics rendering engine during the rendering process of the image to be processed, used to generate the image / video to be displayed. Rendering intermediate results may include, but are not limited to: albedo maps, normal maps, low-quality rendering results, position maps, specular intensity maps, depth maps, ambient occlusion maps, graphic motion vectors, and one or more other types of information generated during the rendering process.

[0180] 6) Rendering parameters: Inputs to the graphics rendering engine, including at least one of the following: various rendering parameters, viewpoint, light source position, color vector, position information of objects in the image to be processed (e.g., position vector), texture sampling method information, motion trajectory information of moving objects, skeletal animation parameters, etc. The objects in the image to be processed can be characters (e.g., players or observers), objects, etc.

[0181] 7) Preset intermediate results: The intermediate results of the image (or a local area of ​​the image) obtained by performing graphic calculations on the image (or a local area of ​​the image) in advance.

[0182] 8) Reconstruction intermediate results: Reconstruction data of intermediate results (e.g., rendering intermediate results) obtained by graphics calculation of the image to be processed.

[0183] 9) Current Frame: Also known as the "Image to be Processed". On the encoding side or server, the image to be processed is the image for which graphic calculations are to be performed. On the decoding side or client, the image to be processed is the image corresponding to the first intermediate result obtained by the server's graphic calculations (e.g., obtained from decoding the bitstream).

[0184] 10) Historical frames: On the encoding side, they represent the image of the first intermediate result that has been encoded; on the decoding side, they represent the image of the reconstructed intermediate result that has been decoded.

[0185] Standard video encoders and standard video decoders: These can be implemented according to industry video compression standards, such as ITU-T H.264, H.265 / HEVC, H.266 / VVC, AVS2, AVS3, or extensions of such standards. The solutions in this application are not limited to any specific encoding / decoding standard.

[0186] Figure 1a is a schematic diagram of the framework of an edge-cloud collaborative system provided in an embodiment of this application. The edge-cloud collaborative system may include: a central server, edge servers, and clients; wherein, a central server may connect to one or more edge servers, and an edge server may connect to one or more clients. This application does not limit the number of edge servers and clients; the specific number can be flexibly set according to the application scenario.

[0187] For example, a client can access the network through wireless access points such as base stations or Wi-Fi access points and communicate with the edge server through the network, or the client and the edge server can also communicate through a wired connection. Similarly, the edge server can also access the network through wireless access points such as base stations or Wi-Fi access points and communicate with the central server through the network, or the edge server and the central server can also communicate through a wired connection.

[0188] For example, the central server can be a single server, a server cluster consisting of multiple servers, or other distributed systems; this application does not impose any restrictions on this.

[0189] The client can be software, applications, browsers, in-vehicle systems, terminal devices, etc. When the client is implemented as a terminal device, it can include, but is not limited to, the following as shown in Figure 1a: mobile phone, personal computer (PC), virtual reality (VR) device, augmented reality (AR) device, tablet computer, laptop computer, etc. The client can also be a smart TV, mobile internet device (MID), wearable device (such as smartwatch, smart glasses, or smart helmet), smart car, wireless terminal device in industrial control, wireless terminal device in self-driving, wireless terminal device in remote medical surgery, wireless terminal device in smart grid, wireless terminal device in transportation safety, wireless terminal device in smart city, wireless terminal device in smart home, etc. The following embodiments do not impose special limitations on the specific form of the client.

[0190] In one possible scenario, some clients may connect directly to the central server instead of the edge server; in another possible scenario, all clients may connect directly to the central server instead of the edge server.

[0191] Furthermore, the edge-cloud collaborative system framework shown in Figure 1a is only an example of the edge-cloud collaborative system framework of this application. In the edge-cloud collaborative system of this application, the central server and the edge server can also be the same server; or the edge-cloud collaborative system of this application does not include edge servers, but the central server connects with each client. This application does not impose any restrictions on this.

[0192] As shown in Figure 1a, the edge-cloud collaborative system can be applied to various edge-cloud collaborative graphics computing scenarios, such as cloud gaming, cloud exhibitions, 3D cloud conferencing, 3D scenes, interior decoration, clothing design, architectural design, and other multi-end collaborative graphics computing scenarios. This application does not limit these applications.

[0193] The following uses a cloud gaming scenario as an example to illustrate the process of edge-cloud collaborative rendering of the game screen in the edge-cloud collaborative system shown in Figure 1a. In other embodiments, the graphics calculations performed by the edge-cloud collaborative system are not limited to the rendering process exemplified here, but can also be physical simulation or animation processing. The implementation principle of this method is similar to the rendering process, and will not be repeated here.

[0194] For example, in a cloud gaming scenario, a client can send a rendering request for an image to be processed to a central server (e.g., a cloud gaming server) via an edge server. This rendering request may include rendering parameters. The central server, in response to the rendering request, can obtain a first region in the image to be processed, which is a region related to scene changes. Next, the central server can perform rendering processing on the first region to obtain an intermediate rendering result for that region in the image. Then, the central server encodes the intermediate rendering result for the first region generated during the rendering process to obtain a bitstream. Finally, the central server distributes the bitstream to each client via an edge server. The client can then decode the received bitstream to obtain the intermediate rendering result for the first region of the image to be processed. Afterward, the client can perform further rendering processing on the scene corresponding to the image to be processed (e.g., a 3D scene) based on the intermediate rendering result for the first region, thereby obtaining the rendered result of the image to be processed (e.g., a game screen).

[0195] As an example, the rendering process of the first region in the image to be processed by the central server is as follows: The central server can load a 3D model (which may include a model of a person or an object) into a 3D scene, thus converting the 3D model into an object in the 3D scene; then, the central server can render the first region in the image to be processed (e.g., multiple 2D images converted from the 3D scene) corresponding to the 3D scene (including the aforementioned objects) to obtain an intermediate rendering result of the first region in the 3D scene. Here, the first region is the region related to changes in the 3D scene. For example, this first region is the area in the 3D scene where the aforementioned objects affect the lighting effects in the 3D scene (e.g., the lighting around a character will change).

[0196] The images to be processed can be multiple 2D images converted from a 3D scene. The 3D scene can be a virtual scene.

[0197] Furthermore, the first region in this application embodiment is a region related to changes in the scene (3D scene). The factors that cause changes in the scene may include, but are not limited to, at least one of the following: changes in the position of objects in the scene, changes in time, changes in viewpoint, etc.

[0198] For example, if the position of an object in a 3D scene changes, the scene corresponding to the image to be processed will also change. When a character in a cloud game enters a 3D scene, the lighting around the character needs to change. Therefore, the area around the character in the 3D scene can be an area related to the scene change.

[0199] For example, changes in time can cause changes in the scene corresponding to the image to be processed: for instance, the game time in a cloud game changes, and based on the game logic of the cloud game, some scenes in the game screen at the next moment are updated. The two-dimensional image area converted into the updated part of the scene is the area in the image to be processed that is related to the scene change.

[0200] For example, a change in perspective can cause a change in the scene corresponding to the image to be processed. For instance, in cloud gaming, a change in perspective can cause changes in objects and lighting in certain areas of the game scene (an example of a 3D scene). The area in the image to be processed that causes changes in objects and lighting in the game scene due to the change in perspective is the area related to the scene change.

[0201] It should be understood that the first region defined in this application is the region related to scene changes. Therefore, the scene changes are not limited to changes in lighting within the scene, but can also include changes in objects within the scene (such as characters, objects, etc.). This is not a limitation.

[0202] The following example illustrates the edge-cloud collaborative rendering process using Dynamic Diffuse Global Illumination (DDGI) rendering. DDGI is a probe-based scene diffuse rendering method, as shown in Figure 1b. To simulate the lighting effects of diffuse reflection in a 3D scene, the central server can place multiple probes within the 3D scene during rendering to detect the surrounding environment and obtain probe data (e.g., probe positions). Then, the central server renders the 3D scene (e.g., object positions, shapes, and other scene description information) based on the probe data. Specifically, in this embodiment, to reduce the rendering computation load on the central server, it can render only the first region based on the probe data within that region of the 3D scene, thus obtaining an intermediate rendering result for the first region in the image to be processed. The central server then encodes the intermediate rendering result for the first region and transmits the resulting bitstream to the client via the network. The client only needs to perform simple rendering processing using the intermediate rendering data for the first region to obtain a high-quality diffuse global illumination effect.

[0203] In this embodiment, since the central server only needs to perform rendering processing on the first region of the image to be processed that is related to scene changes to obtain the intermediate rendering result corresponding to the first region, without needing to perform rendering processing on the entire image to be processed, the computational load and overhead of the central server on the intermediate rendering result are reduced. Furthermore, the bitstream sent by the central server to the client is the bitstream obtained by encoding the intermediate rendering result of the first region, rather than the bitstream obtained by encoding the intermediate rendering result of the entire image to be processed, thereby reducing the amount of encoded data on the central server side and reducing the bandwidth occupation of the transmitted bitstream.

[0204] Based on the application scenario shown in Figure 1a, this application proposes an end-to-cloud collaborative system, as shown in Figure 1c.

[0205] Figure 1c is a schematic diagram of an end-to-cloud collaborative system according to an exemplary embodiment of this application. As shown in Figure 1c, the end-to-cloud collaborative system may include a server 100 and a client 200.

[0206] Referring to Figure 1c, the functions of server 100 can be implemented through software or hardware.

[0207] In the first example, when the functionality of server 100 is implemented through software, server 100 may be, for example, an application running on a computing instance, such as a virtual machine, container, or host.

[0208] In the second example, when the functionality of server 100 is implemented through hardware, server 100 may be implemented through at least one physical device including a processor, such as a server.

[0209] The processor can be a central processing unit (CPU) or a graphics processing unit (GPU), or it can be any type of processor or any combination thereof, such as an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a system-on-chip (SoC), a software-defined infrastructure (SDI) chip, an artificial intelligence (AI) chip, or a data processing unit (DPU).

[0210] Furthermore, the number of processors included in the server 100 can be arbitrary, and the types of processors included can be one or more. The specific number and types of processors can be set according to the actual business needs of the application, and this application does not limit this.

[0211] The physical device can be a server (such as the central server or edge server in Figure 1a), a base station, a relay device, a satellite device, etc., and there are no restrictions here.

[0212] In the third example, when the functionality of server 100 is implemented through hardware, server 100 can also be a computing cluster comprising multiple computing nodes. Further, these multiple computing nodes can communicate through the at least one switching node. Exemplarily, a computing node can be a computing server including an accelerator card. This accelerator card can be, for example, a deep-learning processing unit (DPU), a data processing unit (DPU), a graphics processing unit (GPU), a neural-network processing unit (NPU), or a tensor processing unit (TPU), or other types of accelerator cards. Alternatively, a computing node can be a computing server including a general-purpose processor (such as a central processing unit (CPU)).

[0213] As shown by the dashed arrow in Figure 1c, the server 100 and the client 200 can communicate with each other (e.g., through a direct network connection or through an indirect connection via a server, communication device, etc., which is not limited here). Alternatively, in other embodiments, the server 100 and the client 200 may not be connected.

[0214] For example, the server 100 may include a first graphics calculation module 101 and an encoding module 102, and optionally may also include a first communication module 103.

[0215] The first graphics calculation module 101 can be used to obtain a first region in the image to be processed and perform graphics calculations (e.g., first graphics calculations) on the first region to obtain a first intermediate result of the first region of the image to be processed, wherein the first region is a region related to scene changes.

[0216] Graphics computing can include, but is not limited to, at least one of the following: rendering, physical simulation, or animation processing.

[0217] For example, when the graphic is calculated for rendering, the aforementioned first intermediate result can be called a "rendering intermediate result".

[0218] For example, when the graph is calculated as a physical simulation, the aforementioned first intermediate result can be called the "physical simulation intermediate result".

[0219] For example, when the graphic is processed as an animation, the first intermediate result mentioned above can be called the "animation processing intermediate result".

[0220] For example, graph computing may include two stages: a first graph computing processing stage and a second graph computing processing stage. The first graph computing requires higher computing power, while the second graph computing requires lower computing power, and the object processed by the second graph computing is the result obtained from the first graph computing (i.e., an intermediate result).

[0221] Taking graphics computing as a rendering process as an example, the server 100 (which has higher computing power) can perform high-power rendering processing on the first region of the image to be processed (an example of first graphics computing) to obtain the intermediate rendering result of the first region (an example of the first intermediate result); the client (such as the client 200 described below, which has lower computing power) can perform further rendering processing on the above intermediate rendering result (an example of second graphics computing) to obtain the rendering result of the first region (e.g., a game screen).

[0222] The encoding module 102 can be used to encode the first intermediate result to obtain an encoded bitstream.

[0223] The first communication module 103 can be used to communicate with other devices.

[0224] The encoding module 102 can be used to output the encoded bitstream to a local storage device or storage medium, or to a remote device or server (such as a video website), or to output the bitstream to the client 200 through the first communication module 103. There are no restrictions on this.

[0225] Referring to Figure 1c, the client 200 may include, but is not limited to: a decoding module 202, a second graphics calculation module 201, and optionally a second communication module 203 and a display module 204.

[0226] The second communication module 203 can be used to communicate with other devices, for example, to receive the code stream sent by the server 100.

[0227] The decoding module 202 can be used to decode the input bitstream to obtain the decoded data corresponding to the first intermediate result. The decoding module 202 can read the bitstream from a local storage device or storage medium, or receive the bitstream from a remote device or server (such as a video website), or receive the input bitstream through the second communication module 203 mentioned above, and then decode the acquired bitstream.

[0228] The second graphics calculation module 201 can be used to obtain the reconstruction data of the intermediate result of the image to be processed based on the decoding data corresponding to the first intermediate result, and to perform further graphics calculations on the reconstruction data of the intermediate result of the image to be processed to obtain the graphics calculation result of the image to be processed.

[0229] Display module 204 can be used to display the results of graphics calculations. For example, the results of graphics calculations are displayable images.

[0230] For example, when the graphics calculation performed in the first graphics calculation module 101 and the second graphics calculation module 201 is a rendering process, the graphics calculation result can be the rendering result of the image to be processed. For instance, the rendering process is based on scene description information of the three-dimensional scene (such as the position and shape of objects) to render the three-dimensional scene using a rendering algorithm, so as to obtain the rendering result of the image to be processed corresponding to the three-dimensional scene. This rendering result is the rendered image of the image to be processed.

[0231] For example, when the graphics calculations performed in the first graphics calculation module 101 and the second graphics calculation module 201 are physical simulations or animation processing, the resulting graphics calculation results are usually scene description information of a three-dimensional scene.

[0232] For example, the process of physically simulating water droplets is as follows: The second graphics calculation module 201 can abstract water droplets into particles, calculate the interactions between particles through physical operations, and finally obtain scene description information such as the position and deformation of each water droplet after a physical simulation. Then, the second graphics calculation module 201 can be used to further render the position, deformation and other information of each water droplet obtained from the physical simulation to obtain the image formed by each water droplet in the three-dimensional scene.

[0233] For example, the process of using skeletons to record character animation (skeletal animation) is as follows: The second graphics calculation module 201 can perform animation processing on the character's skeleton to obtain graphics calculation results, specifically scene description information such as the position and rotation of the character's skeleton. Then, the second graphics calculation module 201 can be used to further render the scene description information such as the position and rotation of the character's skeleton obtained from the animation processing to obtain the image formed by the character animation in a three-dimensional scene.

[0234] As an example, the encoding module 102 can be implemented as an encoder, and the decoding module 202 can be implemented as a decoder.

[0235] It should be understood that the embodiments of this application do not limit the video coding standards used by the encoding module 102 and the decoding module 202. For example, they may include, but are not limited to: H.264 / Advanced Video Coding (AVC), H.265 / High Efficiency Video Coding (HEVC), H.266 / Versatile Video Coding (VVC), AV1 (AOMedia Video 1, "AOMedia" refers to a video coding standard developed by the Open Media Video Consortium), and extended standards of these video coding standards; in addition, they may also include new video coding standards and extended standards that have emerged with the development of video coding and decoding technologies.

[0236] It should be understood that Figure 1c is only an example of an embodiment of this application. The server 100 and client 200 of this application embodiment may include more or fewer modules than shown in Figure 1c, and the modules in the server 100 may be split or merged, and the modules in the client 200 may be split or merged. This application does not make specific limitations in this regard.

[0237] The image processing procedure of this application will be described below using Figures 1a and 1c as examples.

[0238] Figure 2a is an exemplary schematic diagram of an image processing process provided in an embodiment of this application. In the embodiment of Figure 2a, the rendering process is taken as an example of the graphics calculation performed by the server 100 on the image to be processed, and the intermediate rendering result of a frame of rendered image to be processed is used as an example for illustration. In other embodiments, when the graphics calculation performed by the server 100 on the image to be processed is physical simulation or animation processing, the implementation principle of its image processing process is similar, and will not be repeated here.

[0239] S201, Server 100 obtains the first region in the image to be processed.

[0240] For example, the first region is the area related to scene changes. This scene is the 3D scene corresponding to the image to be processed. The 3D scene can be converted into one or more 2D images, which are called the images to be processed.

[0241] The server 100 can obtain information indicating the first region from the client 200, thereby obtaining the first region. Alternatively, it can determine the first region in the image to be processed by processing the rendering request of the client 200. No limitation is imposed here.

[0242] S202, Server 100 renders the first region in the image to be processed to obtain the first intermediate rendering result of the image to be processed.

[0243] The first intermediate rendering result is the intermediate rendering result of the first region in the image to be processed obtained by the server 100 after rendering the first region.

[0244] The rendering parameters used in the rendering process can include rendering parameters obtained from the client 200, rendering parameters stored locally on the server 100, or pre-set rendering parameters, etc., without any restrictions.

[0245] For example, server 100 can obtain rendering parameters; then, the rendering parameters can be input to the graphics rendering engine, which will then render the portion of the 3D scene corresponding to the first region based on the rendering parameters, to obtain the first intermediate rendering result of the first region. For example, the graphics rendering engine can belong to the first graphics calculation module 101 shown in Figure 1c.

[0246] In some embodiments, the server 100 may obtain rendering parameters from the client 200 at a preset period, wherein the preset period may be set as needed, and this application does not specifically limit it. In some embodiments, the rendering parameters obtained by the server 100 from the client 200 each time may be used for rendering one or more frames of the image to be processed.

[0247] For example, the rendering process of a graphics rendering engine can be as follows: Based on the input rendering parameters, the graphics rendering engine can perform geometric transformations, projection transformations, perspective transformations, and window clipping on the 3D object model corresponding to the first region in the 3D scene, and then generate the first intermediate rendering result of the first region through the acquired material and lighting information.

[0248] S203, Server 100 encodes the first intermediate rendering result to obtain the bitstream.

[0249] As an example, S201 to S202 can be executed by the first graphics calculation module 101 in Figure 1c, and S203 can be executed by the encoding module 102 shown in Figure 1c.

[0250] Figure 2b is an exemplary schematic diagram of the encoding framework of an embodiment of this application. The embodiment of Figure 2b illustrates the encoding framework of the encoding module 102 (here, the encoder) in Figure 1c. The encoder shown in Figure 2b can encode a first intermediate result of the image to be processed, which can be each frame of the video to be encoded, without limitation. The first intermediate result can be, for example, the first intermediate rendering result shown in Figure 2a.

[0251] Referring to Figure 2b, an encoder may, by way of example, include: a block module, an intra-frame prediction module, an inter-frame prediction module, a transform module, a quantization module, an entropy coding module, an inverse quantization module, an inverse transform module, a loop filtering module, and a memory.

[0252] For example, the encoding process of the encoder can be as follows: After the block segmentation module of the encoder obtains the first intermediate result of the image to be processed (e.g., the first intermediate rendering result shown in Figure 2a), it can segment the first intermediate result into blocks to obtain blocks to be encoded. For each block to be encoded, the decision module (not shown in Figure 2b) can determine whether to perform inter-frame prediction or intra-frame prediction for the block to be encoded.

[0253] When intra-frame prediction is determined, the intra-frame prediction module performs intra-frame prediction to determine the prediction block corresponding to the block to be coded. After obtaining the prediction block, the residual block between the block to be coded and the prediction block can be determined. Then, the residual block is input to the transform module, which transforms the residual block, obtains the transform result, and outputs the transform result to the quantization module. Subsequently, the quantization module quantizes the transform result, obtains the quantization result, and outputs the quantization result to the entropy coding module. Then, the entropy coding module entropy codes the quantization result to obtain the encoded data of the residual block, and writes the encoded data of the residual block into the bitstream.

[0254] For example, the quantization module can also output the quantization result to the inverse quantization module, which performs inverse quantization to obtain the inverse quantization result and outputs it to the inverse transform module. Next, the inverse transform module performs an inverse transform on the inverse quantization result to obtain a decoded residual block. Then, the decoded residual block can be superimposed with the prediction block output by the intra-frame prediction module to obtain a reconstructed block. Subsequently, the reconstructed block is input to the loop filtering module, which performs loop filtering on the reconstructed block to obtain a filtered reconstructed block and outputs it to memory. For example, after obtaining the filtered reconstructed block, the loop filtering module can stitch the filtered reconstructed block to the corresponding position of the first intermediate result to be reconstructed; after obtaining the reconstructed data of the first intermediate result, it can output the reconstructed data of the first intermediate result of the first region to memory for storage. Alternatively, for example, after obtaining the filtered reconstruction block, the loop filtering module can stitch the filtered reconstruction block to the corresponding position of the intermediate reconstruction result to be reconstructed (e.g., the reconstruction data of the intermediate result of the complete region of the image to be processed); after obtaining the intermediate reconstruction result, the intermediate reconstruction result can be output to the memory for storage.

[0255] When it is determined to be inter-frame prediction, the inter-frame prediction module can perform inter-frame prediction to determine the prediction block corresponding to the block to be encoded.

[0256] In one embodiment, the inter-frame prediction module can select an intermediate result (e.g., an intermediate result of a local region (e.g., a first region) or an intermediate result of a complete region of the reference image) corresponding to the image to be processed from a list of reference frames in the memory as a reference frame for the image to be processed. For the block to be encoded in the first intermediate result of the image to be processed, a matching prediction block can be found in the reference frame. The reference image can be any frame whose display time or encoding time is before the image to be processed, such as the previous frame whose display time is before the image to be processed. The reference frame described in the various embodiments of this application does not refer to the image referenced when encoding the first intermediate result of the image to be processed (i.e., the reference image), but rather to the intermediate result obtained by graphics calculation from the reference image (e.g., the rendering intermediate result obtained by rendering the first region or complete region of the reference image).

[0257] In one embodiment, the inter-frame prediction module can select original reference frames from the reference frame list in the memory as candidate reference frames for the first intermediate result of the image to be processed; where the original reference frame refers to the intermediate result of the reference image corresponding to the image to be processed. For the block to be encoded in the first intermediate result, inter-frame prediction can be performed based on multiple candidate reference frames to determine multiple prediction blocks; where one candidate reference frame corresponds to one prediction block; then, an optimal prediction block can be selected from the multiple prediction blocks (for example, the prediction block with the lowest rate-distortion cost can be determined as the optimal prediction block using rate-distortion cost as the evaluation criterion). After obtaining the prediction blocks, the residual blocks between the block to be encoded and the prediction blocks can be determined; then, the residual blocks are input to the transform module, which transforms the residual blocks to obtain the transform result and outputs the transform result to the quantization module. Subsequently, the quantization module can quantize the transform result to obtain the quantization result and output the quantization result to the entropy coding module; then, the entropy coding module can entropy code the quantization result to obtain the encoded data of the residual blocks and write the encoded data of the residual blocks into the bitstream (that is, the bitstream obtained in S203 above).

[0258] For example, the quantization module can also output the quantization result to the inverse quantization module, which performs inverse quantization to obtain the inverse quantization result and outputs it to the inverse transform module. Next, the inverse transform module performs an inverse transform on the inverse quantization result to obtain a decoded residual block. Then, the decoded residual block can be superimposed with the prediction block output by the intra-frame prediction module to obtain a reconstructed block. Subsequently, the reconstructed block is input to the loop filtering module, which performs loop filtering on the reconstructed block to obtain a filtered reconstructed block and outputs it to memory. For example, after obtaining the filtered reconstructed block, the loop filtering module can stitch the filtered reconstructed block to the corresponding position of the first intermediate result to be reconstructed; after obtaining the reconstructed data of the first intermediate result, it can output the reconstructed data of the first intermediate result of the first region to memory for storage. Alternatively, for example, after obtaining the filtered reconstruction block, the loop filtering module can stitch the filtered reconstruction block to the corresponding position of the intermediate reconstruction result to be reconstructed (e.g., the reconstruction data of the intermediate result of the complete region of the image to be processed); after obtaining the intermediate reconstruction result, the intermediate reconstruction result can be output to the memory for storage.

[0259] Figure 3a is a schematic diagram illustrating an image processing process according to an exemplary embodiment of this application. Figure 3a shows the image processing process corresponding to Figure 2a. In the embodiment of Figure 3a, decoding a frame of an image to be processed is used as an example for explanation. The client 200 can perform the following steps:

[0260] S301, Client 200 receives the bitstream.

[0261] For example, after the server 100 sends the bitstream, the client 200 can receive the bitstream.

[0262] S302, Client 200 decodes the bitstream to obtain the decoded data corresponding to the first intermediate rendering result of the image to be processed.

[0263] The first intermediate rendering result is obtained by rendering the first region in the image to be processed, which is the region related to scene changes.

[0264] Then, the client 200 can obtain the rendering result of the image to be processed based on the decoded data corresponding to the first intermediate rendering result.

[0265] As an example, as shown in Figure 3a, client 200 can be implemented via S303 and S304.

[0266] Optionally, in S303, the client 200 decodes the image to be processed based on the decoding data corresponding to the first rendering intermediate result to obtain the reconstruction rendering intermediate result (specifically, the reconstruction data of the rendering intermediate result).

[0267] For example, client 200 can use the decoding data corresponding to the first rendering intermediate result of a local region (here, the first region) in the image to be processed to reconstruct the reconstruction data of the rendering intermediate result of the entire image to be processed (i.e., the complete region of the image to be processed).

[0268] S304, Client 200 performs further rendering processing on the intermediate results of the reconstruction rendering to obtain the rendering result of the image to be processed.

[0269] For example, client 200 can perform further rendering processing on the intermediate results of the reconstruction rendering of the image to be processed based on locally generated rendering parameters, so as to obtain the rendering result of the image to be processed (such as game screen).

[0270] Figure 3b is a schematic diagram of an exemplary decoding framework. The embodiment of Figure 3b illustrates the decoding framework of the decoding module 202 (described herein as a decoder) shown in Figure 1c.

[0271] Referring to Figure 3b, for example, the decoder may include: an entropy decoding module, an inverse quantization module, an inverse transform module, an intra-frame prediction module, an inter-frame prediction module, a loop filtering module, and a memory.

[0272] For example, after receiving the bitstream, the client 200 can decode the bitstream to obtain the decoded data corresponding to the first intermediate result of the image to be processed. The image processing procedure can be as follows: The decoder can parse the encoded data of the residual block from the bitstream, and then input the encoded data of the residual block into the entropy decoding module. The entropy decoding module performs entropy decoding on the encoded data of the residual block to obtain entropy-decoded data. Next, the entropy-decoded data can be input into the dequantization module, which performs dequantization to obtain dequantized data, and then inputs the dequantized data into the inverse transform module. Subsequently, the inverse transform module performs an inverse transform on the dequantized data to obtain the decoded residual block.

[0273] It should be understood that when the encoder performs lossy encoding of the residual block, the residual block decoded by the decoder will differ from the residual block encoded at the encoder. When the encoder performs lossless encoding of the residual block, the residual block decoded by the decoder will be the same as the residual block encoded by the encoder.

[0274] It should be noted that bitstream description information can also be parsed from the bitstream to describe the bitstream. In this case, there is no need to perform entropy decoding, inverse quantization, and inverse transform operations on the bitstream description information. This bitstream description information can be used in the subsequent decoding process.

[0275] For example, when it is determined to be intra-frame prediction based on the bitstream description information parsed from the bitstream, the intra-frame prediction module can perform intra-frame prediction to obtain a prediction block. Then, a reconstructed block can be obtained based on the decoded residual block and the prediction block. Next, a loop filter can be applied to the reconstructed block to obtain a filtered reconstructed block, which is then output to a memory. Specifically, after obtaining the filtered reconstructed block, the loop filter module can concatenate the filtered reconstructed block to the corresponding position in the first intermediate result to be reconstructed (e.g., the first rendering intermediate result shown in Figure 3a); after obtaining the reconstructed data of the first intermediate result, the reconstructed data of the first intermediate result can be output to a memory for storage.

[0276] For example, when it is determined to be inter-frame prediction based on the bitstream description information parsed from the bitstream, the inter-frame prediction module can perform inter-frame prediction to obtain the prediction block.

[0277] In one embodiment, when the inter-frame prediction module determines that the reference frame is the original reference frame based on the bitstream description information decoded from the bitstream, the inter-frame prediction module can select the corresponding original reference frame from the reference frame list in the memory as the reference frame; then, it can determine the prediction block based on the original reference frame. Then, a reconstructed block is obtained based on the decoded residual block and the prediction block. Then, a loop filter can be performed on the reconstructed block to obtain a filtered reconstructed block, and the filtered reconstructed block is output to the memory. For example, after obtaining the filtered reconstructed block, the loop filter module can output the reconstructed data of the first intermediate result of the first region to the memory for storage. Alternatively, for example, after obtaining the filtered reconstructed block, the loop filter module can stitch the filtered reconstructed block to the corresponding position of the intermediate reconstructed result to be reconstructed (e.g., the reconstructed data of the intermediate result of the complete region of the image to be processed); after obtaining the intermediate reconstructed result, the intermediate reconstructed result can be output to the memory for storage.

[0278] In S303 and S304 above, the client 200 can obtain the reconstruction data of the intermediate result of the image to be processed (i.e., the reconstruction rendering intermediate result) based on the decoding data corresponding to the first intermediate result of the image to be processed, such as the rendering intermediate result of the complete area of ​​the image to be processed; then, the client 200 can perform further rendering processing on the reconstruction data of the intermediate result through the graphics rendering engine to obtain the rendered image of the image to be processed (the rendering result of the image to be processed here). In this way, the rendering results of multiple frames of the image to be processed can be combined into a video.

[0279] It should be noted that S301 to S302 can be executed by the decoding module 202 shown in Figure 1c, and S303 to S304 can be executed by the second graphics calculation module 201 of the client 200. In some embodiments, S303 can also be executed by the decoding module 202 shown in Figure 1c, and there is no limitation here.

[0280] Figure 4 is a flowchart of process 400 of the image processing method provided in an embodiment of this application. Process 400 can be executed by the server 100 described above. Process 400 is described as a series of steps or operations. It should be understood that process 400 can be executed in various orders and / or occur simultaneously, and is not limited to the execution order shown in Figure 4. Process 400 may include:

[0281] S401a, the server 100 obtains a first region in the image to be processed, the first region being a region related to scene changes.

[0282] In the application of the image processing method of this application to perform graphics calculations on a three-dimensional scene, the image to be processed can be multiple two-dimensional images converted from the three-dimensional scene. For example, a three-dimensional model can be converted into multiple two-dimensional textures (such as position maps, material maps, etc.).

[0283] The following uses a 3D scene as an example to illustrate that in a cloud gaming scenario, the 3D scene corresponding to the image to be processed can be a virtual scene. The rendered image (e.g., lighting effects) of one or more areas in the virtual scene will change as the virtual scene changes, while the rendered image (e.g., lighting effects) of other areas in the virtual scene will not change as the virtual scene changes.

[0284] Therefore, the method of this application embodiment aims to obtain the region in the virtual scene whose rendering effect (e.g., lighting effect) changes with the changes in the virtual scene. The region in the virtual scene obtained here can be mapped to the first region in the image to be processed. Based on this, this step can obtain the first region in the image to be processed whose rendering effect changes with the changes in the virtual scene.

[0285] Taking lighting effects as an example of rendering, as an implementation example, factors that can cause changes in the virtual scene can be changes in the lighting effects of a certain area in the virtual scene caused by changes in game time. For example, the position of a character in the virtual scene does not change between two frames (e.g., the previous frame and the current frame (the image to be processed in step 401a)). However, as time changes, due to game logic, the lighting effects of a certain area in the current frame (e.g., the area around the character, but not limited to the area around the character) will change between two frames (e.g., the area around the character changes from having little shadow to having a lot of shadow). Then, the aforementioned area in the current frame is the first area.

[0286] As another implementation example, factors that can cause changes in the virtual scene can also be changes in the lighting effects of a certain area in the virtual scene caused by changes in the position of a character. For example, if the position of a character in the virtual scene changes between two frames (e.g., the previous frame and the current frame (the image to be processed in step 401a)) (e.g., the character is displaced), then the lighting effects of the area around the character will change between the two frames (e.g., the area around the character changes from having very little shadow to having a lot of shadow), and then the area around the character in the current frame is the first area.

[0287] As another implementation example, factors that can cause changes in the virtual scene can also be changes in the lighting effects of a certain area in the virtual scene caused by changes in the game's perspective. For example, although the position of a character in the virtual scene does not change between two frames (e.g., the previous frame and the current frame (the image to be processed in step 401a)), the game's perspective changes between the two frames, resulting in the lighting effects of one or more areas (e.g., the area around the character) in the current frame after switching the game perspective being different from the lighting effects of the same one or more areas (e.g., the area around the character) in the previous frame. In this case, the aforementioned one or more areas (e.g., the area around the character) in the current frame is the first area.

[0288] The above three implementation examples use the area where the lighting effect changes in the virtual scene as the first area as an example. In other embodiments, the first area can also be an area in the virtual scene where other rendering effects change. For example, if an object is added in the current frame compared to the previous frame, then the area around the object can be the first area.

[0289] In one example, the image to be processed can be the image frame that the server 100 is currently processing. For example, the server 100 can process each image frame in sequence according to the display order of the image sequence in the game. In this embodiment and subsequent embodiments, "current frame" refers to the image to be processed by the server 100.

[0290] In another example, the image to be processed can be a local image of a frame (e.g., the current frame), such as an image patch, a region of interest (ROI), etc. This application does not impose specific limitations on the size and shape of the local image.

[0291] In S401a above, the server 100 can determine the first region in the image to be processed on its own, or it can obtain the information of the first region in the image to be processed from the client 200 or other devices. There are no restrictions here.

[0292] S402, the server 100 performs graphic calculations on the first region to obtain the first intermediate result of the image to be processed.

[0293] Graphics computation can be flexibly configured according to application requirements. In one example, graphics computation can be rendering processing. After the server 100 performs rendering processing on the first region of the image to be processed, it can obtain the first intermediate rendering result of the image to be processed (an example of the first intermediate result).

[0294] In other examples, graphics computation can also be physical simulation or animation processing, in which case the first intermediate result in step 402 is either the first physical simulation intermediate result or the first animation intermediate result from server 100. The first physical simulation intermediate result is the intermediate data generated by server 100 during the process of generating the physical simulation result after performing physical simulation processing on the first region. Similarly, the first animation intermediate result is the intermediate data generated by server 100 during the process of generating the animation result after performing animation processing on the first region.

[0295] Taking graphics computing as an example of rendering, the first intermediate result (e.g., the first rendering intermediate result) in various embodiments of this application can refer to the intermediate data generated by the graphics rendering engine of server 100 during the process of generating the image to be displayed in the first region (i.e., the rendered image). The graphics rendering engine can perform rendering processing on the region in the three-dimensional scene corresponding to the first region to obtain the first rendering intermediate result.

[0296] For example, intermediate rendering results may include, but are not limited to, one or more of the following: Computer Graphics Motion Vector (CGMV), intermediate rendering images, position map, normal map, albedo map, specular intensity map, depth map, ambient occlusion map, mesh ID, material ID (where each material map corresponds to a material ID), render ID (each object (or a 3D object model) corresponds to a render ID), depth information, etc. This application embodiment does not specifically limit these.

[0297] Intermediate rendering results can refer to intermediate rendering images generated before the final rendered image is generated. The computational complexity of generating intermediate rendering images is lower than that of generating the final rendered image. For example, intermediate rendering images can be intermediate rendering images without indirect lighting rendering, intermediate rendering images without specular reflection processing, intermediate rendering images without specular highlight processing, etc.

[0298] S403a, the server 100 encodes the first intermediate result to obtain a bitstream.

[0299] The server 100 can encode the first intermediate result to obtain a bitstream using a standard video encoder. This standard video encoder can be implemented according to industry video compression standards, such as ITU-T H.264, H.265 / HEVC, H.266 / VVC, AVS2, AVS3 standards, or extensions of such standards. The solution in this application is not limited to any specific encoding standard.

[0300] Taking Figure 5a as an example, Figure 5a exemplarily shows a schematic diagram of the image to be processed 300. The three-dimensional scene corresponding to the image to be processed 300 includes multiple probes. As shown in Figure 5a, when character 10 enters the three-dimensional scene (e.g., a game scene) in the current frame (i.e., the image to be processed 300), character 10 can cause changes in the lighting around character 10 in the image to be processed (e.g., the first region 11). Therefore, the server 100 needs to update the lighting of the first region 11 around character 10. However, for regions in the game scene that are far from character 10, such as the second region 12, the lighting changes are usually not affected by character 10, so that the lighting change pattern of the second region 12 follows the original change pattern (specifically, a preset pattern that is unrelated to scene changes, such as the sunlight change pattern).

[0301] Based on this, the server 100 in this embodiment can calculate the lighting characteristics (e.g., at least one lighting feature such as DDGI, direct lighting, reflected lighting, shadows, indirect lighting, spherical reflection, ambient occlusion, etc.) of the first region 11 to obtain a first intermediate result (specifically, a rendering intermediate result) of the first region 11 in the current frame. The rendering effect of the lighting characteristics of the second region 12 is not affected by the position of the character 10, allowing the rendering effect of the lighting characteristics of the second region 12 to follow its original variation pattern (e.g., sunlight pattern). Therefore, the server 100 does not need to calculate the lighting characteristics of the second region 12, but instead reuses the rendering intermediate result of the lighting characteristics of the second region 12 pre-calculated during the game's production phase.

[0302] In this embodiment, the server 100 only needs to perform graphics calculations on the region of the image to be processed that is related to scene changes (which is a local region in the image to be processed) to obtain the first intermediate result (e.g., the first rendering intermediate result) of the first region in the image to be processed, without having to perform graphics calculations on the entire region of the image to be processed. This significantly reduces the amount of computation and computational cost of the server 100 performing graphics calculations on the image to be processed to obtain the intermediate result of the image to be processed. In addition, during encoding, the server 100 only needs to encode the first intermediate result of the local region (e.g., the first region), thereby reducing the amount of encoding and shortening the length of the bitstream, thereby reducing the proportion of the bitstream to the transmission bandwidth.

[0303] As shown in the dashed boxes and dashed arrows in Figure 4, the server 100 may also optionally perform some steps to achieve the corresponding functions.

[0304] As shown in Figure 4, in one embodiment, the process 400 may further include S401b and S403b.

[0305] S401b, server 100 obtains the first information.

[0306] S403b, the server 100 encodes the first information into the bitstream.

[0307] The first information is used to indicate the position of the first region in the image to be processed in S401a.

[0308] The first information may be determined by the server 100 based on the position of the first region in the image to be processed after acquiring the first region. Alternatively, the first information may also be received by the server 100 from the client 200, and there is no restriction here. Therefore, this application does not restrict the execution order between the processes S401b and S403b and the processes S401a, S402, and S403a described above.

[0309] In this way, the bitstream not only includes the first intermediate result of the first region, but also the first information indicating the position of the first region in the image to be processed. Thus, the server 100 not only encodes the first intermediate result of the first region in the image to be processed after graphic calculation, but also encodes the information of the position of the first intermediate result in the image to be processed. This allows the decoding end (e.g., the client 200 mentioned above) to use the decoded bitstream to reconstruct the reconstruction intermediate result of the image to be processed (referring to the intermediate result of graphic calculation of the complete region of the image to be processed, such as the rendering intermediate result). This reduces the amount of computation on the end side (e.g., the client 200) when calculating which region in the image to be processed corresponds to the first intermediate result.

[0310] In one embodiment, when the server 100 executes the above S401a, it can obtain the second information and then obtain the first region in the image to be processed based on the second information, wherein the second information is used to divide the image to be processed into regions.

[0311] The server 100 can generate second information, receive second information (e.g., receive it from the client 200), and process some information sent by the client 200 (the second information mentioned in the process 600 executed by the client 200 below) to obtain second information. No restrictions are imposed here.

[0312] In this embodiment of the application, the server 100 can obtain second information for dividing the image to be processed into regions, thereby dividing the image to be processed into regions to obtain a first region in the image to be processed that is related to scene changes, thus ensuring the accuracy of the first region.

[0313] In one embodiment, the second information obtained in S401a may include at least one of the following: scene information corresponding to the image to be processed; one or more of the following: network status information, computing power, power consumption, and heat threshold of the server 100 and / or client 200; and communication link information of the communication link between the server 100 and the client 200, wherein the server 100 and the client 200 communicate.

[0314] In some embodiments, the scene information corresponding to the image to be processed may include, but is not limited to, at least one of the following: game state information, probe information (e.g., probe position), distance threshold, etc., wherein the game state information may include, but is not limited to, at least one of the following: game time information, character position information, viewpoint information, etc.

[0315] For example, server 100 can determine a first region in the image to be processed that is related to scene changes based on one or more of the distance threshold, the character's position, and the position of probes in the 3D scene corresponding to the image to be processed. For example, probes in the 3D scene whose distance from the character's position is less than or equal to the distance threshold can be identified, and the first region in the image to be processed can be determined based on the identified positions of these probes in the 3D scene. In this way, the method of this embodiment can change the rendering effect of the lighting in the area around the character in the image to be processed.

[0316] For example, changes in game time can alter game logic, leading to changes in the graphics calculation effects (e.g., rendering effects) of certain areas in the game's 3D scene. Server 100 can then determine the game logic based on the game time information, and subsequently identify areas in the image to be processed that can cause changes in lighting effects. For instance, if the lights in the 3D scene corresponding to the game logic at time t1 are off, and the lights in the 3D scene corresponding to the game logic at the next time t2 are illuminating a certain area, then server 100 can determine, based on time t2, the first area in the image to be processed that corresponds to the illuminated area in the 3D scene.

[0317] For example, changes in perspective information can alter game logic, leading to changes in the graphics calculation effects (e.g., rendering effects) of certain areas in the game's 3D scene. Therefore, server 100 can also determine, based on the perspective information, the regions within the image to be processed that can cause changes in lighting effects. For instance, if the game user switches their perspective from perspective 1 to perspective 2, where perspective 1 illuminates region 1 in the 3D scene and perspective 2 illuminates region 2, then after switching perspectives, server 100 can determine, based on the changed perspective, the first region in the image to be processed corresponding to region 2 (e.g., region 2 converted to a 2D region).

[0318] In some embodiments, the network status information of the server 100 or the client 200 is used to indicate the network status of the device to which the server 100 or the client 200 belongs, such as at least one of bandwidth, latency, packet loss rate, signal strength, and network congestion level.

[0319] Because server 100 needs to send the encoded bitstream of the first intermediate result of the first region to client 200, in order to match this bitstream with the network status information of server 100 or client 200, and to avoid situations where the data volume of the bitstream sent by server 100 to client 200 is large and causes the client 200 to experience buffering issues due to poor network conditions, server 100 determines a relatively large first region when its network condition is good, and a relatively small first region when the client 200's network condition is poor. This ensures that the bitstream is adapted to the network conditions of both server 100 and client 200.

[0320] A good network condition can be characterized by high bandwidth, low latency, low packet loss rate, strong signal strength, and low network congestion. A poor network condition is the opposite of a good network condition, which will not be elaborated here.

[0321] In this way, the first region, which is divided based on the network status information of the edge or the cloud, can be adapted to the network status of either the edge or the cloud.

[0322] In some embodiments, the computing power of server 100 or client 200 is used to indicate the computing power of the device to which server 100 or client 200 belongs. For example, it indicates the amount of data that the device can process per unit time. Indicators affecting the computing power of a device may include, but are not limited to: processor performance, memory capacity and speed, storage speed (e.g., hard disk read / write speed, input / output throughput, etc.).

[0323] For example, if the server 100 has strong computing power or the client 200 has weak computing power, it can determine a relatively large first region, thereby enabling the server 100 to perform graphics calculations on the larger first region, thus reducing the graphics calculation pressure on the client 200.

[0324] For example, when the server 100 has weak computing power or the client 200 has strong computing power, it can determine a relatively small first region, thereby enabling the client 200 to handle the graphic calculations of the region outside the first region in the image to be processed.

[0325] In some embodiments, the power information of the server 100 or the power information of the client 200 is used to indicate the power information of the device to which the server 100 or the client 200 belongs, such as at least one of the following: remaining power, power consumption, charging status (whether it is charging, or the percentage of charging, etc.).

[0326] For example, server 100 can determine a larger first region when server 100 has sufficient power or client 200 has insufficient power, and vice versa.

[0327] In some embodiments, the thermal threshold information of server 100 or client 200 is used to indicate the thermal threshold information of the device to which server 100 or client 200 belongs. The thermal threshold refers to the temperature range of the device under normal operating conditions, and the maximum safe operating temperature of the device when it overheats. For example, the thermal threshold information may include at least one of the following: operating temperature range, overheat protection mechanism, heat dissipation performance information, etc. The operating temperature range refers to the temperature range that the device should maintain during normal operation. Exceeding this range may lead to decreased device performance or damage. The overheat protection mechanism refers to the overheat protection mechanism built into some devices, such as automatic frequency reduction, shutdown, or limitation of certain functions, to protect the device from overheating damage. Heat dissipation performance refers to the device's ability to effectively manage and dissipate heat; poor heat dissipation can lead to a rapid increase in the thermal threshold, thereby affecting the stability and reliability of the device.

[0328] For example, server 100 can determine the current heat dissipation status of server 100 or client 200 based on heat threshold information. If the heat dissipation status of server 100 is poor or the heat dissipation status of client 200 is good, a relatively small first area is determined; if the heat dissipation status of server 100 is good or the heat dissipation status of client 200 is poor, a relatively large first area is determined.

[0329] In this way, the size of the first region can be flexibly adjusted based on the hot threshold of 100 on the server or 200 on the client, so that the computing pressure on both ends can be adapted to their respective hot thresholds.

[0330] In some embodiments, communication link information may include, but is not limited to, at least one of the following: communication link quality, delay, capacity, etc. In some embodiments, communication link quality typically refers to the ability of a signal to maintain its integrity and reliability during transmission. Key metrics for measuring communication link quality may include, but are not limited to, bit error rate (BER), signal-to-noise ratio (SNR), packet loss rate, etc.

[0331] In some embodiments, the latency of a communication link refers to the time required for data to travel from the source to the destination. Latency directly affects the quality of real-time communication (such as video). For example, latency can be divided into several components: propagation delay, transmission delay, queuing delay, and processing delay.

[0332] Propagation delay: The time it takes for a signal to travel through a medium, usually related to the signal propagation speed and distance. Propagation delay = distance / propagation speed.

[0333] Transmission delay: The time it takes for a data packet to be sent over a link, which depends on the size of the data packet and the bandwidth of the link. Transmission delay = Data packet size / Bandwidth.

[0334] Queuing delay: The time it takes for data to queue for processing in network devices (such as routers and switches), which is related to the degree of network congestion.

[0335] Processing latency: The time required for a network device to process data packets, including packet inspection and forwarding.

[0336] In some embodiments, the capacity of a communication link refers to the amount of data that the communication link can transmit per unit time, typically expressed in bits per second (bps), such as 1 G / s. Factors affecting capacity may include, but are not limited to, bandwidth, modulation method, and signal quality.

[0337] For example, when the communication link between the server 100 and the client 200 is strong, the server 100 can determine a larger first region, thereby increasing the bandwidth usage of the bitstream encoded in the first region, so as to quickly and efficiently realize the graphics calculation of the image to be processed; when the communication link between the server 100 and the client 200 is poor, the server can determine a smaller first region, thereby reducing the bandwidth usage of the bitstream encoded in the first region, thereby reducing the packet loss rate and avoiding reducing the display effect of the image after graphics calculation.

[0338] For example, a strong communication link can be reflected in at least one of the following indicators: high communication link quality, low latency, and large capacity.

[0339] In this way, the size of the first region can be flexibly adjusted based on the status of the communication link between the server 100 and the client 200, thereby ensuring fast and accurate transmission of the bitstream when transmitting the bitstream encoded from the first intermediate result of the first region through the communication link.

[0340] In this embodiment, the server 100 can use scene information corresponding to the image to be processed to divide the image into regions, so that the first region obtained is closely related to the scene of the image to be processed. In this way, when the scene corresponding to the image to be processed changes, the first region determined by the scene information in this embodiment can also change flexibly with the scene. In addition, the server 100 can also use at least one of the network status information, computing power, power consumption, and heat threshold of the server 100 and / or the client 200 communicating with the server 100 to divide the image into regions. In this way, the first region divided by the server 100 can be adaptively adjusted in combination with the network status, computing power, power consumption, heat threshold, etc. of both ends, avoiding unnecessary calculations, saving the amount of calculation for the first intermediate result, and reducing the transmission bandwidth of the bitstream obtained by encoding the first intermediate result. In addition, the server 100 can adaptively determine the first region in the image to be processed based on the information of the communication link between the two ends, so that the size of the determined first region can match the condition of the communication link. For example, it can avoid the situation where the capacity of the communication link is too small to transmit the bit stream, reduce network congestion, and reduce the latency of end-to-cloud interaction.

[0341] The second piece of information mentioned above is also referred to as "split information (info_Split)" in the following embodiments.

[0342] In some embodiments, the process 400 of the image processing method of this application may further include: obtaining a second region in the image to be processed based on the second information, wherein the scene change degree of the second region is lower than that of the first region; encoding the fourth information to obtain the bitstream, wherein the fourth information is used to indicate the position of the second region in the image to be processed.

[0343] In this embodiment, the server 100 can not only use second information (e.g., segmentation basis information) to segment a first region in the image to be processed that is related to scene changes, but also segment a second region in the image to be processed. In the image to be processed, the degree of scene change in the second region is lower than the degree of scene change in the first region.

[0344] For example, the second region could be a region where there is no scene change.

[0345] In some embodiments, there may be no overlapping positions between the first region and the second region, and the first region and the second region may constitute a complete image to be processed. Alternatively, there may be overlapping positions (e.g., regions) between the first region and the second region, such that the union of the set of positions corresponding to the first region and the set of positions corresponding to the second region is the set of all positions corresponding to the image to be processed.

[0346] In this embodiment of the application, the server 100 encodes the fourth information used to indicate the position of the second region in the image to be processed into the bitstream, so that the client 100 can determine the second region in the image to be processed according to the fourth information, and then use the second region and the first intermediate result corresponding to the first region to reconstruct the intermediate result of the image to be processed, thereby reducing the amount of computation required for the client 200 to determine the second region.

[0347] In some embodiments, the server 100 may not encode the fourth information into the bitstream. In this way, the client 200 may use the region other than the first region (or a local region in the region other than the first region) in the image to be processed as the second region, and the bandwidth usage of the bitstream may be reduced.

[0348] Referring again to FIG4, in some embodiments, the process 400 of the image processing method may further include: S401c.

[0349] S401c, the encoding method for the server 100 to obtain the first intermediate result.

[0350] When the server 100 performs the above S403a to encode the first intermediate result, it can obtain the encoding method of the first intermediate result and encode the first intermediate result based on the encoding method to obtain the above bitstream.

[0351] In this embodiment of the application, the server 100 can flexibly determine the encoding method of the first intermediate result, thereby enabling flexible encoding of the first intermediate result.

[0352] The implementation process of S401c and S403a will be described below in conjunction with different encoding methods.

[0353] Encoding method 1

[0354] Server 100 can directly encode the first intermediate result according to encoding method 1 to obtain the bitstream.

[0355] Referring to Figure 5a and Figure 5b (3), the server 100 can directly encode the first intermediate result B of the first region 11 to obtain the bitstream.

[0356] In this embodiment of the application, the server 100 can directly encode the first intermediate result of the first region, thereby simplifying the encoding complexity of the first intermediate result and improving the encoding efficiency.

[0357] Based on the above encoding method 1, encoding method 1 can be further refined into encoding method 1.1.

[0358] The server 100 can obtain a reference frame, which is an intermediate result of a reference image corresponding to the image to be processed or a first intermediate result of the first region of the reference image; then, the server 100 can encode the first intermediate result in the image to be processed according to the reference frame to obtain the bitstream.

[0359] The reference image corresponding to the image to be processed can be any one or more frames that precede the image to be processed in either the display order or the encoding order. Taking the cloud gaming scenario mentioned above as an example, the reference image could be the previous frame of the game screen that was displayed before the image to be processed.

[0360] Referring to Figure 5a, the intermediate result of the reference image corresponding to the image to be processed represents the intermediate result obtained by graphics calculation of the complete region of the reference image (e.g., the first region 11 and the second region 12). The first intermediate result of the first region of the reference image corresponding to the image to be processed represents the first intermediate result obtained by graphics calculation of the first region 11, which is the same region as the first region 11 of the image to be processed shown in Figure 5a. That is to say, the reference frame in the various embodiments of this application does not refer to the reference image corresponding to the image to be processed, but rather to the intermediate result corresponding to the reference image corresponding to the image to be processed, such as the rendering intermediate result obtained by rendering the complete region (also called the whole scene) of the reference image, or the rendering intermediate result of the first region obtained by rendering the first region of the reference image.

[0361] In some embodiments, the server 100 can obtain a reference frame through the encoder shown in FIG2b, and encode the first intermediate result of the first region of the image to be processed by means of intra-frame prediction or inter-frame prediction based on the reference frame.

[0362] In this embodiment, the intermediate result of the reference image corresponding to the image to be processed (the intermediate result of the complete area of ​​the reference image or the intermediate result of the first area of ​​the reference image) is used as the reference frame for the first intermediate result of the first area in the image to be processed to encode the first intermediate result. In this way, when the scene changes of the first area in the two or more frames before and after are small (or the scene changes are not obvious), such as when the lighting changes are small, the intermediate result of the previous frame or several frames can be used as the reference frame to encode the first intermediate result of the first area of ​​the current frame (i.e., the image to be processed). This can reduce the amount of data encoded from the time dimension, shorten the length of the bitstream, and further reduce the bitstream's occupation of the transmission bandwidth.

[0363] Encoding method 2

[0364] The server 100 obtains preset data corresponding to the image to be processed, the preset data including a first preset intermediate result corresponding to the first region in the image to be processed; then, the server 100 can encode the first intermediate result according to the preset data to obtain the bitstream.

[0365] As an example, taking cloud gaming as an example, and using lighting calculation in graphics rendering as an example, in the production phase, after the game scene is built, server 100 can calculate lighting data for game scenes with no scene changes; and server 100 can calculate lighting data for game scenes with regular scene changes, thereby obtaining preset lighting data for each scene in the game. For example, for changing lighting in a scene (such as the alternating day and night lighting over a 24-hour period), lighting data can be calculated every preset time interval (such as calculating a piece of lighting data corresponding to the current moment every five minutes) to obtain the preset data for that scene.

[0366] Before cloud gaming runs, the server 100 can save the preset data of each frame in the game in the memory. In addition, the preset data of each scene can be synchronized to the client 200.

[0367] The first preset intermediate result (e.g., the first preset rendering intermediate result) corresponding to the first region included in the aforementioned preset data can be preset data about the first region in the image to be processed, or it can be preset data about the first region in the "nearest frame" of the image to be processed. Here, the "nearest frame" of the image to be processed refers to at least one frame whose display time or encoding time is before and / or after the image to be processed.

[0368] In this embodiment, the server 100 can encode the first intermediate result of the first region in the image to be processed based on preset data corresponding to the image to be processed. This preset data includes the first preset intermediate result corresponding to the first region. Thus, the encoding of the first intermediate result of the first region can be achieved by referring to the preset data (specifically, the aforementioned first preset intermediate result) used for graphic calculations on the image to be processed in the spatial dimension of the image to be processed. Therefore, when a scene switch occurs from the previous frame to the current frame (specifically, the image to be processed), compared to the direct encoding scheme mentioned in method 1 for encoding the first intermediate result of the current frame, this embodiment reduces the amount of encoded data and improves encoding efficiency by referencing the preset data corresponding to the current frame to encode the first intermediate result of the first region of the current frame.

[0369] In some embodiments, the preset data corresponding to the image to be processed in the above encoding method 2 is the preset intermediate result of the image to be processed. Then, the first preset intermediate result corresponding to the first region is specifically the preset intermediate result of the first region in the preset intermediate result of the image to be processed.

[0370] As an example, referring to Figure 5b (1) in conjunction with Figure 5a, the preset data 100 can be an intermediate result of the whole scene (first region 11 and second region 12) of the image to be processed 300 obtained by pre-graphic calculation, such as the preset data obtained by performing lighting calculation on the whole scene of the image to be processed 300 in the production state.

[0371] As shown in Figure 5b(1), the preset intermediate result (i.e., preset data 100) of the image to be processed 300 may include the preset intermediate result A of the first region 11, and optionally also include the preset intermediate result of the second region 12 (e.g., preset data of illumination).

[0372] In this embodiment, the server 100 can encode the first intermediate result of the first region in the image to be processed based on the preset intermediate result of the image to be processed. In this way, when the image to be processed has a corresponding preset intermediate result, the preset intermediate result of the current frame can be directly reused to quickly encode the first intermediate result of the first region of the current frame and improve the encoding efficiency.

[0373] Alternatively, in some embodiments, the preset data corresponding to the image to be processed in the above encoding method 2 is: a preset intermediate result of the reference image corresponding to the image to be processed. Then the first preset intermediate result corresponding to the first region is the preset data corresponding to the first region in the reference image corresponding to the image to be processed (i.e., the intermediate result obtained by pre-performing graphic calculations). The "reference image" mentioned in this embodiment is different from the "reference image" mentioned in the above "encoding method 1.1". In this embodiment, the reference image corresponding to the image to be processed is the "nearby frame" of the image to be processed.

[0374] As mentioned above, the server 100 performed illumination calculations for each scene and obtained the corresponding preset data for each scene. The preset data here is the preset data obtained by the server 100 from the illumination calculation of the "nearest frame" (e.g., the whole scene) of the image to be processed.

[0375] In some scenarios, the server 100 may not have pre-calculated the preset intermediate results of the image to be processed. However, it may have pre-calculated the preset intermediate results of the "nearby frames" of the image to be processed. In this way, the server 100 can use the preset intermediate results of the "nearby frames" to obtain (e.g., through interpolation) the preset intermediate results of the current frame. Thus, the server 100 can refer to the obtained preset intermediate results of the current frame to encode the first intermediate result of the first region in the image to be processed, thereby reducing the amount of encoded data and improving encoding efficiency when the scene changes in the current frame.

[0376] Based on the above encoding method 2, encoding method 2 can be further refined into encoding method 2.1.

[0377] When the server 100 encodes the first intermediate result according to the preset data using the encoding method 2 to obtain the bitstream, the server 100 can obtain the first preset intermediate result corresponding to the first region in the image to be processed based on the preset data; then, the server 100 can obtain the first difference data based on the first intermediate result and the first preset intermediate result corresponding to the first region; finally, the server 100 can encode the first difference data to obtain the bitstream.

[0378] Referring to Figure 5a and then to Figure 5b(1), the server 100 can obtain the preset intermediate result A (e.g., a pre-calculated rendering intermediate result) of the first region 11 from the preset data 100 of the image to be processed 300. The server 100 pre-calculates the illumination of the image to be processed to obtain the preset intermediate result of the image to be processed (specifically, the preset data 100). Therefore, the preset intermediate result A in the preset data 100 is the first preset intermediate result corresponding to the first region. In other embodiments, the server 100 does not pre-calculate the illumination of the image to be processed, but instead pre-calculates the illumination of its "nearby frames." Based on the preset intermediate result of the first region 11 in the "nearby frames" (an example of the first preset intermediate result), it can interpolate to obtain the preset intermediate result of the first region 11 of the image to be processed.

[0379] Furthermore, Figure 5b(1) also shows the first intermediate result B of the first region 11 (for example, the intermediate result obtained by real-time graphic calculation in S402 of Figure 4 above). In this embodiment, as shown in Figure 5b(2), the server 100 can calculate the difference C1 between the first intermediate result B of the first region 11 and the preset intermediate result A of the first region, and then encode the difference C1 to obtain the above-mentioned bitstream.

[0380] In other embodiments, if the preset data is a preset intermediate result of a reference image corresponding to the image to be processed, the server 100 can obtain a preset intermediate result of the first region in the image to be processed (e.g., obtained by interpolation) based on the preset data, and then calculate the first difference data to encode the first intermediate result of the first region.

[0381] In this embodiment, a preset intermediate result for a first region in the image to be processed can be obtained based on preset data of the image to be processed. A difference data is calculated between the first intermediate result and the preset intermediate result for the first region, and this difference data is encoded to achieve the encoding of the first intermediate result for the first region. In this scheme, the difference data between the preset data for the first region and the first intermediate result for the first region in the image to be processed can be calculated, and this difference data can be encoded (for example, by inputting the difference data into the encoder shown in Figure 2b for encoding), thereby reducing the amount of encoded data, improving encoding efficiency, shortening the bitstream length, and reducing bandwidth usage.

[0382] Encoding method 3

[0383] Encoding method 3 is a combination of encoding method 1 and encoding method 2.1. In encoding method 3, the intermediate result obtained by graphic calculation on a part of the first region of the image to be processed is encoded using the "difference" scheme mentioned in encoding method 2.1, while the intermediate result obtained by graphic calculation on another part of the first region is encoded using the "direct encoding" scheme mentioned in encoding method 1.

[0384] For example, a first intermediate result of a first region in the image to be processed may include a second intermediate result obtained by the graph calculation of a portion of the first region, and a third intermediate result obtained by the graph calculation of another portion of the first region.

[0385] The server 100 can obtain a preset intermediate result of the other part of the first region in the image to be processed based on the preset data; then, the server 100 can obtain a second difference data based on the second intermediate result and the preset intermediate result of the other part; after that, the server 100 can encode the second difference data, the second intermediate result and the third information to obtain the bitstream, wherein the third information is used to indicate the position of a part of the first region and / or the other part in the image to be processed.

[0386] As an example, referring to Figures 5a and 5b(1) above, and referring to Figure 5c, as shown in Figure 5c(1), the first region 11 can be divided into at least two parts, namely region 11c (an example of the other part mentioned above) and region 11d (an example of one part mentioned above). Then the first intermediate result B of the first region 11 can include the intermediate result B1 of region 11c (an example of the third intermediate result) and the intermediate result B2 of region 11d (an example of the second intermediate result). Similarly, the preset intermediate result A of the first region 11 can also be divided into the preset intermediate result A1 of region 11c and the preset intermediate result A2 of region 11d. When the server 100 encodes the first intermediate result B (specifically intermediate result B1 and intermediate result B2) of the first region 11 shown in Figure 5c(1), it can calculate the difference C2 between the intermediate result B1 of region 11c and the preset intermediate result A1 of region 11c, as shown in Figure 5c(2). Then, as shown in Figure 5c(1), the difference C2 of region 11c, the intermediate result B2 of region 11d in the first intermediate result B and the third information can be encoded to realize the encoding of the first intermediate result B (composed of intermediate result B1 and intermediate result B2) of the first region 11 shown in Figure 5c(1) and obtain the above-mentioned bitstream.

[0387] The third information may be information used to indicate the position of at least one of regions 11c and 11d as shown in FIG5c(1) in the first region 11.

[0388] Regarding the division of regions 11c and 11d within the first region 11, the server 100 can flexibly divide them according to its needs, and there are no restrictions on this division strategy here.

[0389] In this embodiment, when encoding the first intermediate result of the first region in the image to be processed, the intermediate result of a portion of the first intermediate result can be directly encoded, and the intermediate result of another portion of the first intermediate result can be calculated by referring to the preset data of that other portion of the intermediate result, and the difference can be encoded to encode the intermediate result of the other portion of the first intermediate result. Thus, when the preset data of the image to be processed may be incomplete, for example, the preset data may not include the preset intermediate result of the aforementioned portion of the first region, the server can flexibly apply the above-mentioned "difference encoding" scheme (e.g., encoding method 2.1) to the regions with preset intermediate results on the client side, based on the regions corresponding to the preset data of the image to be processed on the client side, and use the above-mentioned "direct encoding" scheme (e.g., encoding method 1) to flexibly and completely encode the first region of the image to be processed.

[0390] It should be understood that Figures 5a to 5c are related figures, and the same reference numerals and illustrations among them have the same or similar meanings, which will not be elaborated here.

[0391] Referring again to Figure 4, in some embodiments, the image processing method may further include: S401d, S403d.

[0392] S401d, the server 100 obtains the reconstruction method of the image to be processed.

[0393] For example, the reconstruction method corresponds to the encoding method.

[0394] The server 100 can obtain a reconstruction method corresponding to the encoding method of the first intermediate result, which is the reconstruction method of the image to be processed (e.g., the reconstruction method of the intermediate result of the image to be processed). For example, taking graphics computation as rendering processing, the reconstruction intermediate result is the reconstruction data of the rendering intermediate result of the image to be processed. It should be understood that the reconstruction intermediate result is not only the reconstruction data of the rendering intermediate result of the first region of the image to be processed (an example of the first intermediate result), but may also include the reconstruction data of the rendering intermediate result of the region outside the first region of the image to be processed (e.g., the second region mentioned above).

[0395] S403d, the server 100 encodes the reconstruction method into the above bitstream.

[0396] The following exemplarily illustrates a data structure for a bitstream according to an embodiment of this application:

[0397] 1) Data Header (Optional): Used to record metadata of the bitstream.

[0398] 2) Encoded data corresponding to the first intermediate result (required): When the server 100 encodes the first intermediate result of the first region in the image to be processed, the encoded data object can be, for example, the first intermediate result mentioned in encoding method 1 above, the "difference data" mentioned in encoding method 2.1 above, or the "difference data" of a part of the first region and the "intermediate result" of another part of the first region mentioned in encoding method 2.2.

[0399] 3) First information (optional): Used to indicate the location of the first region in the image to be processed.

[0400] 4) Fourth information (optional): used to indicate the location of the second region in the image to be processed.

[0401] 5) Reconstruction method of the image to be processed (optional): Corresponds to the encoding method of the first intermediate result.

[0402] In other embodiments, the bitstream may include more or fewer fields, which is not limited here.

[0403] If the first intermediate result is encoded using the above-described encoding method 1, then the reconstruction method of the image to be processed can be reconstruction method 1 of the following embodiment;

[0404] If the first intermediate result is encoded using the above-described encoding method 1, then the reconstruction method of the image to be processed can be reconstruction method 1 of the following embodiment;

[0405] If the first intermediate result is encoded using the above-described encoding method 1.1, then the reconstruction method of the image to be processed can be reconstruction method 1.1 of the following embodiment;

[0406] If the first intermediate result is encoded using the above-described encoding method 2, then the reconstruction method of the image to be processed can be the reconstruction method 2 of the following embodiment;

[0407] If the first intermediate result is encoded using the above-described encoding method 3, then the reconstruction method of the image to be processed can be the reconstruction method 3 of the following embodiment.

[0408] For example, the bitstream can define an enumeration value for the reconstruction method. This enumeration value can be 0, 1, or 2, where 0, 1, and 2 correspond to reconstruction method 1, reconstruction method 2, and reconstruction method 3, respectively. Furthermore, the reconstruction methods carried in the bitstream are not limited to the examples given here; they can include more or fewer reconstruction methods, without limitation.

[0409] Of course, the data structure representing this reconstruction method in the bitstream is not limited to enumeration values; it can also be a linked list, array, etc. There are no restrictions here.

[0410] In this embodiment, the server 100 can obtain a reconstruction method corresponding to the encoding method of the first intermediate result of the first region, and encode the reconstruction method into the bitstream. Thus, the client 200 can reconstruct the intermediate result of the image to be processed according to the reconstruction method indicated by the server 100 in the bitstream, and then perform further graphics calculations on the intermediate result of the image to be processed to obtain the graphics calculation result (e.g., rendering result) of the image to be processed. The rendering result of the image to be processed is the rendered image of the image to be processed. Therefore, the client 200 does not need to determine the reconstruction method of the image to be processed itself; it can directly decode the reconstruction method of the image to be processed from the bitstream, thereby reducing the decoding calculation load of the client 200. Furthermore, since the reconstruction method corresponds to the encoding method of the first intermediate result, the reconstruction intermediate result of the image to be processed obtained by reconstructing with this reconstruction method can include the accurate reconstruction data of the first intermediate result.

[0411] Referring again to Figure 4, in some embodiments, the process 400 of the image processing method may further include: S401e, S403e.

[0412] S401e, the server 100 obtains the indication information corresponding to the image to be processed, and the indication information is used to indicate the preset data corresponding to the image to be processed.

[0413] As an example of implementation 1, for instance, if the graphics calculation performed by the server 100 on the first region is DDGI rendering processing, and the preset data corresponding to the image to be processed is the preset intermediate result of the image to be processed, then the indication information may include the identifier of the preset data of the image to be processed, which can be used to indicate the preset intermediate result of the image to be processed.

[0414] As an implementation example 2, for example, if the graphics calculation performed by the server 100 on the first region is DDGI rendering processing, and the preset data corresponding to the image to be processed is the preset intermediate result of the reference image (e.g., "nearby frame") mentioned in the above embodiments, then the indication information may include an identifier of the preset data of the reference image, which can be used to indicate the preset intermediate result of the reference image, and the indication information also includes interpolation coefficients, so that the client 200 can use the interpolation coefficients and the preset intermediate result of the reference image to obtain the preset intermediate result of the image to be processed.

[0415] As an implementation example 3, for instance, if the server 100 performs a spherical reflection rendering process on the first region, and the preset data corresponding to the image to be processed is a preset intermediate result of the image to be processed, then the indication information may include the identifier of the reflection probe in the image to be processed and the identifier of the face of the reflection probe that can reuse preset data (e.g., the coordinates of the face). Here, a reflection probe has 6 faces. Thus, the client 200 can use the indication information to obtain the preset data of the reflection probe indicated by the identifier of the aforementioned reflection probe, and based on the identifier of the aforementioned face, extract the preset data of the face indicated by the identifier of the aforementioned face from the preset data.

[0416] It should be understood that the above three implementation examples are only used to illustrate the indication information. In other embodiments, the indication information may have other information that can indicate the preset data corresponding to the image to be processed, which is not limited here.

[0417] S403e, the server 100 encodes the indication information into the bitstream.

[0418] In this embodiment, since the server 100 performs graphic calculations on the first region in the current frame (specifically, the image to be processed) and encodes the first intermediate result of the first region obtained from the graphic calculation into the bitstream, the decoding end (e.g., the client 200) can reuse the preset data of the current frame or adjacent frames and combine it with the first intermediate result to reconstruct the intermediate result of the entire scene reconstruction of the current frame. Therefore, the server 100 can encode the indication information used to indicate the preset data corresponding to the image to be processed into the bitstream and synchronize it to the client 200, so that the client 200 can use the indication information to find the preset data corresponding to the image to be processed and combine it with the first intermediate result to reconstruct the intermediate result of the image to be processed.

[0419] In some embodiments, when the client 200 decodes and reconstructs different frames to obtain the intermediate reconstruction results of the corresponding frames, it references preset data of the same frame (or a certain face of the same probe). For example, when encoding the first intermediate result of the first region of the previous frame of the current frame using the method of this application embodiment, preset data of the adjacent frame (e.g., the current frame is frame 9, and the adjacent frame is frame 10) is referenced. Similarly, when encoding the first intermediate result of the first region of the current frame, the preset data of frame 10 is also referenced. That is, the preset data of the same frame is referenced when encoding the two frames, so the server 100 does not need to encode the indication information into the bitstream. In this way, the client 200 can directly use the preset data corresponding to the previous frame to decode the current frame to obtain the intermediate reconstruction result of the current frame.

[0420] In some embodiments, the indication information in S401e may include information identifying preset data corresponding to the image to be processed. For example, the indication information may include the acquisition address of the preset data corresponding to the image to be processed.

[0421] Taking the scenario of Example 1 above as an example, in one possible implementation, the server 100 can store the preset intermediate result of the image to be processed on a remote server, and encode the access address (an example of obtaining the address) of the preset intermediate result of the image to be processed on the remote server as indication information into the bitstream for synchronization with the client 200. In this way, the client 200 can decode the access address from the bitstream and access the remote server according to the access address to obtain the preset intermediate result of the image to be processed.

[0422] In this embodiment, the server 100 can encode the acquisition address of the preset data corresponding to the image to be processed into a bitstream and synchronize it to the client 200, so that the client 200 can use the acquisition address to acquire the preset data corresponding to the image to be processed, and use it to decode and obtain the intermediate reconstruction result of the image to be processed.

[0423] Based on the data structure of the bitstream shown in the above embodiments, the following further illustrates a data structure of the bitstream according to an embodiment of this application, which may further include:

[0424] 6) Flag (optional): Used to record whether the bitstream carries indication information.

[0425] 7) Indication information corresponding to the image to be processed (optional): When the value of the flag field is 0, it means that the bitstream does not carry indication information, and there is no field for "indication information" in the bitstream; when the value of the flag field is 1, the bitstream has the field for "indication information".

[0426] In another possible data structure, only the field containing the aforementioned indication information corresponding to the image to be processed can be included, without including the aforementioned flag field.

[0427] In this embodiment of the application, the server 100 can encode the indication information used to indicate the preset data corresponding to the image to be processed into the bitstream. In this way, the decoding end (e.g., the client 200) can use the indication information obtained from the bitstream to directly obtain the preset data that needs to be referenced, so as to combine the first intermediate result of the first region obtained by decoding to reconstruct the intermediate result of the image to be processed, thereby improving the reconstruction efficiency of the image to be processed.

[0428] Figure 6 is a flowchart of a process 600 of an image processing method provided in an embodiment of this application. Process 600 can be executed by the client 200 described above. Process 600 is described as a series of steps or operations. It should be understood that process 600 can be executed in various orders and / or occur simultaneously, and is not limited to the execution order shown in Figure 6. Process 600 may include:

[0429] S601a, the client 200 decodes the bitstream to obtain the decoded data corresponding to the first intermediate result of the image to be processed.

[0430] The first intermediate result is obtained by performing a first graphic calculation on a first region in the image to be processed, where the first region is a region related to scene changes.

[0431] Client 200 can decode the bitstream using a standard video decoder to obtain decoded data corresponding to the first intermediate result, such as the first intermediate result or the difference data corresponding to the first intermediate result. This standard video decoder can be implemented according to industry video compression standards, such as ITU-T H.264, H.265 / HEVC, H.266 / VVC, AVS2, AVS3 standards, or extensions of such standards. The technology in this application is not limited to any specific decoding standard.

[0432] For a detailed description of the first region, please refer to the corresponding description of S401a in the embodiment of Figure 4 on the encoding side, which will not be repeated here.

[0433] S603, the client 200 obtains the graphic calculation result of the image to be processed based on the decoded data corresponding to the first intermediate result.

[0434] For example, taking Figures 5a and 5b(1) as examples, the client 200 can decode the first intermediate result B of the first region 11 shown in Figure 5b(1), and then use the first intermediate result B to obtain the graphics calculation result of the whole scene (e.g., the first region 11 and the second region 12) of the image to be processed 300 shown in Figure 5a (e.g., if the graphics calculation is rendering processing, then the graphics calculation result is the rendering result).

[0435] In this embodiment, the client 200 can decode the bitstream to obtain decoded data corresponding to a first intermediate result of the image to be processed. This first intermediate result is obtained by performing graphical calculations on a first region of the image to be processed that is related to scene changes, rather than by performing graphical calculations on the entire scene of the image to be processed. Therefore, compared to decoding the bitstream to obtain decoded data of the intermediate result of the entire scene of the image to be processed, this embodiment can decode the bitstream to obtain decoded data corresponding to the first intermediate result of a local region of the image to be processed. This reduces the amount of decoded data, improves decoding efficiency, reduces the bitstream's bandwidth usage, and thus reduces latency. Furthermore, the client 200 can obtain the graphical calculation result of the image to be processed based on the decoded data corresponding to the first intermediate result. This allows for the reconstruction of the graphical calculation result of the entire scene of the image to be processed, even when decoding the decoded data corresponding to the intermediate result of a local region of the image to be processed, ensuring the accuracy of the graphical calculation result of the image to be processed.

[0436] Optionally, in one embodiment, the process 600 may further include S602.

[0437] S602, the client 200 obtains the reconstruction intermediate result of the image to be processed (i.e., the reconstruction data of the intermediate result of the image to be processed) based on the decoding data corresponding to the first intermediate result.

[0438] Among them, the reconstruction intermediate result is the reconstruction data of the intermediate result obtained after the image to be processed is processed by graphics calculation (such as the rendering intermediate result of the image to be processed).

[0439] When client 200 executes S603, the graphic calculation result of the image to be processed can be obtained based on the intermediate reconstruction result of the image to be processed.

[0440] Specifically, the client 200 can perform a second graphics calculation (e.g., simple rendering) based on the intermediate reconstruction results to obtain the graphics calculation results (e.g., rendering results) of the image to be processed.

[0441] Compared to the first graph calculation mentioned above, the second graph calculation requires less computing power.

[0442] In this embodiment of the application, the client can use the decoded data corresponding to the first intermediate result of the first region of the image to be processed obtained by decoding the bitstream to obtain the reconstruction data of the intermediate result of the image to be processed (e.g., the whole scene). In this way, the client does not need to perform graphics calculations on the first region related to scene changes, thereby reducing the pressure of graphics calculations on the end side and improving the efficiency of obtaining the graphics calculation results of the image to be processed.

[0443] In some embodiments, the client 200 may save the reconstruction data of the intermediate results of the obtained image to be processed (e.g., the reconstruction intermediate results described above) to a remote storage server.

[0444] Thus, in this embodiment of the application, by saving the intermediate reconstruction result of the current frame on a local or remote server, the reconstruction result can be used as the preset data for the subsequent frames of the current frame, so as to reconstruct the intermediate reconstruction result of the subsequent frames. This can realize the updating of the preset data saved by the client, and also facilitate the sharing of preset data between the client and other end sides, thereby reducing the computing overhead of the end-to-cloud.

[0445] As shown in the dashed box and dashed arrow in Figure 6, the process 600 of this application embodiment may also include some optional steps, such as S601b and S602b.

[0446] In one embodiment, the client 200 may also obtain first information via S601b, the first information being used to indicate the location of the first region in the image to be processed; then, via S602b, the client obtains the graphic calculation result of the image to be processed based on the decoded data corresponding to the first intermediate result and the first information (optionally further including S602).

[0447] In some embodiments, the first information may be the first information encoded into the bitstream in the image processing method executed on the server 100 side (e.g., the first information mentioned in S403b in FIG4).

[0448] Taking Figure 5a as an example, the first information can be used to indicate the position of the first region 11 in the image 300 to be processed. For example, the first information can be the index of each probe in the first region 11.

[0449] Based on the differences in the content indicated by the decoded data, when the client 200 uses the decoded data and the first information to obtain the graphics calculation result through S602b, in some embodiments, the client 200 can use the first information to find the first region 11 shown in FIG5a, and use the decoded data and the first region 11 to obtain the graphics calculation result; in other embodiments, the client 200 can also use the first information to find the first region 11 and the second region 12, and use these two regions and the decoded data to obtain the graphics calculation result.

[0450] In the example of Figure 5a, the first region 11 and the second region 12 can constitute the entire region (i.e., the complete region) of the image to be processed, and there is no overlap between the first region 11 and the second region 12. In other embodiments, the first region and the second region can together constitute a local region of the image to be processed, rather than the complete region of the image to be processed, which is not a limitation here.

[0451] In this context, the first region is related to scene changes, while the second region is the region in the image to be processed that is unrelated to scene changes. For example, in the case of Figure 5a, when character 10 enters the image to be processed 300, the lighting in the second region 12 will not change, but only the lighting in the first region 11 will change.

[0452] In some embodiments, as shown in FIG6, when the client 200 obtains the graphic calculation result of the image to be processed based on the decoding data corresponding to the first information and the first intermediate result through S602b, it can obtain the reconstruction intermediate result of the image to be processed based on the decoding data corresponding to the first information and the first intermediate result, and then obtain the graphic calculation result of the image to be processed based on the reconstruction intermediate result.

[0453] In this embodiment, the client 200 can obtain first information, which indicates the position of a first region related to scene changes in the image to be processed. This first information allows the client to determine the location of the first region in the image to be processed that corresponds to the first intermediate result. Thus, the client 200 can use the position of the first region in the image to be processed, corresponding to the first intermediate result, and the decoded data corresponding to the first intermediate result, to obtain the graphics calculation result of the image to be processed, ensuring the accuracy of the graphics calculation result obtained by the client 200.

[0454] In some embodiments, when obtaining the first information, the client 200 may also determine the position of the first region in the image to be processed based on the division criteria information (such as the second information described below) to obtain the first information.

[0455] In this way, when the client 200 has superior computing performance, it can flexibly divide the regions in the image to be processed according to the division criteria information used to divide the image into regions, so that the client 200 can determine the position of the first region in the image to be processed.

[0456] In some embodiments, the client 200 may send first information to the server 100 before receiving the aforementioned bitstream. In this way, when the server 100 executes S401b shown in FIG4, it can receive the first information from the client 200 without using the second information to perform region division in order to obtain the first information, thereby reducing the computational load of the server 100.

[0457] In other embodiments, as shown in FIG6, when the client 200 obtains the first information, it can also obtain the first information by decoding the above-mentioned bitstream.

[0458] In this embodiment of the application, the client 200 can decode the first information indicating the position of the first region in the image to be processed from the bitstream, so as to reduce the amount of computation required for the client 200 to calculate the first information. Even in scenarios where the client 200 has poor computing performance, the reconstruction of the graphic calculation results of the image to be processed can still be achieved.

[0459] In one embodiment, prior to S601a, process 600 may further include S501 and S502.

[0460] S501, Client 200 obtains the second information.

[0461] The second information is used to divide the image to be processed into regions.

[0462] In the following embodiments, the second information is also referred to as "division basis information".

[0463] S502, Client 200 sends the second message.

[0464] For example, client 200 can send the second information to server 100 in a single message.

[0465] Alternatively, client 200 may include the second information in the graphics calculation request (e.g., a rendering request) when sending it to server 100. In this way, server 100 can respond to the graphics calculation request by executing the relevant steps of the image processing method described above to obtain a bitstream, and then send the bitstream to client 200.

[0466] In this embodiment of the application, the client 200 may send second information for dividing the image to be processed into regions (e.g., dividing the first region mentioned above), so that the encoding side (e.g., the server 200) can obtain the first region related to scene changes based on the received second information, thereby improving the overall processing speed of the server 200 in the graphics calculation of the image to be processed.

[0467] In one embodiment, the second information in S502 may include at least one of the following: scene information corresponding to the image to be processed; one or more of the following: network status information, computing power, power consumption, and heat threshold of the server 100 and / or client 200; communication link information of the communication link between the server 100 and the client 200, wherein the server 100 and the client 200 communicate.

[0468] For example, one or more of the network status information, computing power, power consumption, and heat threshold information obtained by the client 200 from the server 100 may be information obtained in advance from the server 100 before the client 200 sends a graphics computing request to the server 100.

[0469] The above examples of the second information can be found in the image processing method on the encoding side (e.g., server 100). The description of the second information will not be repeated here.

[0470] In this embodiment, the client 100 can obtain scene information corresponding to the image to be processed, which can be used by the server 100 or the client 200 to divide the image to be processed into regions, so that the first region obtained is closely related to the scene of the image to be processed. In addition, the client 100 can also obtain at least one of the following information: network status, computing power, battery level, and heat threshold of the server 100 and / or the client 200 communicating with the server 100, which can be used by the server 100 or the client 200 to divide the image to be processed into regions, so as to ensure that at least one of the first region and the second region obtained can be adaptively adjusted based on the network status, computing power, battery level, heat threshold, etc. of both ends. In addition, the client 100 can also obtain the communication link information between the two ends for the purpose of dividing the image to be processed into regions, so that the first region obtained can be matched with the status of the communication link.

[0471] Referring again to Figure 6, in some embodiments, the process 600 may further include: S601c and S602c, to obtain the graphic calculation result of the image to be processed based on the decoded data corresponding to the first information and the first intermediate result.

[0472] S601c, Client 200 obtains the reconstruction method of the image to be processed.

[0473] For example, this reconstruction method can be a reconstruction strategy used to reconstruct intermediate results obtained from graphical computation of the image to be processed.

[0474] The reconstruction method can be determined by the client 200, or it can be obtained by the client 200 from the above bitstream; there is no restriction here.

[0475] S602c, the client 200 obtains the graphic calculation result of the image to be processed according to the first information and the decoded data and the reconstruction method (optionally, the intermediate reconstruction result of the image to be processed can be obtained first, and then the graphic calculation result can be obtained based on the intermediate reconstruction result).

[0476] For example, referring to Figure 6, S602c can be combined with the above S602. When the client 200 executes S602c, the client 200 can obtain the reconstruction intermediate result of the image to be processed according to the above reconstruction method based on the decoding data and first information corresponding to the first intermediate result, and then obtain the graphic calculation result of the image to be processed based on the reconstruction intermediate result.

[0477] In this embodiment, the client 200 can obtain the reconstruction method of the image to be processed, and obtain the graphics calculation result of the image to be processed according to the reconstruction method based on the first information and the decoded data. Thus, the client 200 can reconstruct the intermediate result of the image to be processed according to the reconstruction method, and then use the reconstructed intermediate result to obtain the graphics calculation result of the image to be processed, thereby improving the quality of the graphics calculation result of the image to be processed.

[0478] Based on the embodiments of S601c and S602c described above, as an implementation example 1, when the client 200 obtains the reconstruction method in S601c, it can be implemented through S701. In S701, the client 200 obtains the reconstruction method of the image to be processed based on the first information obtained above; then, when executing S602c, the client 200 can obtain the reconstruction intermediate result of the image to be processed based on the first information and the first intermediate result according to the reconstruction method, and then obtain the image calculation result.

[0479] The client 200 can flexibly determine the reconstruction method for reconstructing the intermediate results of the image to be processed based on the first information obtained (obtained from the bitstream decoding or determined based on the second information mentioned above, without limitation here).

[0480] For example, client 200 can select a portion of the data in the first intermediate result based on the client 200's status information, while the other portion of data not selected from the first intermediate result can use preset data. In addition, client 200 can obtain preset data of the second region in the image to be processed, excluding the first region, based on the first information (indicating the location of the first region). In this way, the reconstruction intermediate result of the image to be processed can be reconstructed.

[0481] For example, if the current network status of client 200 is poor, and its computing performance is also poor, while the first intermediate result obtained by server 100 requires a high-performance graphics computing engine, then client 200 can select a portion of the intermediate results from the first intermediate results based on the client-side status. All intermediate results in the image to be processed other than this selected portion will use preset data (i.e., preset intermediate results) to ensure that the graphics computing results of the image to be processed can be displayed smoothly. As for which portion of the first intermediate results to select, client 200 can determine this based on the first information.

[0482] Referring to Figure 5a above, Figure 7a is used as an example. Figure 7a(1) shows the preset data 100 of the image 300 to be processed stored on the client 200 side. The preset data 100 may include the preset intermediate result of the first region 11 (for example, the preset intermediate result A shown in Figure 5b(1)) and the preset intermediate result D of the second region 12. Figure 7a(2) shows the first intermediate result of the first region 11 obtained by the client 200 from decoding the bitstream (i.e., the preset intermediate result B shown in Figure 5b(1)).

[0483] When obtaining the intermediate reconstruction result of the image 300 to be processed as shown in Figure 7a(3), the client 200 can determine the position of the first region 11 in the image 300 to be processed based on the first information (e.g., the position of the second region 12 in the image 300 to be processed). The client 200 can divide the first intermediate result of the first region 11 as shown in Figure 7a(2) into two parts, namely the intermediate result B3 of region 11a and the intermediate result B4 of region 11b. When reconstructing the above intermediate reconstruction result, the intermediate result B3 (shown in shaded area here) is not used. Instead, the intermediate result of region 11a is given preset data (specifically, the preset intermediate result A3 of region 11a shown in Figure 7a(1)).

[0484] Regarding how to find the preset intermediate result A3, the preset intermediate result A3 of region 11a can be obtained from the preset data 100 based on the first information and the position of region 11a in the image to be processed 300.

[0485] Finally, the client 200 can use the first information to obtain the preset intermediate result D of the second region 12 from the preset data shown in FIG7a(1); and the client 200 also obtains the preset intermediate result A3 of the region 11a and the intermediate result B4 of the region 11b. Then, according to the position of the intermediate result in the image to be processed 300, the reconstruction intermediate result of the image to be processed 300 shown in FIG7a(3) can be obtained. The reconstruction intermediate result is composed of the preset intermediate result D of the second region 12, the preset intermediate result A3 of the region 11a and the intermediate result B4 of the region 11b.

[0486] In this embodiment, the client 200 can independently determine the reconstruction method of the image to be processed based on the position of the first region related to scene changes indicated by the first information in the image to be processed. This allows for flexible determination of the reconstruction method of the intermediate results of the image to be processed, without relying on the reconstruction method decided by the server 100, thus improving the flexibility of the client 200 in reconstructing the intermediate results of the image to be processed. For example, the client 200 can flexibly decide which part of the intermediate result calculated by the server 100 should be used to reconstruct the intermediate results of the image to be processed. Furthermore, since the client 200 can determine the reconstruction method of the intermediate results of the image to be processed independently, the server 100 does not need to send the reconstruction method to the client 200 via the bitstream, thereby reducing the data in the bitstream and reducing the bandwidth usage of the bitstream.

[0487] Based on the embodiments of S601c and S602c described above, as another implementation example 2, when the client 200 obtains the reconstruction method in S601c, as shown in Figure 6, the client 200 can decode the bitstream to obtain the reconstruction method of the image to be processed, which corresponds to the encoding method of the first intermediate result. Then, when executing S602c, the client 200 can obtain the reconstruction intermediate result of the image to be processed based on the first intermediate result and the first information, according to this reconstruction method.

[0488] In some embodiments, when the client 200 obtains the reconstruction intermediate result of the image to be processed based on the first intermediate result and the first information according to the reconstruction method, it may obtain the reconstruction intermediate result of the image to be processed based on the first intermediate result and the first information (which may be determined by itself or obtained from the bitstream decoding) to indicate the position of the first region in the image to be processed.

[0489] The reconstruction method is the reconstruction method of the intermediate result of the image to be processed. The reconstruction method corresponds to the encoding method of the first intermediate result. That is, if the encoding method of the first intermediate result changes, the reconstruction method will also be adjusted accordingly.

[0490] In this embodiment, the client 200 can decode the reconstruction method of the image to be processed from the bitstream, and then obtain the intermediate reconstruction result of the image to be processed based on the first intermediate result according to the reconstruction method, since the reconstruction method corresponds to the encoding method of the first intermediate result. In this way, the reconstruction quality of the intermediate reconstruction result of the image to be processed obtained according to the reconstruction method is higher, thereby improving the quality of the obtained graphics calculation result.

[0491] In some embodiments, when the client 200 executes S601a as shown in FIG6, as a specific example A, it can obtain a reference frame, which is the reconstructed data of the intermediate result of the reference image corresponding to the image to be processed; then, based on the reference frame, it decodes the bitstream to obtain the decoded data corresponding to the first intermediate result of the image to be processed.

[0492] Referring to Figure 3b, the client 200 can input the bitstream to the decoder so that the decoder can use the reference frame to decode the bitstream and obtain the decoded data corresponding to the first intermediate result.

[0493] For example, the intermediate result of the reference image mentioned above can be the intermediate result of the first region of the reference image or the intermediate result of the entire region of the reference image; there is no limitation here.

[0494] The description of the reference frame and reference image can be found in the relevant description in the embodiment of encoding method 1 on the encoding side, and will not be repeated here.

[0495] In this embodiment, a reference frame can be obtained, wherein the reference frame is the reconstructed data of the intermediate result of the reference image corresponding to the image to be processed. The bitstream can be decoded based on the reference frame (e.g., through inter-frame prediction or intra-frame prediction, which is not limited here) to obtain the decoded data corresponding to the first intermediate result of the image to be processed. In this way, when the scene changes of the first region in two or more consecutive frames are small (or the scene changes are not obvious), such as when the lighting changes are small, the reconstructed data of the intermediate result of the previous frame or several previous frames can be used as the reference frame. The bitstream can be decoded using the reference frame from the time dimension, thereby obtaining the decoded data corresponding to the first intermediate result of the first region of the current frame (i.e., the image to be processed). This can reduce the amount of data to be decoded and improve decoding efficiency.

[0496] In some embodiments, regardless of the reconstruction method corresponding to the above encoding method, when executing S602b (specifically, obtaining the graphic calculation result of the image to be processed based on the decoding data corresponding to the first intermediate result and the first information), preset data corresponding to the image to be processed can be obtained. The preset data includes the first preset intermediate result corresponding to the first region in the image to be processed. Then, the graphic calculation result of the image to be processed is obtained according to the preset data, the first information and the decoding data corresponding to the first intermediate result.

[0497] The preset data corresponding to the image to be processed obtained by the client 200 can be obtained based on the instruction information decoded from the bitstream, or it can be the preset data used when decoding the previous frame of the image to be processed (the previous frame whose decoding order or display order is before the image to be processed) to obtain its reconstruction intermediate result. There is no restriction here.

[0498] For example, the client 200 can use the first information and preset data to obtain a preset intermediate result for a region outside the first region (e.g., the second region); or, it can use the first information and preset data to obtain a preset intermediate result for the complete region of the image to be processed; then, the client 200 can use the preset intermediate result of the second region (or the preset intermediate result of the complete region of the image to be processed) and the decoding data corresponding to the first intermediate result to obtain the graphics calculation result of the image to be processed (e.g., first obtain the reconstruction intermediate result of the image to be processed, and then obtain the graphics calculation result based on the reconstruction intermediate result).

[0499] In this embodiment, the client 200 can obtain preset data corresponding to the image to be processed, and obtain the graphics calculation result of the image to be processed based on the preset data, first information indicating the position of the first region in the image to be processed, and decoding data corresponding to the first intermediate result of the first region. The preset data includes the first preset intermediate result corresponding to the first region, so that the intermediate result (e.g., reconstruction intermediate result) of the image to be processed can be reconstructed by referring to the preset data (specifically the aforementioned first preset intermediate result) for graphics calculation of the image to be processed in the spatial dimension of the image to be processed. Therefore, when a scene switch occurs from the previous frame to the current frame (specifically the image to be processed), this embodiment can reconstruct the reconstruction intermediate result of the current frame by referring to the preset data corresponding to the current frame and the decoding data corresponding to the first intermediate result of the first region of the current frame, and then obtain the graphics calculation result of the current frame, which can reduce the amount of data to be decoded and improve decoding accuracy and decoding efficiency.

[0500] In some embodiments, the preset data corresponding to the image to be processed is a preset intermediate result of the image to be processed, and the first preset intermediate result corresponding to the first region is specifically the preset intermediate result of the first region in the preset intermediate result of the image to be processed.

[0501] In this embodiment, the client 200 can reconstruct the intermediate reconstruction result of the image to be processed based on the preset intermediate result of the image to be processed, the decoding data corresponding to the first intermediate result of the first region in the image to be processed, and the first information. This allows for the decoding of the intermediate reconstruction result of the image to be processed in the spatial dimension of the image to be processed, with reference to the preset data used for graphic calculations on the image to be processed (specifically, the preset intermediate result of the image to be processed). Therefore, when a scene switch occurs from the previous frame to the current frame (specifically, the image to be processed), compared to intra-frame decoding of the first intermediate result of the current frame or using the intermediate reconstruction result of the previous frame as a reference frame to decode the intermediate reconstruction result of the current frame, this embodiment obtains the intermediate reconstruction result of the current frame by referencing the preset intermediate result of the current frame and the decoding data corresponding to the first intermediate result of the first region in the current frame. This reduces the amount of decoding data and improves decoding efficiency.

[0502] Alternatively, in some embodiments, the preset data corresponding to the image to be processed is: a preset intermediate result of a reference image corresponding to the image to be processed.

[0503] The "reference image" mentioned in this embodiment is different from the "reference image" mentioned in Specific Example A above. In this embodiment, the reference image corresponding to the image to be processed is the "nearby frame" of the image to be processed.

[0504] As mentioned above, the server 100 pre-calculates the illumination for each scene. The preset data here is the preset data obtained by the server 100 in pre-calculating the illumination of the "nearest frames" (e.g., the entire scene) of the image to be processed.

[0505] In some scenarios, the server 100 may not have pre-calculated the preset intermediate results for the image to be processed. However, it may have pre-calculated the preset intermediate results for the "nearby frames" of the image to be processed. The client 200 can then use the preset intermediate results of the "nearby frames" to obtain (e.g., through interpolation) the preset intermediate results for the current frame. In this way, the client 200 can refer to the decoded data corresponding to the obtained preset intermediate results for the current frame and the first intermediate results for the first region in the image to be processed to reconstruct the reconstructed intermediate results for the image to be processed. This reduces the amount of decoded data and improves decoding efficiency when the scene changes within the current frame.

[0506] The image processing process of server 200 is described below in conjunction with different reconstruction methods.

[0507] Reconstruction method 1 (also known as "segmentation method") corresponds to the encoding method 1 mentioned above.

[0508] When the server 100 encodes the first intermediate result, it uses encoding method 1 to directly encode the first intermediate result to obtain the bitstream. In this embodiment, the client 200 can decode the bitstream to directly obtain the first intermediate result (i.e., the decoded data corresponding to the first intermediate result indicates the first intermediate result).

[0509] For example, corresponding to the above encoding method 1.1, reconstruction method 1 can also be refined into reconstruction method 1.1. When the server 100 uses a reference frame to encode the first intermediate result to obtain a bitstream, the client 200 can decode the bitstream based on the reference frame through the process described in Example A above to obtain the first intermediate result. Afterwards, the client 200 can use a "concatenation" method to "concatenate" the preset data and the first intermediate result to obtain the graphic calculation result of the image to be processed.

[0510] In reconstruction method 1, or reconstruction method 1.1, the decoded data corresponding to the first intermediate result is used to indicate the first intermediate result. For example, the decoded data is the first intermediate result.

[0511] The following describes the "splicing" method:

[0512] When client 200 obtains the graphic calculation result of the image to be processed based on the preset data, the first information, and the decoding data corresponding to the first intermediate result, it can obtain the reconstruction intermediate result of the image to be processed based on the preset data and the first intermediate result. It can also obtain the preset intermediate result of the second region in the image to be processed based on the preset data and the first information. The second region is a region in the image to be processed that is different from the first region. Then, based on the first intermediate result and the preset intermediate result of the second region, the graphic calculation result of the image to be processed can be obtained (for example, first obtaining the reconstruction intermediate result of the image to be processed based on the first intermediate result and the preset intermediate result of the second region, and then obtaining the graphic calculation result based on the reconstruction intermediate result).

[0513] The following example, Figure 7b, illustrates the process of obtaining the intermediate reconstruction result 500 of the image to be processed according to reconstruction method 1.

[0514] The reconstruction method 1 shown in Figure 7b corresponds to the encoding method 1 shown in Figures 5b(1) and 5b(3).

[0515] Referring to Figure 5a, please refer to Figure 7b. As shown in Figure 7b(1), the client 200 can delete the preset intermediate result A of the first region 11 in the preset data 100 (represented by a vertical shaded line, for example, set to 0) based on the preset data 100 and the first information (used to indicate the position of the first region 11 in the image to be processed), thereby obtaining the preset intermediate result D of the second region 12.

[0516] Then, as shown in Figure 7b(2), the client 200 can stitch the preset intermediate result D of the second region 12 and the first intermediate result B of the decoded first region 11 according to the position of the first region 11 in the image to be processed 300, that is, fill the original position where the preset intermediate result A was deleted, thereby obtaining the reconstruction intermediate result 500 of the image to be processed 300. For example, reconstruction mode 1 can instruct the preset intermediate result D of the region outside the first region 11 (e.g., the second region 12) to be stitched with the first intermediate result B of the first region 11. As shown in Figure 7b(2), the reconstruction intermediate result 500 includes the preset intermediate result D of the second region 12 and the first intermediate result B of the first region 11.

[0517] In this embodiment, when obtaining the graphics calculation result of the image to be processed, the client 200 can obtain the preset intermediate result of the region other than the first region (e.g., the second region) in the preset data of the image to be processed, and "concatenate" the first intermediate result of the decoded first region with the preset intermediate result of the second region to obtain the reconstruction intermediate result of the complete region of the image to be processed, including the first region and the second region, thereby improving the reconstruction flexibility of the client 200 in the reconstruction intermediate result of the image to be processed. Then, the client 200 can use this reconstruction intermediate result to obtain the graphics calculation result of the image to be processed.

[0518] Reconstruction method 2 (also known as "overlay" method) corresponds to the above-mentioned coding method 2.1.

[0519] In this reconstruction method 2, the decoding data corresponding to the first intermediate result is used to indicate the first difference data. When the client 200 obtains the graphic calculation result of the image to be processed based on the preset data, the first information and the decoding data corresponding to the first intermediate result, it can obtain the graphic calculation result of the image to be processed based on the preset data, the first information and the first difference data. The first difference data is the difference between the first intermediate result of the image to be processed and the preset intermediate result of the first region in the image to be processed.

[0520] In some embodiments, the client 200 may obtain a preset intermediate result of the image to be processed based on the preset data.

[0521] For example, if the preset data is a preset intermediate result of a reference image corresponding to the image to be processed, then the preset intermediate result of the image to be processed can be determined (e.g., by interpolation) based on the preset intermediate result of the reference image.

[0522] In some embodiments, the preset data may also be a preset intermediate result of the image to be processed, so that the client 200 can directly reuse the preset data as a preset intermediate result of the image to be processed.

[0523] In some embodiments, the client 200 may obtain the reconstruction intermediate result of the image to be processed according to the reconstruction method 1 based on the preset intermediate result of the image to be processed, the first information, and the first difference data.

[0524] The following example, Figure 7c, illustrates the process of obtaining the intermediate reconstruction result 500 of the image to be processed according to reconstruction method 2.

[0525] The reconstruction method 2 shown in Figure 7c corresponds to the encoding method 2.1 shown in Figures 5b(1) and 5b(2).

[0526] Referring to Figure 5a and then to Figure 7c, the client 200 can decode the bitstream to obtain the difference C1 of the first region 11 (an example of the first difference data); the client 200 can also obtain the preset intermediate result of the image to be processed (here, preset data 100); furthermore, the client 200 can use the first information to determine the position of the first region 11. Thus, the client 200 can use the position of the first region 11 in the image to be processed 300 to find the preset intermediate result A of the first region 11 in the preset data 100. Then, as shown in Figure 7c, the client 200 can perform superposition processing on the preset data 100 (specifically, the preset intermediate result of the image to be processed 300) and the difference C1 according to reconstruction method 2. During this superposition process, the client 200 can add the preset intermediate result A of the first region 11 and the difference C1 of the first region 11 according to the data at the same position, thereby obtaining the first intermediate result B of the first region 11. For example, reconstruction method 2 can instruct the superposition of the preset intermediate result A of the first region 11 and the difference C1 of the first region 11. Finally, through the above overlay process, the intermediate reconstruction result 500 of the image 300 to be processed, as shown in Figure 7c, can be obtained. The intermediate reconstruction result 500 includes the preset intermediate result D of the second region 12 and the first intermediate result B of the first region 11.

[0527] In this embodiment, the client 200 decodes the bitstream to obtain the first difference data corresponding to the first intermediate result. Compared to the data volume of the first intermediate result, the difference data has a smaller data volume, thereby improving decoding efficiency. Furthermore, when the decoded data obtained from decoding the bitstream indicates the first difference data corresponding to the first intermediate result, the client 200 can obtain a preset intermediate result (e.g., preset data for the entire scene) of the image to be processed based on preset data of the image to be processed. This allows the client to reuse the preset intermediate result of the entire scene of the image to be processed and the decoded first difference data to reconstruct the reconstruction intermediate result of the entire scene of the image to be processed. In this process, the client 200 does not need to perform graphic calculations on regions other than the first region (e.g., the second region) in the image to be processed. Instead, it can directly reuse the preset data of the image to be processed and combine it with the first intermediate result of the first region calculated in real time by the server 100 to reconstruct the intermediate result of the image to be processed. This avoids the repeated graphic calculations on the regions (e.g., the second region) where the scene has not changed by the end side and the cloud side (e.g., the server 100). This can improve the reconstruction efficiency of the intermediate result of the image to be processed and improve the efficiency of obtaining the graphic calculation result of the image to be processed.

[0528] Reconstruction method 3 corresponds to the above-mentioned encoding method 3.

[0529] Reconstruction method 3 is a combination of reconstruction method 1 and reconstruction method 2. In reconstruction method 3, a portion of the first region of the image to be processed is reconstructed using the overlay scheme mentioned in reconstruction method 1, while another portion of the first region is reconstructed using the stitching scheme mentioned in reconstruction method 2. The reconstruction of the second region reuses the preset data (e.g., preset intermediate results) obtained by the server 200 in advance by performing graphic calculations on the second region.

[0530] In this reconstruction method 3, the decoding data corresponding to the first intermediate result may include third information, a second intermediate result calculated from the first graph for a portion of the first region, and second difference data corresponding to another portion of the first region. After decoding the bitstream, the client 200 can obtain the second difference data, the second intermediate result, and the third information. The second difference data is the difference between the third intermediate result calculated from the first graph for another portion of the first region and the preset intermediate result of the other portion. The second intermediate result is the intermediate result calculated from the first graph for a portion of the first region. The third information is used to indicate the position of a portion of the first region and / or the other portion in the image to be processed.

[0531] Referring to Figure 5c(2), the client 200 decodes the bitstream and obtains the difference C2 of region 11c in the first region 11 (an example of the second difference data mentioned above), the intermediate result B2 of region 11d in the first region 11 (an example of the second intermediate result), and third information used to indicate the position of region 11c and / or region 11d in the image to be processed.

[0532] In this reconstruction method 3, when the client 200 obtains the graphic calculation result of the image to be processed based on the preset data, the first information and the decoding data corresponding to the first intermediate result, it can obtain the preset intermediate result of the third region in the image to be processed, excluding the first part, based on the preset data and the third information; then, it can obtain the graphic calculation result of the image to be processed based on the preset intermediate result of the third region, the second difference data and the second intermediate result.

[0533] For example, the third region may include the second region and the other portion of the first region described above.

[0534] For example, the client 200 can obtain a preset intermediate result of a third region in the image to be processed, excluding the aforementioned portion, based on the preset data and the third information; and process the preset intermediate result of the third region, the second difference data, and the second intermediate result according to the reconstruction method 3 to obtain a reconstruction intermediate result of the image to be processed; then, based on the reconstruction intermediate result, the graphic calculation result of the image to be processed can be obtained.

[0535] The following example, Figure 7d, illustrates the process of obtaining the intermediate reconstruction result 500 of the image to be processed according to reconstruction method 3.

[0536] The reconstruction method 3 shown in Figure 7d corresponds to the encoding method 3 shown in Figure 5c.

[0537] Referring to Figure 5a and Figure 7d, as shown in Figure 7d(1), in this embodiment, the preset data 100 corresponding to the image to be processed 300 is the same as the preset intermediate result of the image to be processed 300. The client 200 can determine the position of region 11d in the image to be processed 300 based on the third information (used to indicate the position of region 11c and / or region 11d in the image to be processed 300), thereby determining the preset intermediate result (specifically, preset intermediate result A2) of the corresponding region 11d in the preset data 100, so as to delete the preset intermediate result A2 in the preset data 100 (for example, set it to 0), thereby obtaining the preset intermediate result of the third region (specifically, the preset intermediate result D of the second region 12 and the preset intermediate result A1 of region 11c indicated by the arrow in Figure 7d(1)).

[0538] Next, as shown in Figure 7d(2), reconstruction method 3 can be used to stitch together the preset intermediate results of the third region (specifically, preset intermediate results D and preset intermediate results A1) and the intermediate result B2 of the decoded region 11d (an example of the second intermediate result mentioned above). Specifically, according to the position of region 11d in the image 300 to be processed, intermediate result B2 is stitched to the region 11d missing in the preset intermediate results of the third region (e.g., region 11d set to 0) as shown by the arrow. And according to reconstruction method 3, as shown by the arrow, the difference C2 between the preset intermediate result A1 of region 11c in the preset intermediate results of the third region and the decoded region 1c is superimposed (to obtain the intermediate result B1 shown in Figure 5c). For example, reconstruction method 3 can instruct "superimpose" on region 11c and "stitch" on region 11d.

[0539] So, through the above "stitching" and "overlay" operations, the reconstruction intermediate result 500 of the image 300 to be processed, as shown in Figure 7d(2), can be obtained. As shown in Figure 7b(2), the reconstruction intermediate result 500 includes the preset intermediate result D of the second region 12 and the first intermediate result B of the first region 11 (specifically, the intermediate result B1 of region 11c and the intermediate result B2 of region 11d).

[0540] In this embodiment, the client 200 decodes the bitstream to obtain the difference of another part in the first region, as well as an intermediate result of a part of the first region (the intermediate result obtained by the server 100 through the first graphic calculation), and information indicating the position of the aforementioned part or the other part in the first region. This allows for the reconstruction of the intermediate results of the entire scene of the image to be processed using the decoded data. This increases the reconstruction complexity and ensures secure data decoding. Furthermore, the reconstruction method used by the client 200 to reconstruct the intermediate results of the image to be processed can be dynamically adjusted based on the preset intermediate results of the image to be processed and the first intermediate results of the first region of the image to be processed. This is equivalent to transforming the data of the first intermediate results based on the preset data corresponding to the image to be processed, making the transformed first intermediate results more suitable for encoding and decoding, and improving decoding efficiency.

[0541] Furthermore, in the embodiments of this application, when obtaining the intermediate reconstruction result of the image to be processed, different decoding schemes (such as "stitching" and "overlay") can be used for the two parts in the first region. Combined with the preset data of the image to be processed, the intermediate reconstruction result of the image to be processed can be decoded. This can increase the decoding complexity of the intermediate reconstruction result and increase the difficulty of cracking the first intermediate result. In this way, in the case of the first intermediate result of the image to be processed which is sensitive to transmission, the secure transmission of the first intermediate result can be ensured.

[0542] It should be understood that Figures 7c to 7d above are schematic diagrams of scenarios under different reconstruction methods. They can be related to Figures 5a to 5c. The same reference numerals and illustrations in Figures 5a to 5c and Figures 7c to 7d have the same or similar meanings, which will not be elaborated here.

[0543] In one embodiment, when the client 200 obtains the preset data corresponding to the image to be processed, it can decode the bitstream to obtain the indication information corresponding to the image to be processed. The indication information is used to indicate the preset data corresponding to the image to be processed. Then, the client 200 can obtain the preset data corresponding to the image to be processed according to the indication information.

[0544] The description of this instruction information can be found in the relevant description of S401e in process 400 shown in Figure 4 of the image processing method, which will not be repeated here.

[0545] In this embodiment, since the client 200 can only decode the first intermediate result of the first region in the bitstream, and cannot decode the intermediate result obtained by the server 200 through graphics calculation of the regions other than the first region in the image to be processed, the decoding end (e.g., the client 200) can reuse the preset data of the current frame or adjacent frames and combine it with the first intermediate result to reconstruct the aforementioned reconstruction intermediate result of the image to be processed in order to reconstruct the reconstruction intermediate result of the entire scene of the current frame. Therefore, the client 200 can decode the indication information used to indicate the preset data corresponding to the image to be processed from the bitstream, so that the client 200 can use the indication information to find the preset data corresponding to the image to be processed, and combine it with the first intermediate result to reconstruct the reconstruction intermediate result of the image to be processed. Furthermore, when reconstructing the intermediate results of the current frame, if the preset data referenced is different from the preset data referenced by the previously decoded frame (e.g., the previous frame), the client 200 can decode the indication information from the bitstream to obtain the preset data for reconstructing the intermediate results of the current frame, thus ensuring the accurate decoding of the intermediate results of the current frame by the client 200.

[0546] In some embodiments, the indication information may include the address for obtaining preset data corresponding to the image to be processed.

[0547] For example, server 100 can store preset data corresponding to the image to be processed in a remote server, and encode the access address of the preset data on the remote server (an example of obtaining the address) as indication information into the bitstream for synchronization with client 200. In this way, client 200 can decode the access address from the bitstream and access the remote server according to the access address to obtain the preset data corresponding to the image to be processed.

[0548] In this embodiment of the application, if the client 200 does not store the preset data corresponding to the image to be processed locally, the client 200 can decode the acquisition address of the preset data corresponding to the image to be processed from the bitstream, so as to access the preset data according to the acquisition address, and use it to decode the intermediate reconstruction result of the image to be processed.

[0549] The following uses several specific embodiments to describe in detail the technical solution of the above image processing method and the embodiment of the image processing method in a cloud-edge collaborative interaction.

[0550] The image processing methods and some technical terms used in these methods will be described with more specific names in the following specific embodiments.

[0551] The "first intermediate result" in the above embodiments is referred to as "real-time updated data" in the following specific embodiments.

[0552] The “preset intermediate results” mentioned in the above embodiments are referred to as “preset data” in the following specific embodiments.

[0553] The "second information" in the above embodiments is referred to as "division basis information" in the following specific embodiments.

[0554] The "first information" on the encoding side in the above embodiments is referred to as "location classification information" in the following specific embodiments. The "location classification information" and the "reconstruction method" in the above embodiments are uniformly summarized as "reconstruction indication information" in the following specific embodiments. That is, the "reconstruction indication information" may include the "location classification information" and optionally may also include the "reconstruction method" of the image to be processed.

[0555] In the above embodiments, when the decoding side obtains the intermediate reconstruction results of the image to be processed, the preset data referenced (such as the preset intermediate results of the first region and / or the preset intermediate results of the second region) is uniformly named "reused data" in the following specific embodiments.

[0556] The “instruction information” in the above embodiments is described as “reusable data synchronization information” in the following specific embodiments.

[0557] Example 1

[0558] Figure 8 is a schematic diagram of the edge-cloud encoding and decoding architecture according to an embodiment of this application. Taking DDGI rendering as an example, the edge-cloud collaborative rendering process may include the following steps:

[0559] 1. Pre-calculation of the manufacturing state:

[0560] Before actual rendering, the cloud platform pre-calculates multiple high-precision intermediate rendering texture maps for scenes unaffected by player interaction, i.e., preset data F. PreComp (An example of the preset data in the above embodiments) The rendering intermediate data map pre-calculated from multiple images to be processed may include the rendering intermediate data map pre-calculated for each probe in each image to be processed (e.g., each probe in Figure 5a). In this embodiment, the rendering intermediate data refers to DDGI scene visibility (distance) data. In addition, in other embodiments, the rendering intermediate data may also be DDGI irradiance map data, etc.

[0561] In some embodiments, the preset data F PreComp For example, the steps for generating DDGI scene visibility data are as follows:

[0562] a) Unfold each probe from the sphere into a two-dimensional plane;

[0563] a.1 Map the probe spherical surface onto an octahedron;

[0564] a.2 Unfold the octahedron into two-dimensional data;

[0565] b) Emit light rays: Starting from each probe, N light rays are uniformly emitted from the spherical surface, and scene information is recorded. Specifically, the scene information includes the coordinates of the intersection points of each light ray with objects in the scene, and the normal information of the object surfaces at the intersection points. In other embodiments, when the reflected light rays are emitted from the probe, they can also be emitted non-uniformly from the spherical surface of the probe, emitting N light rays, where N is a positive integer and is not specifically limited.

[0566] c) Visibility fusion: The scene information obtained in b) is filled into the corresponding pixels in the 2D plane obtained in a).

[0567] r cur This indicates the pixel value written to the corresponding pixel position; For each two-dimensional plane pixel, the corresponding probe surface normal vector is used; for... Summate the scene information recorded by all emitted rays within the hemisphere of the normal vector. i Θ is the distance between the point where the emitted rays intersect in the scene and the point on the probe surface. i To emit light and The included angle, s is an empirical parameter.

[0568] d) Fusion with historical frame information: The value of the current frame obtained in c) is fused with the value of the historical frame to obtain better performance: as the final pixel value written to the corresponding pixel position to form a two-dimensional texture.

[0569] Here, α is an adjustable parameter; the larger the value, the more dependent it is on historical frame information.

[0570] 2. Preset data synchronization between the end and the cloud:

[0571] The preset data F obtained in step 1 Precomp (This is the preset data (also called preset results) of the DDGI characteristics of all probes in this 3D scene) synchronized to the end side.

[0572] In this embodiment, the preset data can be included as a scene asset in an application (e.g., a game application, which is not specifically limited). The client can obtain the preset data of each frame of the game application by downloading the application. Alternatively, the cloud can send the application's preset data to the client via a communication network when the connection between the client and cloud is established for the first time. Furthermore, the cloud can also send the preset data to the client during application runtime, before the application uses the preset data for the first time, or other methods may also be used, which are not limited here.

[0573] 3. Update the game logic state on the client side:

[0574] In runtime, the game logic module can determine whether the current frame needs to update the intermediate rendering data for lighting characteristics like DDGI. If an update is needed, a corresponding notification is generated to inform the client-side coordination module that the current frame requires updating the DDGI rendering intermediate data.

[0575] DDGI is a renderable lighting characteristic. In other embodiments, the type F of the lighting characteristic that needs to be updated in the current frame is not limited to DDGI; it can also be spherical reflection, direct lighting, reflected lighting, shadows, indirect lighting, ambient occlusion, etc. There are no restrictions here. Because the type F of the lighting characteristic is different, the type F' of the intermediate rendering data for that lighting characteristic is also different. Therefore, type F' can be any type of intermediate rendering data supported by edge-cloud collaborative rendering, such as DDGI data, reflection probe data, shadow data, etc.

[0576] 4. The edge-side collaboration module generates intermediate rendering data requests:

[0577] a) The collaboration module receives the DDGI update instruction and game logic data generated in step 3;

[0578] The game logic information may include game state information, which may include, but is not limited to, game time, character position, and game perspective.

[0579] b) The collaboration module generates a DDGI rendering intermediate data request, wherein the DDGI rendering intermediate data request may include a DDGI rendering intermediate data request identifier (an example of type F') and indication information for indicating the region division for cloud-side data reuse and real-time data updates;

[0580] The indication information in b) may include "split basis information (info_Split)" determined based on type F' or type F.

[0581] For example, the information used for classification may include game status information, distance thresholds, etc.

[0582] For example, the client can use DDGI to select the game state data required by the cloud side for dividing the first and second regions from the game logic data, such as character position information P. In other embodiments, it may also include game time information T.

[0583] In addition, the indication information in b) may also include a distance threshold D generated by the endpoint based on its state (e.g., network status information, load information, etc.). threshold This distance threshold determines the location of the first region (e.g., the location and number of probes that need to be updated via DDGI) in the current frame.

[0584] For example, if the network on the edge is poor (or the load is high), the distance threshold is smaller to sacrifice the rendering effect of the current frame on the edge in order to ensure the real-time transmission of real-time updated data on the cloud side.

[0585] It should be understood that traditional end-to-end collaboration modules can also transmit game state information (such as game time information, character position information, game view information, etc.).

[0586] 5. The end-side transmission module 1a transmits the rendering intermediate data request from step 4 to the cloud side;

[0587] 6. The cloud side receives intermediate rendering data requests to update the game logic;

[0588] 7. The cloud-side synchronization strategy generation module uses the info_Split passed in the intermediate data request during rendering on the client side to generate reconstruction instruction information;

[0589] a) Calculate the character's position information P and the distance D between it and each probe in the current frame's 3D scene (the position of each probe is fixed):

[0590] Where P is the position coordinate of the character (e.g., the position coordinate of character 10 in the image 300 to be processed in Figure 5a), P probe (·) represents the probe coordinates, and idx represents the probe index.

[0591] b) Based on the distance D generated in a) and the received distance threshold D threshold The system separates reused data from real-time updated data and generates location classification information, specifically using the identifier flag(idx):

[0592] Among them, flag PreComp This indicates that probes in the current frame's 3D scene (e.g., each probe within the first region 11 in Figure 5a) can use preset data as intermediate rendering data. RTUpdate This indicates that the probe (e.g., each probe within the second region 12 in Figure 5a) needs to use real-time updated data as intermediate data for rendering.

[0593] The end-side reconstruction indication information may include the partition identifier flag(idx) generated in b), which is equivalent to the classification result of the probes, to indicate which probes in the current frame use preset data and which probes use real-time updated data, so as to realize the partitioning of the first region and the second region.

[0594] 8. The cloud-side multiplexed data synchronization module generates multiplexed data synchronization information:

[0595] a) Based on game time T, obtain preset data for several frames near game time T (referred to as "nearby frames" above), for example, these frames are frames whose display time is near the game time T of the current frame; and record the frame identifier information of each of these frames.

[0596] b) Calculate the interpolation coefficients t of the preset data for the i-th frame within several frames in a) according to formulas 5, 6, and 7. i , 1≤i≤N, where N is the number of frames of the preset data selected in a) (i.e., the number of frames of the above-mentioned several frames);

[0597] c) Combine the frame identifier information generated in a) with the interpolation coefficient t generated in b) i Information is used as synchronized data for reuse;

[0598] Optionally, d) based on the preset data of several frames obtained in a) and the interpolation coefficients t in b) i Perform interpolation to generate multiplexed data frames F reuse :

[0599] F PreComp (i) is the preset data of the i-th frame among the several frames selected in a) (the preset intermediate result of the entire scene of the i-th frame (e.g., the first and second regions mentioned above).

[0600] Thus, when encoding the real-time update data of the first region of the current frame, the multiplexed data frame of the current frame (obtained according to Formula 8) can be used to encode the real-time update data of the current frame; or, the multiplexed data frame of the current frame can also be used to encode the real-time update data of the first region of frames other than the current frame (such as the next frame or the previous frame) to improve encoding efficiency.

[0601] 9. The cloud-side rendering module calculates the DDGI for each probe in the first region of the current frame in real time to obtain the real-time updated data of the first region; the cloud-side encoding module encodes the real-time updated data to generate a bitstream;

[0602] 10. Cloud-side transmission module 2b synchronizes the reconstruction instruction information generated in step 7, the multiplexing data synchronization information generated in step 8, and the real-time update data generated in step 9 to the end side via a bitstream.

[0603] 11. The end-side transmission module 1b receives the reconstruction indication information, multiplexed data synchronization information, and real-time updated data bitstream transmitted from the cloud side;

[0604] 12. End-side strategy selection module:

[0605] a) Based on the game logic, determine whether to cache the currently requested intermediate rendering data (which is the intermediate rendering result of the entire scene in the current frame, such as any one of the reconstruction results 500 in Figures 7c to 7d above) locally, so as to reuse the data for subsequent frames (frames whose encoding order or display order is after the current frame).

[0606] The method for determining this can be: calculating the game time T and the cache time T of the previously cached reused data. pre The time interval between frames is used to determine whether to cache the intermediate reconstruction results of the current frame as reuse data for subsequent frames, according to Formula 9.

[0607] Among them, T threshold The threshold is set in advance.

[0608] Among them, the time interval (TT) in Formula 9 pre If the time interval is short, it means that the frame changes are small, so the previously cached reused data can continue to be reused by subsequent frames, and there is no need to cache the intermediate reconstruction results of this time as reused data; conversely, if the time interval is long, it means that the frame changes may be large, so the intermediate reconstruction results of this time can be cached as reused data.

[0609] Alternatively, the determination method could be to detect the difference between the intermediate reconstruction results of the current frame and the intermediate reconstruction results of the previous frame, and then determine whether to cache the intermediate reconstruction data of the current frame as reuse data for subsequent frames.

[0610] 13. The end-side multiplexed data acquisition module generates multiplexed data frames F based on the multiplexed data synchronization information transmitted from the cloud side. reuse :

[0611] a) Based on the frame identifier information (e.g., frame identifiers for several frames) in the multiplexed data synchronization information, read the preset data F of the corresponding frame from the local machine. PreComp (·);F PreComp (·) can be the preset data of the entire scene of the corresponding frame, or the preset data of the area of ​​the corresponding frame other than the first area. There is no restriction here.

[0612] b) Referring to Formula 8 above, based on the preset data of the selected frames (e.g., N frames) in a) and the interpolation coefficient t of the i-th frame in the N frames in the multiplexing data synchronization information. i Interpolation is performed to generate multiplexed data frames F. reuse (An example of a preset intermediate result of the image to be processed in the above embodiments);

[0613] 14. End-side decoding module, which decodes the bitstream of real-time updated data in the first region to obtain real-time updated data (an example of the first intermediate result in the above embodiment);

[0614] 15. The terminal-side rendering intermediate data reconstruction module, based on the terminal-side reconstruction instruction information and the multiplexed data frame F obtained in step 13, reconstructs the data. reuse Reconstruct the data using the real-time updated data obtained in step 14 to obtain the requested intermediate rendering data (specifically, the intermediate reconstruction result of the current frame):

[0615] a) The reconstruction instruction information may carry the reconstruction method, and the intermediate reconstruction result of the current frame may be obtained by reconstructing according to the reconstruction method (such as any one of the three reconstruction methods mentioned above: "stitching", "overlaying", and "stitching" + "overlaying").

[0616] b) Based on the flag generated in step 12 Cache This determines whether to perform a caching operation on the intermediate reconstruction results of the current frame.

[0617] 16. The edge rendering module uses the intermediate rendering data reconstructed in step 15 to perform subsequent simple rendering to generate the rendering result of the current frame (e.g., game screen) and display it on the edge.

[0618] In this embodiment, the end-side collaboration module can select indication information (such as character location information, game time information, distance threshold, etc.) that instructs the cloud side to perform region division of reused data and real-time updated data. This division information can be used by the cloud-side synchronization strategy generation module to perform region division of the requested rendering intermediate data, so as to divide the requested rendering intermediate data into a reused part and a real-time updated part.

[0619] Furthermore, in this embodiment, the cloud-side multiplexing data synchronization module can generate multiplexing data synchronization information based on the region division performed by the synchronization strategy generation module and using the preset data that has been synchronized to the end side in advance. In this way, the cloud side does not need to perform rendering calculations for the second region obtained by the region division, nor does it need to encode and transmit the intermediate rendering results of the second region obtained by the rendering calculation, thereby reducing the repeated calculations and bandwidth occupation on the cloud side.

[0620] In addition, in this embodiment of the application, the end-side multiplexing data acquisition module can obtain the reused part of the rendering intermediate data (full scene) of the current frame from the preset data synchronized from the cloud side in advance based on the multiplexing data synchronization information. In this way, the rendering intermediate data of the full scene of the current frame can be obtained even when the cloud side does not perform real-time rendering calculation on the second region.

[0621] Furthermore, in this embodiment of the application, the edge rendering intermediate data reconstruction module can reconstruct the rendering intermediate data of the current frame from the acquired reused data frame and the real-time updated data to obtain the requested rendering intermediate data, thus ensuring a complete response to the requested data.

[0622] Thus, in this embodiment, the cloud side can use the data sent by the end-side collaboration module to divide the rendering intermediate data requested by the end side into regions. The reusable parts (e.g., the rendering intermediate data of the second region) do not need to be rendered, encoded, or transmitted in real time. With the corresponding steps on the end side, this embodiment can save cloud side computing costs and the transmission bandwidth of rendering intermediate data.

[0623] Example 2

[0624] The accompanying drawings for this embodiment 2 are the same as those for Figure 8 in embodiment 1 above. The process of edge-cloud collaborative rendering in this embodiment 2 is mostly the same as that in embodiment 1 above. Only the differences will be described in detail below. For the similarities, please refer to the relevant description of embodiment 1. They will not be repeated here.

[0625] As shown in Figure 8 of this embodiment, the end-side collaboration module can still perform the functions of a traditional collaboration module to transmit game status information (such as game time information, character position information, game view information, etc.) and end-side network status information. However, the end-side collaboration module does not need to generate indication information (such as distance threshold) for instructing the cloud side to perform data reuse and real-time data update, as mentioned in step 4 of embodiment 1.

[0626] In this way, the cloud-side synchronization strategy generation module can select character position information P in the updated game logic (e.g., game state information) based on the type F' of the rendered intermediate data (e.g., DDGI) or the type F of the rendered lighting characteristics; and adaptively determine the distance threshold D based on the terminal-side network state information. threshold This generates edge reconstruction indication information (e.g., including location classification information). In this way, the cloud side can independently determine indication information (e.g., distance thresholds) for instructing the cloud side to perform data reuse and real-time data updates, ensuring that the first and second regions defined based on this indication information are adapted to the network state at the edge.

[0627] Example 3

[0628] The process of edge-cloud collaborative rendering in this embodiment 3 is mostly the same as that in embodiment 1 above. The differences will be explained in detail below. For the similarities, please refer to the relevant introduction of embodiment 1. They will not be repeated here.

[0629] The main difference between this embodiment 3 and the above-mentioned embodiments 1 and 2 is that:

[0630] Difference 1: Different application scenarios: This embodiment 3 is applied to the reflection probe scenario, while the above embodiments 1 and 2 are applied to the DDGI scenario.

[0631] Difference 2: The behavior of the end-side collaboration module is different: In this embodiment 3, the end-side collaboration module can realize the regional division of reused data and real-time updated data of the rendering intermediate data (e.g., the division of the first region and the second region); while in the above embodiments 1 and 2, the above-mentioned regional division operation is implemented by the cloud-side synchronization strategy generation module.

[0632] Difference 3: The cloud-side encoding module and the edge-side decoding module behave differently: In this embodiment 3, when encoding real-time updated data, the encoding module can use multiplexed data (e.g., multiplexed data frames) as reference frames to encode the real-time updated data; similarly, the edge-side decoding module also uses multiplexed data as reference frames to decode the bitstream of the real-time updated data. This improves encoding and decoding efficiency.

[0633] Regarding the aforementioned difference 1, the edge-cloud collaborative rendering process of reflection probes is a scene reflection rendering method based on probes. One or more reflection probes can be placed to represent light reflection in a 3D scene. A reflection probe can be viewed as a sphere with six faces. Figure 9a shows the intermediate rendering data (i.e., intermediate rendering result) of the six faces of a reflection probe, specifically the reflection probe textures for the six faces, i.e., six reflection probe textures, from left to right and top to bottom, labeled texture 1 to texture 6.

[0634] User interactions often result in users manipulating game characters, which in turn affects the 3D scene and consequently the reflection probe data. As shown in Figure 9a, when a character enters the detection range of a reflection probe, it affects three of the six reflection probe maps (texture 2, texture 4, and texture 6 in this case), thus requiring updates to the three reflection probe maps containing the character. However, for locations where the character is not occluded, the changes to their reflection probe maps (e.g., texture 1, texture 3, and texture 5) are usually unaffected by the character's interaction. Their texture data can be reused from pre-calculated intermediate rendering data after the production scene is built, without the need for real-time updates.

[0635] When using the cloud to update textures 2, 4, and 6 related to scene changes, the cloud can calculate updated reflection probe maps for each of these three textures separately and then replace all three textures with the updated reflection probe maps. Alternatively, the cloud can calculate updated local reflection probe maps for the areas near the character in each of these three textures to replace the reflection probe maps in local areas within these three textures. The areas in textures 2, 4, and 6 other than the area near the character will still use pre-calculated intermediate rendering data (such as reflection probe maps).

[0636] Figure 10 is a schematic diagram of the edge-cloud encoding and decoding architecture according to an embodiment of this application. Taking the rendering of spherical reflection (i.e., reflection probe) as an example, the edge-cloud collaborative rendering process may include the following steps:

[0637] 1. Pre-calculation of the manufacturing state:

[0638] Before actual rendering, the cloud side pre-calculates a high-precision intermediate rendering data texture for the scene unaffected by player interaction, as shown in Figure 9b. The cloud side pre-calculates 6 reflection probe textures for a frame of image to be processed. i Specifically, this includes the reflection probe map for each of the six faces of the reflection probe, and the reflection probe map f i This represents the reflection probe map of the i-th face. In this embodiment, the intermediate rendering data refers to the reflection probe map.

[0639] In some embodiments, the reflective probe map f i The generation steps are as follows:

[0640] Using the position P of the reflection probe as the viewpoint, render it once in each of the six directions: up, down, left, right, front, and back, to obtain the intermediate rendering results of the six faces, which are then stored on six textures.

[0641] f i =Render(P,i), formula 10;

[0642] Among them, f i This is the reflection probe texture for the i-th face of the reflection probe, where i is an element in the set {up, down, front, back, left, right}. Render() is the rendering calculation operation.

[0643] 2. Preset data synchronization between the end and the cloud:

[0644] The cloud side will take the preset data of each 3D scene obtained in step 1, specifically 6 reflection probe maps F for each 3D scene. i (A reflection probe texture of the i-th face of a reflection probe in a 3D scene) is synchronized to the end side. In this embodiment, the preset data can be pre-deployed on the storage server. Store On the client side, during real-time runtime, the server sends data to the client. Store Request preset data. Alternatively, the preset data can be synchronized between the end and cloud using the method described in Example 1, or other methods may be used, which are not limited here.

[0645] 3. Update the game logic state on the client side:

[0646] During runtime, the client-side game logic module determines whether the intermediate rendering data of the reflection probe needs to be updated in the current frame, based on existing methods. If an update is required, a corresponding notification is generated to inform the client-side collaboration module that the intermediate rendering data of the reflection probe needs to be updated in the current frame.

[0647] 4. The edge-side collaboration module generates intermediate rendering data requests:

[0648] a) The collaboration module receives the reflection probe update indication and game logic data generated in step 3;

[0649] b) The identifier for the intermediate rendering data request of the reflection probe generated by the collaborative module (an example of type F');

[0650] Regarding the aforementioned difference 2, and differing from step 4 in embodiment 1, step 4 of this embodiment 3 further includes:

[0651] c) The collaboration module automatically determines the location indication information A that needs to be updated in real time on the cloud side based on the role position and the reflection probe position P. i (x,y) (represents the pixel (x,y) on the i-th face of the reflective probe):

[0652] Among them, the reflection probe position P is a fixed position in the scene, and the character position and the reflection probe position can be obtained from the game logic module.

[0653] Among them, i={up, down, front, back, left, right}. F i Let C be the i-th face of the reflective probe located at position P. i For the projection of the character onto the i-th surface of the reflection probe, Pixel x (·) represents the data pixels of the reflection probe, for example, Pixel. Fi (x,y) represents the pixel with coordinates (x,y) on the i-th surface of the reflective probe. Ci (x,y) represents the pixel position (x,y) of the character's projection on the i-th face of the reflective probe.

[0654] In step c), the coordination module performs a judgment according to Formula 11 on each face (denoted as i) of the reflection probe located at position P to determine whether the character has a projection on the i-th face of the reflection probe. If there is a projection of the character, i.e., This means that the character affects the reflection map of the pixel at coordinates (x, y) on the i-th face of the reflection probe. Therefore, the pixel at coordinates (x, y) on the i-th face needs to have its reflection map updated in real-time; conversely, if... This means that the pixel at coordinates (x, y) of the i-th face can reuse the pre-computed reflection map, without the cloud side needing to perform real-time calculation of the reflection map for the pixel at coordinates (x, y) of the i-th face.

[0655] In this way, according to Formula 11, the six faces of the reflective probe and the pixels inside them can be divided into pixels in the face that need to be updated in real time (an example of the first region) and pixels in the face that reuse preset data (an example of the second region) to achieve classification of the two regions in the current frame.

[0656] In the above embodiments, the granularity of real-time updates is the pixel granularity within the surface of the reflection probe, which is used as an example to illustrate the division and classification of the two regions. In other embodiments, the granularity of real-time updates can also be the surface granularity of the reflection probe. In this way, the smallest unit of each region in the first and second regions is the surface of the reflection probe, rather than the pixel within the surface of the reflection probe.

[0657] In step c) above, taking the position of the role and the position of the reflection probe on the end side as an example, it is explained how to divide the first region and the second region. In other embodiments, the position and size of the first region and the second region can be dynamically adjusted according to the network status and computing power of the end side and / or the cloud side, etc., without limitation here.

[0658] Formula 11 above identifies which locations in the current frame's 3D scene require real-time updates of the reflection maps (A). i (x,y)=Update), which positions (A) i (x,y)=Reuse) uses a pre-computed reflection map.

[0659] In some embodiments, A i (x,y) can be mask information for the position (x,y).

[0660] d) The collaborative module generates a rendering intermediate data request for the reflection probe. This rendering intermediate data request may include the identifier of the rendering intermediate data request for the reflection probe generated in step b). Unlike Embodiments 1 and 2 above, this rendering intermediate data request may also include the location indication information A that needs to be updated in real time on the cloud side, generated in step c). i (x,y), thus the location of the first region related to scene changes can be determined by the edge, and the determined location information (such as an example of the first information in the above image processing method) can be transmitted to the cloud side.

[0661] 7. Cloud-side synchronization strategy generation module:

[0662] a) Based on the location indication information A uploaded from the terminal side i (x,y) generates the region masks for the reused data portion and the real-time updated data portion of the requested intermediate rendering data:

[0663] Among them, mask i (x,y) is the pixel identifier of the i-th face of the reflective probe, where i = {up,down,front,back,left,right}. PreComp This indicates that the coordinates (x, y) of the i-th face can use preset data as intermediate rendering data. (flag) RTUpdate This indicates that the coordinates (x, y) of the i-th face need to be updated in real time.

[0664] In this embodiment, the cloud side divides the current frame into a first region and a second region based on the location indication information uploaded by the endpoint. In other embodiments, the cloud side can further adjust the location indication information uploaded by the endpoint to divide the current frame into a first region and a second region according to the adjusted location indication information. For example, the cloud side can further combine scene information, the cloud side's network status, and the communication link information between the endpoint and the cloud to further adjust the location indication information uploaded by the endpoint.

[0665] b) Record the region mask generated in step a) in the reconstruction instruction information. i (x,y) (An example of location classification information).

[0666] 8. The cloud-side multiplexed data synchronization module generates multiplexed data synchronization information:

[0667] For the mask in the reflection probe in step 7 i (x,y)=flag PreComp Execute step 8 for the pixel position, and for the mask inside the reflection probe in step 7. i (x,y)=flag RTUpdate The pixel position is then calculated in real time by the cloud-side rendering module for the reflection probe mapping.

[0668] a) Based on the game time T and the reflection probe position P, select several frames of preset data; record the frame identifier information of these frames; and record the access address (e.g., Server) of the storage service server that stores the preset data (reflection probe texture) for these frames. Store IP information), and optionally interpolation coefficients.

[0669] In some embodiments, the plurality of frames may be one or more frames whose display time is near the game time T (e.g., the “nearby frames” mentioned above), and the frame identifier information mentioned above is the frame identifier of each frame in the near frame.

[0670] In some embodiments, there may be multiple reflection probes in the three-dimensional scene of the current frame. Then, the multiple frames may be at least one reflection probe in the current frame located near position P, and the above frame identifier information is the identifier of the at least one reflection probe.

[0671] In some embodiments, in step c) of step 4 above, if the granularity of the location classification is the pixel granularity within the plane of the reflective probe, then the frame identifier information recorded in step a) can also be the pixel of the corresponding plane in the reflective probe located at position P (this pixel is a mask). i (x,y)=flag RTUpdate The identifier of the pixel on the i-th side of the reflection probe.

[0672] b) Generate synchronization information for the multiplexed data, including the frame identifier information generated in step a) and the Server... Store The IP information may optionally include interpolation coefficients.

[0673] 9. The cloud-side rendering module performs real-time calculation of the reflection probe map for each pixel in the first region of the reflection probe at position P in the current frame to obtain the real-time updated data of the first region; the cloud-side encoding module encodes the real-time updated data to generate a bitstream.

[0674] Regarding the aforementioned difference 3, and differing from Embodiments 1 and 2, in this embodiment, a data frame can be reused as a reference frame to encode the real-time updated data. Thus, when a scene change occurs in the current frame, compared to intra-frame encoding of the real-time updated data of the current frame or using the reconstruction intermediate result of the previous frame as a reference frame to encode the real-time updated data, this embodiment reduces the amount of encoded data and improves encoding efficiency by referencing the reused data frame of the current frame to encode the real-time updated data of the first region of the current frame.

[0675] The method for obtaining multiplexed data frames can be referred to the relevant description in Formula 8 of Embodiment 1, which will not be repeated here.

[0676] Step 12, End-side strategy selection module:

[0677] a) Based on the game logic, determine whether to upload the currently requested intermediate rendering data to the storage server. Store This allows for the sharing and reuse of data with other endpoints, saving cloud-side computing costs. The determination method is the same as step 12 in Example 1, and will not be repeated here.

[0678] Unlike Embodiments 1 and 2 above, in this Embodiment 3, the intermediate rendering data of the entire scene of the current frame can be uploaded to the storage server. This allows the client 200 to reuse the intermediate rendering data (also known as the reconstruction intermediate result) of the current frame, and it can also be reused by other users' clients.

[0679] Step 13: The end-side multiplexed data acquisition module acquires multiplexed data frame F based on the multiplexed data synchronization information transmitted from the cloud side. reuse :

[0680] a) Based on the frame identifier information in the multiplexed data synchronization information and the Server Store IP information, sent to the server Store Request to reuse data frame F reuse .

[0681] Unlike Embodiments 1 and 2 above, the multiplexed data frames in this embodiment of the application may not be directly generated by the end side, but may be downloaded from the storage server. The storage server may generate multiplexed data frames based on frame identifier information and corresponding preset data. The specific generation principle is the same as that of the end side generating multiplexed data frames in Embodiment 1, and will not be repeated here.

[0682] Step 14: The end-side decoding module decodes the bitstream of the real-time updated data using multiplexed data frames as reference frames to obtain the real-time updated data.

[0683] Regarding the above difference 3, unlike the above embodiments 1 and 2, in this embodiment, the data frame can be reused as the reference frame to realize the real-time updating of the data of the current frame, so as to improve the decoding efficiency when the scene changes in the current frame.

[0684] 15. a) to c) Same as Example 1

[0685] Step 15 may further include:

[0686] d) Based on the flag generated in step 12 Cache To determine whether to send the reconstructed intermediate rendering data to the server. Store Upload;

[0687] In this embodiment, the endpoint can divide the current frame into a first region related to scene changes and a second region unrelated to scene changes, thereby sending indication information indicating the position of at least one region in the first and second regions within the current frame to the cloud. The cloud can flexibly choose whether to use the regions divided by the endpoint for calculating real-time update data for local areas, depending on the application scenario. Furthermore, for encoding and decoding real-time update data, this embodiment can use pre-synchronized multiplexed data frames as reference frames to encode the real-time update data. If a scene change occurs in the current frame (e.g., frame N), encoding the real-time update data of frame N using intra-frame encoding or referencing data from the previous scene in frame N-1 results in a large difference in the calculated data. However, this embodiment can refer to the multiplexed data of the scene corresponding to the current frame to calculate the difference data, reducing the amount of difference data and improving encoding / decoding efficiency.

[0688] Example 4

[0689] The process of edge-cloud collaborative rendering in this embodiment 4 is mostly the same as that in the above embodiment 3. The differences will be explained in detail below. For the similarities, please refer to the relevant introduction of embodiment 1. They will not be repeated here.

[0690] Figure 11 is a schematic diagram of the end-to-cloud encoding and decoding architecture of this application embodiment. Most of the content in Figure 11 is the same as that in Figure 10. Comparing Figure 10 and referring to Figure 11, the main difference between this embodiment 4 and the above embodiment 3 is as follows:

[0691] Difference 1: The endpoint can generate multiplexed data synchronization information, so the cloud side does not need to generate and send multiplexed data synchronization information to the endpoint. The cloud-side multiplexed data synchronization module only obtains multiplexed data frames based on the location indication information uploaded by the endpoint, and uses them as reference frames for encoding real-time updated data, without needing to generate multiplexed data synchronization information to synchronize to the endpoint.

[0692] Difference 2: The edge device can classify the first and second regions based on its own determined position indication information, and can also generate the reconstruction method of the rendering intermediate data of the current frame (the whole scene). Based on this, the edge device can generate reconstruction indication information on its own, so the cloud side does not need to generate reconstruction indication information and does not need to send reconstruction indication information to the edge device.

[0693] Thus, in this embodiment of the application, the cloud side does not need to transmit synchronization information and reconstruction instruction information of multiplexed data to the end side, but only needs to transmit real-time updated data, thereby saving cloud side computing costs and transmission bandwidth and improving encoding and decoding efficiency.

[0694] Figure 12 is a schematic diagram of the structure of the image processing device 700 according to an embodiment of this application. As shown in Figure 12, the image processing device 700 of this embodiment can be applied to the server in the above embodiments. The image processing device 700 may include: an acquisition module 701, used to acquire a first region in the image to be processed, the first region being a region related to scene changes; a graphics calculation module 702, used to perform graphics calculation on the first region to obtain a first intermediate result of the image to be processed; and an encoding module 703, used to encode the first intermediate result to obtain an encoded bitstream.

[0695] In one embodiment, the encoding module 703 is further configured to encode first information into the bitstream, the first information being used to indicate the position of the first region in the image to be processed.

[0696] In one embodiment, the acquisition module 701 is specifically used to: acquire second information, which is used to divide the image to be processed into regions; and obtain a first region in the image to be processed based on the second information.

[0697] In one embodiment, the image processing apparatus 700 operates on a first device, and the second information includes at least one of the following: scene information corresponding to the image to be processed; one or more of the following: network status information, computing power, power consumption, and thermal threshold of the first device and / or the second device; and communication link information between the first device and the second device. The second device communicates with the first device.

[0698] In one embodiment, the encoding module 703 is specifically used to: obtain the encoding method of the first intermediate result; and encode the first intermediate result based on the above encoding method to obtain a bitstream.

[0699] In one embodiment, the acquisition module 701 is further configured to acquire the reconstruction method of the image to be processed, which corresponds to the above-described encoding method; the encoding module 703 is further configured to encode the above-described reconstruction method into the bitstream.

[0700] In one embodiment, the encoding module 703 is specifically used to: obtain a reference frame, the reference frame being an intermediate result of a reference image corresponding to the image to be processed; and encode the first intermediate result based on the reference frame.

[0701] In one embodiment, the encoding module 703 is specifically used to: acquire preset data corresponding to the image to be processed, the preset data including a first preset intermediate result corresponding to a first region; and encode the first intermediate result according to the preset data.

[0702] In one embodiment, the preset data is a preset intermediate result of the image to be processed, or the preset data is a preset intermediate result of a reference image corresponding to the image to be processed.

[0703] In one embodiment, the encoding module 703 is specifically used to: obtain a first preset intermediate result based on preset data; obtain a first difference data based on the first intermediate result and the first preset intermediate result; and encode the first difference data.

[0704] In one embodiment, the first intermediate result includes a second intermediate result obtained by graphical calculation of a portion of the first region and a third intermediate result obtained by graphical calculation of another portion of the first region. The encoding module 703 is specifically used to: obtain a preset intermediate result of another portion of the first region in the image to be processed based on preset data; obtain second difference data based on the third intermediate result and the preset intermediate result of the other portion; and encode the second difference data, the second intermediate result, and the third information, wherein the third information is used to indicate the position of a portion and / or another portion of the first region in the image to be processed.

[0705] In one embodiment, the acquisition module 701 is further configured to acquire indication information corresponding to the image to be processed, the indication information being used to indicate preset data corresponding to the image to be processed; the encoding module 703 is further configured to encode the indication information into the bitstream.

[0706] In one embodiment, the indication information includes the address for obtaining preset data corresponding to the image to be processed.

[0707] In one embodiment, the acquisition module 701 is further configured to obtain a second region in the image to be processed based on the second information, wherein the scene change degree of the second region is lower than that of the first region; the encoding module 703 is further configured to encode the fourth information into the bitstream, wherein the fourth information is used to indicate the position of the second region in the image to be processed.

[0708] The image processing apparatus 700 of this embodiment can be used to execute the technical solutions of the various method embodiments executed by the encoding end or the server end described above. The implementation principle and technical effect are similar, and will not be repeated here.

[0709] Figure 13 is a schematic diagram of the structure of the image processing device 800 according to an embodiment of this application. As shown in Figure 13, the image processing device 800 of this embodiment can be applied to the client in the above embodiments. The image processing device 800 may include: a decoding module 801, used to decode the bitstream to obtain decoded data corresponding to a first intermediate result of the image to be processed, wherein the first intermediate result is obtained by performing a first graphic calculation on a first region in the image to be processed, and the first region is a region related to scene changes; and an acquisition module 802, used to obtain the graphic calculation result of the image to be processed based on the decoded data corresponding to the first intermediate result.

[0710] In one embodiment, the acquisition module 802 is specifically used to: acquire first information, which indicates the position of the first region in the image to be processed; and obtain the graphic calculation result of the image to be processed based on the first information and the decoded data corresponding to the first intermediate result.

[0711] In one embodiment, the acquisition module 802 is specifically used to: acquire the reconstruction method of the image to be processed; and obtain the graphic calculation result of the image to be processed according to the reconstruction method based on the first information and the decoded data.

[0712] In one embodiment, the acquisition module 802 is specifically used to acquire the reconstruction method of the image to be processed based on the first information.

[0713] In one embodiment, the decoding module 801 is also used to obtain the reconstruction method of the image to be processed from the bitstream.

[0714] In one embodiment, the acquisition module 802 is specifically used to: acquire preset data corresponding to the image to be processed, the preset data including a first preset intermediate result corresponding to a first region in the image to be processed; and obtain the graphic calculation result of the image to be processed based on the preset data, the first information and the decoding data corresponding to the first intermediate result.

[0715] In one embodiment, the preset data is a preset intermediate result of the image to be processed, or the preset data is a preset intermediate result of a reference image corresponding to the image to be processed.

[0716] In one embodiment, the decoded data is used to indicate the first difference data. The acquisition module 802 is specifically used to: obtain the graphic calculation result of the image to be processed based on the preset data, the first information and the first difference data; the first difference data is the difference between the first intermediate result of the image to be processed and the preset intermediate result of the first region in the image to be processed.

[0717] In one embodiment, the decoded data is used to indicate a first intermediate result. The acquisition module 802 is specifically used to: obtain a preset intermediate result of a second region in the image to be processed based on preset data and first information, wherein the second region is a region in the image to be processed that is different from the first region; and obtain the graphic calculation result of the image to be processed based on the first intermediate result and the preset intermediate result of the second region.

[0718] In one embodiment, the decoded data includes third information, a second intermediate result obtained by first graphic calculation of a portion of the first region, and second difference data corresponding to another portion of the first region. The third information is used to indicate the position of a portion and / or another portion of the first region in the image to be processed. The second difference data is the difference between the third intermediate result obtained by first graphic calculation of the other portion and the preset intermediate result of the other portion. The acquisition module 802 is specifically used to: obtain the preset intermediate result of the third region in the image to be processed, excluding the other portion, based on the preset data and the third information; and obtain the graphic calculation result of the image to be processed based on the preset intermediate result of the third region, the second difference data, and the second intermediate result.

[0719] In one embodiment, the decoding module 801 is further configured to obtain indication information corresponding to the image to be processed from the bitstream, the indication information being used to indicate preset data corresponding to the image to be processed; the acquisition module 802 is specifically configured to obtain the preset data corresponding to the image to be processed based on the indication information.

[0720] In one embodiment, the indication information includes the address for obtaining preset data corresponding to the image to be processed.

[0721] In one embodiment, the decoding module 801 is also used to obtain first information from the bitstream.

[0722] In one embodiment, the decoding module 801 is specifically used to: acquire a reference frame, wherein the reference frame is reconstructed data of an intermediate result of a reference image corresponding to the image to be processed; and decode the bitstream based on the reference frame to obtain decoded data corresponding to the first intermediate result of the image to be processed.

[0723] In one embodiment, the acquisition module 802 is specifically used to: obtain reconstructed data of the intermediate result of the image to be processed based on the decoded data corresponding to the first intermediate result; and perform a second graphic calculation on the reconstructed data to obtain the graphic calculation result.

[0724] In one embodiment, the apparatus further includes a storage module (not shown) for storing the reconstructed data to a local or remote server.

[0725] In one embodiment, the acquisition module 802 is further configured to acquire second information, which is used to divide the image to be processed into regions; the image processing device 800 further includes: a first sending module (not shown), configured to send the second information.

[0726] In one embodiment, the image processing device 800 operates on a second device, and the second information includes at least one of the following: scene information corresponding to the image to be processed; one or more of the following: network status information, computing power, power consumption, and thermal threshold of the first device and / or the second device; and communication link information between the first device and the second device. The second device communicates with the first device.

[0727] In one embodiment, the image processing apparatus 800 further includes a second transmitting module (not shown) for transmitting the first information.

[0728] The apparatus in this embodiment can be used to execute the technical solutions of the method embodiments executed by the client or the decoding end described above. Its implementation principle and technical effects are similar, and will not be repeated here.

[0729] In implementation, each step of the above method embodiments can be completed by integrated logic circuits in the processor hardware or by instructions in software form. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware...

Claims

1. An image processing method, characterized in that, The method includes: Obtain a first region in the image to be processed, the first region being a region related to scene changes; Perform graphic calculations on the first region to obtain a first intermediate result of the image to be processed; The first intermediate result is encoded to obtain an encoded bitstream.

2. The method according to claim 1, characterized in that, The method further includes: The first information is encoded into the bitstream, and the first information is used to indicate the position of the first region in the image to be processed.

3. The method according to claim 1 or 2, characterized in that, The step of obtaining the first region in the image to be processed includes: Obtain second information, which is used to divide the image to be processed into regions; Based on the second information, the first region in the image to be processed is obtained.

4. The method according to claim 3, characterized in that, The method is performed by a first device, and the second information includes at least one of the following: The scene information corresponding to the image to be processed. One or more of the following: network status information, computing power, power consumption, and thermal threshold of the first device and / or the second device. The communication link information between the first device and the second device, and the communication between the second device and the first device.

5. The method according to any one of claims 1 to 4, characterized in that, Encoding the first intermediate result to obtain an encoded bitstream includes: Obtain the encoding method of the first intermediate result; Based on the encoding method, the first intermediate result is encoded to obtain the bitstream.

6. The method according to claim 5, characterized in that, The method further includes: Obtain the reconstruction method of the image to be processed, wherein the reconstruction method corresponds to the encoding method; The reconstruction method is encoded into the bitstream.

7. The method according to claim 5 or 6, characterized in that, The encoding method includes: Obtain a reference frame, which is an intermediate result of a reference image corresponding to the image to be processed; The first intermediate result is encoded based on the reference frame.

8. The method according to claim 5 or 6, characterized in that, The encoding method includes: Obtain preset data corresponding to the image to be processed, wherein the preset data includes a first preset intermediate result corresponding to the first region; The first intermediate result is encoded according to the preset data.

9. The method according to claim 8, characterized in that, The preset data is a preset intermediate result of the image to be processed, or the preset data is a preset intermediate result of a reference image corresponding to the image to be processed.

10. The method according to claim 8 or 9, characterized in that, The step of encoding the first intermediate result according to the preset data includes: Based on the preset data, obtain the first preset intermediate result; Based on the first intermediate result and the first preset intermediate result, the first difference data is obtained; The first difference data is encoded.

11. The method according to claim 8 or 9, characterized in that, The first intermediate result includes a second intermediate result obtained by the graph calculation of a portion of the first region, and a third intermediate result obtained by the graph calculation of another portion of the first region. Encoding the first intermediate result according to the preset data includes: Based on the preset data, a preset intermediate result is obtained for the other part of the first region in the image to be processed; Based on the third intermediate result and the other part of the preset intermediate results, the second difference data is obtained; The second difference data, the second intermediate result, and the third information are encoded, wherein the third information is used to indicate the location of a portion of the first region and / or the other portion in the image to be processed.

12. The method according to any one of claims 1 to 11, characterized in that, The method further includes: Obtain indication information corresponding to the image to be processed, wherein the indication information is used to indicate preset data corresponding to the image to be processed; The indication information is encoded into the bitstream.

13. The method according to claim 12, characterized in that, The indication information includes the address for obtaining the preset data corresponding to the image to be processed.

14. The method according to claim 3, characterized in that, The method further includes: Based on the second information, a second region in the image to be processed is obtained, and the scene change degree of the second region is lower than that of the first region; The fourth information is encoded into the bitstream, and the fourth information is used to indicate the position of the second region in the image to be processed.

15. An image processing method, characterized in that, The method includes: The bitstream is decoded to obtain the decoded data corresponding to the first intermediate result of the image to be processed. The first intermediate result is obtained by performing a first graphic calculation on a first region in the image to be processed. The first region is a region related to scene changes. Based on the decoded data corresponding to the first intermediate result, the graphic calculation result of the image to be processed is obtained.

16. The method according to claim 15, characterized in that, The step of obtaining the graphic calculation result of the image to be processed based on the decoded data corresponding to the first intermediate result includes: Obtain first information, which indicates the location of the first region in the image to be processed; Based on the first information and the decoded data corresponding to the first intermediate result, the graphic calculation result of the image to be processed is obtained.

17. The method according to claim 16, characterized in that, The step of obtaining the graphic calculation result of the image to be processed based on the decoded data corresponding to the first information and the first intermediate result includes: Obtain the reconstruction method of the image to be processed; Based on the first information and the decoded data, the graphic calculation result of the image to be processed is obtained according to the reconstruction method.

18. The method according to claim 17, characterized in that, The method for obtaining and reconstructing the image to be processed includes: Based on the first information, the reconstruction method of the image to be processed is obtained.

19. The method according to claim 17, characterized in that, The method for obtaining and reconstructing the image to be processed includes: The reconstruction method of the image to be processed is obtained from the bitstream.

20. The method according to claim 16, characterized in that, The step of obtaining the graphic calculation result of the image to be processed based on the decoded data corresponding to the first information and the first intermediate result includes: Obtain preset data corresponding to the image to be processed, wherein the preset data includes a first preset intermediate result corresponding to the first region in the image to be processed; Based on the preset data, the first information, and the decoded data corresponding to the first intermediate result, the graphic calculation result of the image to be processed is obtained.

21. The method according to claim 20, characterized in that, The preset data is a preset intermediate result of the image to be processed, or the preset data is a preset intermediate result of a reference image corresponding to the image to be processed.

22. The method according to claim 20 or 21, characterized in that, The decoded data is used to indicate the first difference data. The step of obtaining the graphic calculation result of the image to be processed based on the preset data, the first information, and the decoded data corresponding to the first intermediate result includes: Based on the preset data, the first information, and the first difference data, the graphic calculation result of the image to be processed is obtained; The first difference data is the difference between the first intermediate result of the image to be processed and the preset intermediate result of the first region in the image to be processed.

23. The method according to claim 20 or 21, characterized in that, The decoded data is used to indicate the first intermediate result. The step of obtaining the graphic calculation result of the image to be processed based on the preset data, the first information, and the decoded data corresponding to the first intermediate result includes: Based on the preset data and the first information, a preset intermediate result of the second region in the image to be processed is obtained, wherein the second region is a region in the image to be processed that is different from the first region. Based on the first intermediate result and the preset intermediate result of the second region, the graphic calculation result of the image to be processed is obtained.

24. The method according to claim 20 or 21, characterized in that, The decoded data includes third information, a second intermediate result obtained by the first graphic calculation of a portion of the first region, and second difference data corresponding to another portion of the first region. The third information is used to indicate the position of a portion and / or another portion of the first region in the image to be processed. The second difference data is the difference between the third intermediate result obtained by the first graphic calculation of the other portion and a preset intermediate result of the other portion. Obtaining the graphic calculation result of the image to be processed based on the preset data, the first information, and the decoded data corresponding to the first intermediate result includes: Based on the preset data and the third information, a preset intermediate result for the third region in the image to be processed, excluding the aforementioned portion, is obtained; Based on the preset intermediate results of the third region, the second difference data, and the second intermediate results, the graphic calculation results of the image to be processed are obtained.

25. The method according to claim 20, characterized in that, The step of obtaining the preset data corresponding to the image to be processed includes: Obtain indication information corresponding to the image to be processed from the bitstream; the indication information is used to indicate preset data corresponding to the image to be processed. Based on the indicated information, the preset data corresponding to the image to be processed is obtained.

26. The method according to claim 25, characterized in that, The indication information includes the address for obtaining the preset data corresponding to the image to be processed.

27. The method according to claim 16, characterized in that, The acquisition of the first information includes: The first information is obtained from the bitstream.

28. The method according to claim 15, characterized in that, The decoding of the bitstream to obtain the decoded data corresponding to the first intermediate result of the image to be processed includes: Obtain a reference frame, wherein the reference frame is the reconstructed data of the intermediate result of the reference image corresponding to the image to be processed; Based on the reference frame, the bitstream is decoded to obtain the decoded data corresponding to the first intermediate result of the image to be processed.

29. The method according to any one of claims 15 to 28, characterized in that, The step of obtaining the graphic calculation result of the image to be processed based on the decoded data corresponding to the first intermediate result includes: Based on the decoded data corresponding to the first intermediate result, the reconstructed data of the intermediate result of the image to be processed is obtained; A second graphic calculation is performed on the reconstructed data to obtain the graphic calculation result.

30. The method according to claim 29, characterized in that, The method further includes: Save the reconstructed data to a local or remote server.

31. The method according to any one of claims 15 to 30, characterized in that, Before decoding the bitstream, the method further includes: Obtain second information, which is used to divide the image to be processed into regions; Send the second message.

32. The method according to claim 31, characterized in that, The method is performed by a second device, and the second information includes at least one of the following: The scene information corresponding to the image to be processed. One or more of the following: network status information, computing power, power consumption, and thermal threshold of the first device and / or the second device. The communication link information between the first device and the second device, and the communication between the second device and the first device.

33. The method according to claim 16, characterized in that, Before decoding the bitstream, the method further includes: Send the first message.

34. An end-to-cloud collaborative system, the system comprising a client and a server, the server being configured to perform the method as described in any one of claims 1 to 14, and the client being configured to perform the method as described in any one of claims 15 to 33.

35. An image processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire a first region in the image to be processed, wherein the first region is a region related to scene changes; A graphics calculation module is used to perform graphics calculations on the first region to obtain a first intermediate result of the image to be processed; The encoding module is used to encode the first intermediate result to obtain an encoded bitstream.

36. An image processing apparatus, characterized in that, The device includes: The decoding module is used to decode the bitstream to obtain the decoded data corresponding to the first intermediate result of the image to be processed. The first intermediate result is obtained by performing a first graphic calculation on a first region in the image to be processed. The first region is a region related to scene changes. The acquisition module is used to obtain the graphic calculation result of the image to be processed based on the decoded data corresponding to the first intermediate result.

37. A device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 14, or the method as described in any one of claims 15 to 33.

38. A computer-readable storage medium, characterized in that, Includes a computer program that, when executed on the device, causes the device to perform the method of any one of claims 1 to 14, or the method of any one of claims 15 to 33.

39. A computer program product, characterized in that, The computer program product includes computer program code that, when run on a device, causes the device to perform the method of any one of claims 1 to 14, or the method of any one of claims 15 to 33.

40. A bitstream, characterized in that, The bitstream is generated by the method according to any one of claims 1 to 14.

41. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores the bitstream obtained by the method of any one of claims 1 to 14.

Images