Image rendering method, apparatus and computer readable storage medium

Abstract

The present disclosure provides an image rendering method, apparatus and computer readable storage medium, wherein the method is applied to the GPU of an electronic device, at least comprises: obtaining a first image to be rendered, wherein the first image comprises N pixel fragments, each pixel fragment comprising M pixels; N is a positive integer, M is a positive integer greater than 1; obtaining a first color information for rendering the first image, wherein the first color information comprises Q first chrominance information and M first luma information corresponding to each of the N pixel fragments, M is larger than Q; rendering the N pixel fragments based on the first color information to obtain a rendered image of the first image; sending the rendered image to a display device. In this way, GPU can perform subsampled format rendering, thus saving GPU resources and enhancing the rendering efficiency.

Description

[0001] Image Rendering Method, Apparatus and Computer Readable Storage Medium

[0002] TECHNICAL FIELD

[0003] The patent disclosure relates to the field of 3D and 2D graphics images creation and processing, and in particular, to an image rendering method, apparatus and computer readable storage medium.

[0004] BACKGROUND

[0005] In the field of 3D computer graphics, the caliber of image quality is a critical factor in determining a product's competitive edge. This is particularly evident in sectors such as gaming, virtual reality, and environmental simulation, where high-quality imagery is a key distinguishing feature.

[0006] Nonetheless, in scenarios involving real-time remote rendering, the pursuit of ultimate image quality is not essential. Take cloud gaming as an example, where users connect to cloud servers to access games or virtual worlds remotely. In these cases, the video streams that users receive have undergone lossy encoding, which degrades the image quality they perceive. Consequently, while pixel-perfect calculation might seem beneficial, they offer absolutely no improvements to the user experience and unnecessarily consume valuable GPU resources.

[0007] Also for the mobile devices rendering power usage constrains are very important for the devices end users, so the way to minimally decrease image quality while saving device memory and decreasing the power usage is very important.

[0008] SUMMARY

[0009] The present disclosure provides an image rendering method, apparatus and computer readable storage medium, which allows good quality images production while solves the problem of wasting GPU resource by means of subsampled YUV color space usage.

[0010] As a first aspect, an image rendering method is provided, which is applied to the GPU of an electronic device, wherein the method comprises: obtaining a first image to be rendered, wherein the first image comprises N pixel fragments, each pixel fragment comprising M pixels; N is a positive integer, M is a positive integer greater than 1 ; obtaining a first color information for rendering the first image, wherein the first color information comprises Q first chrominance information and M first luma information corresponding to each of the N pixel fragments, M is larger than Q; rendering the N pixel fragments based on the first color information to obtain a rendered image of the first image; sending the rendered image to a display device, wherein the rendered image is used for display on the display device; wherein, the rendering the N pixel fragments based on the first color information comprises: rendering M pixels in each of the pixel fragments based on Q first chrominance information corresponding toeach of the pixel fragments, and rendering M pixels in each of the pixel fragments based on M first luma information corresponding to each of the pixel fragments.

[0011] In this way, as the GPU can process 1 or 2 first chrominance information pairs and M first luma information corresponding to each pixel fragment during rendering, instead of processing color channels information of each pixel one by one, means that the GPU can perform subsampled format rendering, thus saving GPU resources and enhancing the rendering efficiency.

[0012] In a possible implementation of the first aspect, the sending the rendered image to a display device comprises: compressing the rendered image to obtain a compressed version of the rendered image, and sending the compressed version of the rendered image to the display device as a video frame in a video stream.

[0013] In a possible implementation of the first aspect, the ratio of N to M is one of the following ratios: 1:2; 2:4; 1:4.

[0014] In a possible implementation of the first aspect, the sending the rendered image to a display device comprises: obtaining a third image, wherein the third image and the first image are of the same subsampled format; combining the third image with the first image to obtain a combined image; and sending the combined image to a display device.

[0015] In a possible implementation of the first aspect, the third image comprises N pixel fragments, each pixel fragment of the third image comprises M pixels; the third image has second color information, the second color information comprises Q second chrominance information and M second luma information corresponding to each of the N pixel fragments; the combining the third image with the first image to obtain a combined image comprises: combining Q first chrominance information of the i th pixel fragment in the first image with Q second chrominance information of the i th pixel fragment in the third image to obtain the i th combined chrominance information; i is a positive integer greater than or equal to 1 and less than or equal to M; combining M first luma information of the i th pixel fragment in the first image with M second luma information of the i th pixel fragment in the third image to obtain i th set of combined luma information, each set of combined luma information comprising M combined luma information; and determining the chrominance of each pixel in the i th pixel fragment of the combined image based on the i th combined chrominance information, and determining the luma of M pixels in the i th pixel fragment of the combined image based on each of the M combined luma information in the i th set of combined luma information, to obtain the combined image.

[0016] In a possible implementation of the first aspect, the sending the combined image to a display device comprises: inputting the combined image into a deep learning model, to optimize quality parameters of the combined image, and obtain the optimized image, wherein the quality parameters include at least one of the following parameters: resolution, sharpness, color saturation, signal-to-noise ratio; sending the optimized image to a display device.In a possible implementation of the first aspect, the obtaining the second image based on the rendered image comprises: performing a full sampled format conversion on the rendered image to obtain the second image.

[0017] In a possible implementation of the first aspect, the obtaining the first color information for rendering the first image comprises: obtaining the first color information for rendering the first image when the subsampled rendering mode of the electronic device is configured to be enabled.

[0018] In a possible implementation of the first aspect, the obtaining the first color information for rendering the first image comprises: obtaining the third color information, the third color information comprising RGB color information corresponding to each pixel of the N pixel fragments; and performing a subsampled format conversion on the third color information to obtain the first color information for rendering the first image.

[0019] In a possible implementation of the first aspect, the rendering the N pixel fragments based on the first color information to obtain a rendered image of the first image comprises: rendering the N pixel fragments through a subsampled format shader based on the first color information to obtain a rendered image.

[0020] As a second aspect, an apparatus is provided, comprising: at least one processor couple with at least one memory storing computer program codes; wherein when the computer program codes are executed by the at least one processor, cause the first device to: obtain a first image to be rendered, wherein the first image comprises N pixel fragments, each pixel fragment comprising M pixels; N is a positive integer, M is a positive integer greater than 1 ; obtain a first color information for rendering the first image, wherein the first color information comprises Q first chrominance information and M first luma information corresponding to each of the N pixel fragments, M is larger than Q; render the N pixel fragments based on the first color information to obtain a rendered image of the first image; send the rendered image to a display device, wherein the rendered image is used for display on the display device; wherein, the render the N pixel fragments based on the first color information comprises: render M pixels in each of the pixel fragments based on Q first chrominance information corresponding to each of the pixel fragments, and rendering M pixels in each of the pixel fragments based on M first luma information corresponding to each of the pixel fragments.

[0021] In a possible implementation of the first aspect, the send the rendered image to a display device comprises: compress the rendered image to obtain a compressed version of the rendered image, and sending the compressed version of the rendered image to the display device as a video frame in a video stream.

[0022] In a possible implementation of the first aspect, the ratio of N to M is one of the following ratios: 1:2; 2:4; 1:4.

[0023] In a possible implementation of the first aspect, the send the rendered image to a display device comprises: obtain a third image, wherein the third image and the first image are of the same subsampledformat; combine the third image with the first image to obtain a combined image; and send the combined image to a display device.

[0024] In a possible implementation of the first aspect, the third image comprises N pixel fragments, each pixel fragment of the third image comprises M pixels; the third image has second color information, the second color information comprises Q second chrominance information and M second luma information corresponding to each of the N pixel fragments; the combine the third image with the first image to obtain a combined image comprises: combine Q first chrominance information of the i th pixel fragment in the first image with Q second chrominance information of the i th pixel fragment in the third image to obtain the i th combined chrominance information; i is a positive integer greater than or equal to 1 and less than or equal to M; combine M first luma information of the i th pixel fragment in the first image with M second luma information of the i th pixel fragment in the third image to obtain i th set of combined luma information, each set of combined luma information comprising M combined luma information; and determine the chrominance of each pixel in the i th pixel fragment of the combined image based on the i th combined chrominance information, and determine the luma of M pixels in the i th pixel fragment of the combined image based on each of the M combined luma information in the i th set of combined luma information, to obtain the combined image.

[0025] In a possible implementation of the first aspect, the send the combined image to a display device comprises: input the combined image into a deep learning model, to optimize quality parameters of the combined image, and obtain the optimized image, wherein the quality parameters include at least one of the following parameters: resolution, sharpness, color saturation, signal-to-noise ratio; send the optimized image to a display device.

[0026] In a possible implementation of the first aspect, the obtain the second image based on the rendered image comprises: perform a full sampled format conversion on the rendered image to obtain the second image.

[0027] In a possible implementation of the first aspect, the obtain the first color information for rendering the first image comprises: obtain the first color information for rendering the first image when the subsampled rendering mode of the electronic device is configured to be enabled.

[0028] In a possible implementation of the first aspect, the obtain the first color information for rendering the first image comprises: obtain the third color information, the third color information comprising RGB color information corresponding to each pixel of the N pixel fragments; and perform a subsampled format conversion on the third color information to obtain the first color information for rendering the first image.

[0029] In a possible implementation of the first aspect, wherein the render the N pixel fragments based on the first color information to obtain a rendered image of the first image comprises: render the N pixel fragments through a subsampled format shader based on the first color information to obtain a rendered image.As a third aspect, a computing device cluster is provided, comprising at least one computing device, wherein the computing device comprises at least one processor and a memory coupled with the processor, wherein the memory stores instructions which, when executed by the at least one processor, cause the computing device cluster to execute the method described.

[0030] As a fourth aspect, a computer-readable medium storing computer execution instructions is provided, which when run on a computing device cluster, cause the computing device cluster to execute the method described.

[0031] As a fifth aspect, a computer program product is provided, comprising computer execution instructions which, when run on a computing device cluster, cause the computing device cluster to execute the method described.

[0032] BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Fig. 1 illustrates a schematic diagram of a real-time remote rendering process.

[0034] Fig. 2 illustrate a schematic diagram of an apparatus architecture based on an embodiment.

[0035] Fig. 3 illustrates a schematic flowchart of an image rendering method based on an embodiment. Fig. 4 A illustrates a schematic diagram of YUV 444 format rendering pattern based on SOTA. Fig. 4B illustrates a schematic diagram of YUV 420 format rendering pattern based on an embodiment.

[0036] Fig. 5 illustrates a schematic flowchart of an image rendering method based on an embodiment. Fig. 6 A illustrates a schematic diagram of a pixel fragment for YUV420 format after rasterization based on an embodiment.

[0037] Fig. 6B illustrates a schematic diagram of a pixel fragment after shading based on an embodiment. Fig. 6C illustrates a schematic diagram of combining a pixel fragment in YUV420 format with another pixel fragment based on an embodiment.

[0038] Fig. 7 A illustrates a schematic structural diagram of a simulated YUV subsampled rendering pipeline based on an embodiment.

[0039] Fig. 7B illustrates a schematic structural diagram of a RGB rendering pipeline.

[0040] FIG. 8 is a schematic structural diagram of a computing device according to one or more embodiments of the present disclosure.

[0041] FIG. 9 is a schematic diagram of an architecture of a computing device cluster according to one or more embodiments of the present disclosure.

[0042] FIG. 10 is a schematic diagram of a connection between computing devices over a network according to one or more embodiments of the present disclosure.

[0043] DETAILED DESCRIPTION

[0044] To facilitate understanding, below are key terms mentioned in the embodiments.API (Application Programming Interface) - a set of rules implemented in a program code that allows different programs to communicate with each other. APIs hide the internal details of how a system works, exposing only external specifications.

[0045] Graphics API - public API describing communication between software and a GPU.

[0046] OpenGL (Open Graphics Library) - a cross-language, cross-platform Graphics API and open standard for 3D / 2D graphics.

[0047] Graphics Processing Unit - a specialized electronic circuit designed for digital image processing and computer graphics acceleration, being present in desktop computers, workstations, mobile phones and tablets, game consoles and some server models

[0048] Driver - a software program that allows other software programs (typically computer operating system but not only) to communicate with hardware components

[0049] Pipeline - an ordered set of data processing stages

[0050] Rendering - a process of generating images, either photorealistic or non-photorealistic from spatially defined numeric input data by means of a computer program.

[0051] Render Target - a surface (memory area, typically located in VRAM) that receives rendering result - either the on-screen or an off-screen one.

[0052] Pixel - the smallest addressable element in a computer image or in render target. Pixels contain various color channels depending on the color model used.

[0053] Rasterization - part of the rendering process that computes the mapping from scene geometry to pixels.

[0054] Graphic Engine (sometimes referred as "rendering engine", or "3D / 2D engine") - a complex system of software components that provides the functionality for the development of flexible and reusable computer graphics applications. Graphics engines are based on graphics API, but provide essential higher-level functionality

[0055] Shader - a user-defined program typically executed by GPU and designed to run bn some stage of 3D scene rendering. Shaders can also be used for general purpose on-GPU computations Fixed function - not shader-based program that is executed by some dedicated GPU hardware block

[0056] Texture - an image that is applied to a surface in process of rendering, so that it looks as if the image is "painted" onto the surface. Also, on some stage of rendering texture can be used as a render target.

[0057] Color Model - an abstract mathematical model describing the way colors can be represented as a tuple - ordered sequence of numbers (e.g. triples in RGB and YUV). Each tuple element is called a color channel. RGB - a color model in which the red (R), green (G) and blue (B) primary colors of light are added together in various ways to reproduce a broad array of colors.

[0058] RGBA - a color model in which to the RGB color model one more color channel. A (Alpha) is added to represent the level of transparencyYUV or YCbCr - the family of color models used in video and digital photography systems. Y is the luma or luminance (color intensity) component and U (Cb) and V(Cr) are the blue (b)-difference and red (r)-difference chroma components.

[0059] Sampling (to sample) - a process where color samples (values of actual color data in the color channels) are acquired by means of a computer program given their logical coordinates in the image coordinate space.

[0060] Full-sampled Format - All color components of each pixel point are fully sampled meaning that for the RGBA full-sampled format each pixel in the image has independent R, G, B and A values. The YUV full-sampled format, (which is also referred to as "YUV 4:4:4 format") means that each pixel in the image has independent Y, U and V values. Therefore the number of color components in all color channels for an image in full-sampled formats is equal.

[0061] Sampling rate - the number of pixels that share the same color channel entry for the given color format, expressed in l:n form. For full-sampled formats the sampling rate is equal to 1:1 Subsampled Format - Using different sampling rates for the luma and chroma color components. For example, in the YUV subsampled format, the sampling rate of chroma is lower than that of luma. In other words, each pixel in the image has an independent Y value, but multiple pixels share the same U value and the same V value. Therefore the number of color components in luma and chroma channels is different. At the same time, since the human eye is more sensitive to luma than chroma, the YUV subsampled format can reduce the amount of image data without significantly reducing image quality.

[0062] Based on different chroma sampling rates, YUV subsampled formats can include YUV 4:2:2 (also referred to as YUV 422), YUV 4:2:0 (also referred to as YUV 420), and YUV 4:1:1 (also referred to as YUV 411) formats.

[0063] Table 1 lists the characteristics of YUV sampling formats with different chroma sampling rates.

[0064]

[0065]

[0066] Table 1: Characteristics ofYUV Subsampling Formats

[0067] According to Table 1, YUV444 is a YUV full-sampled format, with Y, U, and V sampling rates all at 1 : 1. If the YUV full-sampled format is used for sampling, each pixel's Y, U, and V color channels are independently fully sampled, allowing for complete color calculations for each pixel's luma and chroma.

[0068] YUV422, YUV420, and YUV411 are subsampled formats, with their Y sampling rate at 1:1 and U, V sampling rates less than 1:1. If YUV subsampled formats are used, sampling is performed for a group of pixels' U pipeline and V pipeline, rather than independently sampling each pixel's U pipeline and V pipeline. Thus, multiple pixels in a group share U and V values, compressing storage space and image quality.

[0069] Taking YUV420 as an example, the sampling rates for U and V are both 1 :4, indicating that the U and V color channels are calculated once per 2x2 pixel block, meaning each 2x2 pixel block shares the same U and V value. Due to the reduced sampling rates of U and V, the storage space they require is also correspondingly reduced. The storage amount for either the U channel or the V channel is 1 / 4 of the Y channel, and the combined storage amount for the U and V channels is only half the size of the Y channel.

[0070] According to some embodiments, YUV formats can also include interleaved and planar layouts meaning U and V color channels memory location. Taking the YUV420 format as an example, for three 2x2 pixels blocks, the interleaved U and V layout can be represented as UVUVUV, while the planar layout can be represented as UUUVVV

[0071] As mentioned before, in real-time remote rendering scenarios, extreme image quality does not yield significant benefits in enhancing user experience, but it places high processing demands on the GPU, thereby wasting GPU resources.

[0072] However, real-time remote rendering of scenes is a common scenario. For instance, real-time remote rendering is widely used in at least the following two scenarios:

[0073] 1. Cloud gaming / VR: Remote acquisition of game and VR scenes. For example, when a user remotely connects to a game or virtual world in the cloud, the terminal device downloads and displays image data or video data from the cloud;2. Streaming: A scenario where audio and video content is transmitted over the network, allowing users to watch in real-time or on demand. For example, a user operates on their device and transmits the screen display content stream, such as a video stream composed of multiple frames of screen images, to another user's device or multiple audience’s devices,

[0074] In the real-time remote rendering scenario mentioned above, users who remotely obtain images will not see the full quality images. This is because the device receives not the original 3D rendering result, but the video stream of the original 3D rendering result after lossy encoding and decoding processing, resulting in a decrease in the quality of the image output to the user device compared to the quality of the rendered original image.

[0075] According to some existing embodiments, the essence below describes a real-time remote rendering process as illustrated in Fig. 1. The process includes the following steps:

[0076] S 101 : A first device processes RGB A data through a 3D render to generate a fully sampled RGB A image.

[0077] In some embodiments, each pixel of the rendering target corresponds to an RGBA data, which includes four RGBA color channels’ data, as R, G, B and A. In some embodiments, the 3D Tenderer performs full-color rendering on the rendering target, rendering each pixel of the target separately through the R, G, B, and A color channels to obtain a fully sampled image. Wherein each pixel in the fully sampled image includes complete color data.

[0078] S102: The first device converts the fully sampled rendered RGB image into a YUV subsampled format image.

[0079] In the YUV subsampled format, Y is calculated for each pixel, while U and V are calculated for each pixel group, including at least two pixels.

[0080] When converting the fully sampled RGB image to a YUV subsampled format image, the image quality will decrease. That is, the quality of the YUV subsampled format image generated is lower than that of the fully sampled RGB image.

[0081] For example, if the fully sampled image is converted to a YUV420 subsampled format, the U or V values for every 2x2 pixels block in the fully sampled image are reduced from four to one, thereby causing compression of storage space while reducing image quality.

[0082] S103: The first device encodes the YUV subsampled format image to generate a video stream. The video stream includes a series of image frames (i.e., the YUV subsampled format images from Step SI 02), arranged in order to simulate continuous video playback. It is understood that encoding can compress the total data volume of the image frames, but for a lossy encoding the picture quality of the image frames will further decrease.

[0083] SI 04: The first device transmits the video stream to the second device through network.

[0084] It is understood that the video stream can be transmitted from the first device to the second device via a network. The second device can be any electronic device with a display screen.S 105: The second device decodes the received video stream to obtain the video.

[0085] Once the video stream is decoded, the second device can obtain the video, and display the image frames included in the video on the screen. By playing the decoded image frames in the order of the video frames quickly, the video is presented on the screen.

[0086] In the real-time remote rendering process of S101-S105, image quality degradation occurs in SI 02 and S103. Therefore, even if high-quality rendering is used in SI 01 to obtain high-quality images, the high quality cannot be maintained after SI 02 and SI 03.

[0087] Thus, in real-time remote rendering scenarios, high-quality rendering or pixel-accurate computation is superfluous, as they will not enhance the user experience but will waste GPU resources and electricity.

[0088] To avoid wasting GPU resources and electricity in the rendering process, the present disclosure provides an image rendering method that can directly render first image in the YUV subsampled format on a first device. For example, obtaining input color data in the YUV subsampled format, and rendering the first image based on the color data in the YUV subsampled format to obtain the rendered image.

[0089] In some embodiments, the YUV subsampled format can be the YUV420 format. In this embodiment, rasterization transforms the first image into 2x2 pixel fragments, and subsequent coloring and rendering processes are all carried out in units of these 2x2 pixel fragments.

[0090] In some embodiments, to display on the second device's screen, the first device can encode the rendered image into a video stream and send it to the second device. After decoding the video stream, the second device obtains the video for local playback.

[0091] In some embodiments, after obtaining the rendered image, image processing models, such as deep learning models, can be used to optimize the quality of the rendered image.

[0092] The image rendering method provided in the embodiments of the present disclosure can be executed on electronic devices even with the first device presence only- without further encoding and network transfer. The GPU of the electronic device can support rendering in the YUV subsampled format. Any electronic device capable to perform local 3D / 2D rendering and to display its results, while the device memory usage, performance and / or power saving are important and slight quality drop is acceptable, would benefit from direct 3D / 2D YUV subsampled rendering. The electronic device mentioned comprise but are not limited with - desktops, workstations, embedded devices, mobile phones, tablets, game consoles. The type of electronic device is not limited, and any device capable of executing local 3D / 2D image rendering can be the electronic device.

[0093] The following description, with reference to Fig. 2, describes an exemplary device architecture offered by the present disclosure.

[0094] Fig. 2 illustrates example of apparatus 210. Apparatus 210 may include corresponding modules or units configured to implement methods and / or embodiments described herein. In some embodiments, the apparatus 210 includes a processing unit 212 and a communication unit 213. Optionally, the apparatus 210 may further include a storage unit 211 configured to store apparatus program code (orinstructions) and / or data, including APIs supported by the image API standard extensions, application(APP)s can invoke APIs to make the GPU perform rendering tasks according to the needs of image processing. For instance, through the API, the application can inform the GPU about the first image to be rendered, the second image to be mixed, and rendering data (such as shader code, color data like Y, U, V data), thereby performing subsampled rendering or full-sampled rendering of images.

[0095] For example, the application includes at least one of end-user application, graphic engines, operating systems GUI and so on.

[0096] The apparatus 210 may be a base station side apparatus, for example, a base station or a module in a base station, or a circuit or a chip responsible for a communication function in a base station. In some embodiments, apparatus 210 may be implemented as apparatus 320, accordingly, the processing unit 212 is implemented as processor 260 (the scheduler 223 may also be included), the communication unit 213 is implemented as transmitter 222 and / or receiver 224, and the storage unit 211 is implemented as memory 228.

[0097] In some embodiments, when the apparatus 210 is an ED 110 or a module in an ED 110, a function of the apparatus 210 may be implemented by one or more processors. Specifically, the processor may include a modem chip, or a system on chip SoC chip or a SIP chip that includes a modem core. A function of the communication unit 213 may be implemented by a transceiver circuit.

[0098] In some embodiments, when the apparatus 210 is a circuit or a chip that is responsible for a communication function in a ED 110, for example, a modem chip, a system on chip SoC chip or an SIP chip that includes a modem core, a function of the processing unit 212 may be implemented by a circuit system that is in the chip and that includes one or more processors or processor cores. A function of the communication unit 213 may be implemented by an interface circuit or a data transceiver circuit on the foregoing chip.

[0099] It may be understood that division into the units in the foregoing apparatus is merely logical function division. Each function may correspond to one functional unit, or two or more functions may be integrated into one functional unit. In actual embodiment, all or some of the units may be integrated into one physical entity, or may be distributed in different physical entities. In addition, the foregoing functional units may be implemented in a form of hardware, may be implemented in a form of software, or may be implemented in a form of a combination of hardware and software. Whether a function is performed in a form of hardware or software depends on particular applications and design constraint conditions of the technical solutions. A person skilled in the art may use different methods to implement the described functions for each particular application, but it should not be considered that the embodiment goes beyond the scope of this application.

[0100] In an example, a functional unit in any one of the foregoing apparatuses may be configured as one or more integrated circuits for implementing the foregoing methods, for example, one or more application-specific integrated circuits (application-specific integrated circuits, ASICs), one or more central processing units (central processing units, CPUs), one or more microprocessors (microcontrollerunits, MCUs), one or more digital signal processors (digital signal processors, DSP), one or more field programmable gate arrays (field programmable gate arrays, FPGAs), or a combination of at least two of these integrated circuit forms.

[0101] In an example, the storage unit 211 may include a random access memory, a flash memory, a readonly memory, a programmable read-only memory, an electrically erasable programmable memory, and / or a register.

[0102] A processor may be referred to as a processor system, an application processor, a baseband processor, a processor circuit, or a processor core. The processor may include one or a combination of one or more central processing units (CPUs), one or more digital signal processors (DSPs), one or more microprocessors (microprocessor units, MPUs), one or more microcontrollers (microcontroller units, MCUs), one or more graphics processing units (GPUs), one or more field programmable gate arrays (FPGAs), one or more artificial intelligence processors (Al processors), or one or more neural network processing units (NPUs).

[0103] In some embodiments, when an application calls direct YUV 3D subsampled rendering graphic API or SOTA RGB rendering API, the GPU can handle the 3D image rendering tasks generated by the application. For example, image rendering tasks include the first image to be rendered, the second image to be mixed, and rendering data. Among them, the functions supported by the GPU include but are not limited to: direct YUV subsampled rendering, RGB based rendering (i.e., RGB full-sampled rendering), and conversion between RGB full-sampled format and YUV subsampled format. It is understood that direct YUV subsampled rendering refers to directly rendering the first image to obtain an image in YUV subsampled format.

[0104] In some embodiments, the GPU can perform operations such as reading, processing, and writing image data (such as the first image, the third image, and other image data during the rendering process mentioned I ) based on different pixel fragment units. For example, the pixel fragments supported by the GPU can include 1x1, 1x4, 2x2, 1x2, etc where each fragment corresponds to the subsampled group of pixels sharing some color information. It is understood that the pixel fragment units supported by the GPU are not limited to the above units, and in other embodiments, the GPU can also support other pixel fragment unit sizes.

[0105] According to some embodiments, the GPU is designed to handle image data efficiently by processing pixels in 2x2 blocks, which is common in compressed image formats like YUV 4:2:0. Basically the GPU can read or write 4 values for Y channel and 1 value for U / V channel at one time. This approach optimizes performance as it allows the GPU to execute fragment shaders in parallel for these small pixel groups, enhancing rendering speed.

[0106] According to some embodiments, the GPU is designed to handle image data by processing pixel fragments in other size, which is not limited here. For example, the GPU can read or write M values for Y channel and R value for U / V channel at one time, wherein the M can be 4 or some multiple of 4, the R can be 1 or 2. And the number of M and R invocations are limited by image size only.According to some embodiments, the GPU is designed to adjust the size of pixel fragment to process at one time based on the specific subsampled format.

[0107] For example, an application can provide YUV subsampled format color data, corresponding shader code, and the first image through the direct YUV subsampled rendering API. The GPU driver calls the GPU to render each pixel fragments of the first image through a direct YUV subsampled rendering pipeline, using the shader corresponding to the shader code during the rendering process, and coloring each group of pixels based on the input color data.

[0108] For example, when the application has high requirements for image quality, it provides the first image through the SOTA RGB rendering API, and the GPU driver calls the GPU to render the first image through the RGB rendering pipeline.

[0109] For example, when the application has lower requirements for image quality, it provides RGB format input color data through the YUV subsampled rendering, and the GPU driver calls the GPU to convert the RGB format input color data into YUV subsampled format input color data. For example, convert RGBA to YUV420. Then, rendering each pixel fragment of the first image through a direct YUV subsampled rendering pipeline. In the rendering process memory is extensively used for intermediate data - render targets and textures storage.

[0110] The storage unit 211 may include one or more of the following storage media: a random access memory (RAM), a static random access memory (static RAM, SRAM), a dynamic random access memory (dynamic RAM, DRAM), a phase-change memory (PCM), a resistive random access memory (ReRAM), a magnetoresistive random access memory (MRAM), a ferroelectric random access memory (ferroelectric RAM, FRAM), a cache, a register, a read-only memory (ROM), a flash memory (flash memory), an erasable programmable read-only memory (erasable programmable ROM, EPROM), a hard disk, and the like. In an example, computer program instructions used to execute embodiments may be stored in a non-volatile memory, for example, at least a part of a memory or storage unit (for example, one or more of a ROM, a flash memory, an EPROM, or a hard disk). When a terminal runs, a part or all of corresponding computer program instructions may be loaded to a memory that has a higher transmission speed with the processor, for example, at least a part of a memory or a storage unit (for example, one or more of a RAM, an SRAM, a DRAM, a PCM, a RERAM, an MRAM, a FRAM, a cache, or a register), so that the processor executes the computer program instructions to perform the steps in the method embodiments disclosed herein.

[0111] In some embodiments, GPU dedicated video memory (VRAM) for systems with discreet GPUs, shared memory (RAM) for integrated GPU based devices. With direct YUV subsampled rendering memory used for textures and color render targets will be decreased up to 2 times for YUV420 leading to the total memory savings vs SOTA up to 50%.

[0112] Below, based on Fig. 3, an image rendering method provided in an embodiment is described. As shown in Fig. 3, the exemplary process of the image rendering method includes the following steps:

[0113] S301: Acquire the first image.In some embodiments, the first image can be a three-dimensional image data or scene.

[0114] It can be understood that three-dimensional objects in are composed of multiple vertices, and edges connecting these vertices to form multiple triangles. These triangles serve as primitives that can be processed by the GPU, for example, vertices of primitives can be transformed to 2D and then shaded by the GPU by invoking shaders.

[0115] S302: Rasterize the first image into pixel fragments.

[0116] In some embodiments, the vertices of the first image can be rasterized to obtain pixel fragments. In some embodiments, the size of the pixel fragments obtained by rasterization can match the YUV subsampled format.

[0117] In some embodiments, if the YUV subsampled format is YUV422, the pixel fragment can be a 2x2, or 4x2 pixel fragment.

[0118] In some embodiments, if the YUV subsampled format is YUV420, the pixel fragment can be a 2x2 pixel fragment.

[0119] In some embodiments, if the YUV subsampled format is YUV411 , the pixel fragment can be a 4x1 , 2x2 or 4x2 pixel fragment.

[0120] S303: Shade each pixel fragment to obtain a YUV subsampled image.

[0121] Shading refers to applying color or other visual effects to the pixels to be shaded through a YUV subsampled fragment shader.

[0122] After obtaining the pixel fragments, a YUV subsampled fragment shader can be invoked to shade each pixel fragment.

[0123] In some embodiments, if the YUV subsampled format is YUV420, for a 2x2 pixel fragment the YUV subsampled fragment shader can shade each pixel fragment based on 4 Y values, 1 U value, and 1 V value. For example, apply 4 Y values to each of the 4 pixels, then apply 1 U value to the 4 pixels collectively, and apply 1 V value to the 4 pixels collectively.

[0124] For example, for each pixel fragment, a single instance of the YUV subsampled fragment shader can be invoked to shade each pixel fragment.

[0125] In some embodiments, it should be noted that the rendering process can be carried out in units of pixel fragments. Specifically, the color data of one pixel fragment is first read. Then, the rendering for this particular pixel fragment is completed, and the rendering result of this pixel fragment is written. After that, the operation proceeds to the next pixel fragment.

[0126] For example, each pixel has 1 Y value, and M pixels share 1 U value and 1 V value. Moreover, the number of pixels within a pixel fragment can be M, or larger For instance, M can be 2 or 4, and thus the number of pixels in a pixel fragment can be 4, 8, and so forth. It is important to note that the aspect ratio of the pixel fragment is also not restricted, for example, each pixel fragment can be 2x2, 4x1, 4x2 and so on. Note that in the embodiments described, for a size of pixel fragment as p x q, p stands for horizontal fragment size, and then q stands for vertical fragment size. For example, the YUV subsampled fragment shader can be invoked based on the shader code and the pixel fragments can beshaded based on the color data. The shader code and color data can be provided by the application to the GPU driver through the API, enabling the GPU driver to control the GPU to execute rendering tasks using the shader code and color data.

[0127] S304: Blend the subsampled YUV image with the second image to obtain the combined image. After obtaining the subsampled YUV image, the subsampled YUV image can be combined with the second image to obtain the combined image.

[0128] In some embodiments, the second image is the image to be combined, which can be provided by the application.

[0129] Fig. 4A illustrates a schematic diagram of YUV 444 format rendering pattern, and Fig. 4B illustrates a schematic diagram of YUV 420 format rendering pattern. In this embodiment, for YUV 444, the reading, rendering and writing for luma and chroma are performed in units of each pixel. And for YUV 420, for luma, image combining can be performed in units of each pixel; for chroma, image combining can be performed in units of 2x2 pixel fragments. In other words, reading, rendering and writing of luma and alpha is done per pixel, and same procedure for chroma is performed once for 4 luma pixels.

[0130] For clearer explanation, after all the image rendering and / or processing procedures are completed, the resulting image can be referred to as the third image below.

[0131] As an example, the weighted Y, U, V values of each pixel fragment in the subsampled YUV image, and the weighted Y, U, V values of each pixel fragment in the second image can be added to obtain the target color data; the target color data can be applied to each pixel fragment in a blank image which has the same pixel number as the first image or the second image, to obtain the third image.

[0132] S305: Perform quality optimization on the combined image to obtain the optimized image.

[0133] In some embodiments, the GPU or CPU can perform quality optimization processing on the combined image by computer vision tasks through image optimization models, such as Al models, deep learning models, etc., to obtain the optimized image, as a third image. For example, use a defogging algorithm model to enhance the clarity of the image.

[0134] In some embodiments, after S305 is completed, it can jump to S306 or S307. For example, if the rendering purpose of the application is to display the third image locally or on other electronic devices without encoding, it jumps to S306; if the rendering purpose of the application is to display the video on other electronic devices using video streaming, it jumps to S307.

[0135] S306: Convert the optimized image to RGB format.

[0136] In some embodiments, if the electronic device has a display screen or has display functionality, the GPU can convert the optimized image to RGB format for local screen output on the electronic device, displaying the optimized image on the electronic device.

[0137] In other embodiments, the image converted to RGB format can be sent by the electronic device to other devices for display on other devices.

[0138] S307: Encode the optimized image into a video stream.In some embodiments, the GPU can encode the optimized image into a video stream and send it to other devices, which can decode the video stream after receiving it to obtain a playable video.

[0139] Through the embodiments described, not only can direct YUV sampled rendering of the first image be achieved, but also subsampled rendering of various subsampled formats can be supported, thereby reducing the load on the GPU, improving rendering efficiency; while ensuring the image or video quality when user or audience view images or videos.

[0140] Based on the description of Fig. 3, the GPU can render the second image in a subsampled format. In other embodiments, the GPU can also render the second image in a full-sampled format. When necessary, color data can be converted between formats, such as between subsampled and full-sampled color data.

[0141] Below, further details of an image rendering method provided by this example are introduced based on Fig. 4A, Fig.4B, and Fig. 5. Fig. 4 A illustrates a rendering scenario diagram of full-sampled format, Fig. 4B illustrates a rendering scenario diagram of a YUV subsampled format including optional A component. Fig. 5 illustrates an exemplary flowchart of an image rendering method. The image rendering method shown in Fig. 5 can be executed by an electronic device, specifically by the GPU of the electronic device.

[0142] As shown in Fig. 5, the exemplary flowchart of the image rendering method provided in this example includes the following steps:

[0143] S501: Acquire color data and shader code.

[0144] In some embodiments, the color data can include target color, texture, and other data.

[0145] In some embodiments, the color data can be sent to the GPU by the application through an API. For specific details, refer to the description of Fig. 2; it will not be repeated here.

[0146] In some embodiments, the shader code corresponds to a shader. For example, a shader that supports a subsampled format can shade pixel fragments, while a shader that supports a full-sampled format can shade individual pixels.

[0147] In some embodiments, the shader code cannot be obtained from APPs, or other modules, in this case GPU can generate a shader code based on the specific format of rendering, such as a subsampled format or a full-sampled format.

[0148] In some embodiments, when the application enables YUV subsampled rendering, the application can provide color data like textures or any other color data and shader code in a subsampled format to the GPU. In this case, inside GPU driver direct YUV 3D subsampled rendering pipeline is used.

[0149] In some embodiments, when the application or GPU driver enables YUV subsampled rendering, the application can provide full-sampled color data to the GPU, and the GPU can convert full-sampled color data into subsampled color data. For example, in full-sampled color data, each pixel can correspond to a target color data or a texture data. In this case, inside GPU driver direct YUV 3D subsampled rendering pipeline is used with preliminary in-driver input color data and shader code conversion to YUV subsampled format.In some embodiments when the application has some specific requirements like pixel-precise image quality, some special color processing etc, and the application or GPU driver does not enable YUV subsampled rendering, the application can provide full-sampled color data to the GPU. Inside GPU driver RGB rendering pipeline is used.

[0150] S502: Determine if the color data and / or shader code correspond to a subsampled format. If yes, proceed to S503; if no, proceed to S507.

[0151] The GPU can determine the format corresponding to at least one of the color data or shader code and adopt different rendering strategies based on different formats.

[0152] If the color data corresponds to a subsampled format, proceed to S503 for rendering based on the subsampled format. If it is determined that the color data does not correspond to a subsampled format, then proceed to S507 for further determination of whether the application has enabled subsampled rendering. The rendering method is then determined based on the outcome of this determination.

[0153] In some embodiments, the shader code can be analyzed to determine if the shader is used for color calculations in the YUV subsampled space. If so, it can be determined that the shader code corresponds to a subsampled format, and proceed to S503; if not, proceed to S507.

[0154] S503: Render a first image based on the subsampled format to obtain the target image.

[0155] The target image can be a rendered image or a combined image.

[0156] In some embodiments, the GPU can complete the rendering of the first image through steps such as rasterization, shading, and combining.

[0157] In some embodiments, the applications submit their input data in YUV subsampled format along with the corresponding shader code. Direct YUV subsampled rendering happens in GPU / GPU driver producing YUV subsampled output, that could be optionally passed to any image processing software operating in YUV subsampled format - like Al or computational algorithms based image improvement, recognition etc., and finally to be either converted to RGB for the screen output or passed to the video encoder.

[0158] For example, first rasterize the first image to obtain pixel fragments that comply with the subsampled format; then, call the shader based on the shader code and shade the pixel fragments based on the color data; finally, combine the shaded image with the second image to obtain the third image.

[0159] In some embodiments, rendering can be achieved through the GPU's YUV subsampled rendering pipeline. The rendering can be two-dimensional or three-dimensional; the type of rendering Is not limited.

[0160] Below, referring to Fig. 6A to Fig. 6C, an optional rendering process for S5O3 is introduced as an example where the application enables YUV subsampled rendering, wherein the shaded image is to combined with a second image to obtain the third image, and the subsampled format is YUV420.

[0161] First, rasterize the first image to obtain pixel fragments. The size of the pixel fragment is 2x2, including pixels 0, 1, 2, 3.Then, shade the pixel fragments based on the color data to obtain the shaded image. For example for YUV420 format for each pixel fragment, color data includes 4 Y values, 1 U value, and 1 V value. According to Fig. 6A, each of pixels 0, 1, 2, 3 has a Y value, but they share 1 U value and 1 V value. After shading, the 4 Y values, 1 U value, and 1 V value of each pixel fragment may change.

[0162] For example, as shown in Fig. 6B, after shading, the color information of each pixel fragment can be represented as L0, LI, L2, L3, Cb, Cr, corresponding to Y, Y, Y, Y, U, V.

[0163] According to some embodiments, based on the YUV subsampled format, GPU can determine the step size of reading Y, U, and V values. For example, if 1 bit of data represents one color value, based on a YUV 420 format, GPU can determine to read 4 bits of Y data, 1 bit of U value and then 1 bit of V value for a single rendering pass.

[0164] According to some embodiments, YUV formats can include interleaved and planar layouts. Both can be processed by GPU. For example, for planar layout, if 1 bit of data represents one color value, based on a YUV 420 format, GPU can determine to read 4 bits of Y data, 2 bits of U value, 2 bits of V value for a single rendering pass. For example, for interleaved layout, if 1 bit of data represents one color value, based on a YUV 420 format, GPU can determine to read 4 bits of Y data, 1 bit of U value, 1 bit of V value, then another 1 bit of U value, and another 1 bit of V value, for a single rendering pass.

[0165] Then, combine the shaded image with the second image to obtain the combined image.

[0166] For example, referring to Fig. 4C, if the second image to be combined is also in YUV420 format, the L0, LI, L2, L3, Cb, Cr values of the shaded image and the second image can be merged, that is, by merging the Y values, U values, and V values to obtain the combined image. Note that the second image and the shaded image can be of the same YUV subsampled format. For a single combining pass, combining can be processed based on the unit of each pixel fragment. For example, for a single pass of YUV 420 format combining, combining the L0, LI, L2, L3 of the shaded image and the second image, combining the Cb of the shaded image and the second image, and combining the Cr of the shaded image and the second image, and obtain the L0, LI, L2, L3, Cb, Cr of the combined image.

[0167] As an example, the L0, LI, L2, L3, Cb, Cr of each pixel fragment in the shaded image can be multiplied by the first mixing factor (e.g., n, O^n^l) to obtain the first weighted color information; the L0, LI, L2, L3, Cb, Cr of the second image can be multiplied by the second mixing factor (e.g., m, O^m^l) to obtain the second weighted color information; the first weighted color information and the second weighted color information are added to obtain the target color information, which is applied to each pixel fragment in the second image to obtain the combined image.

[0168] S504: Perform quality optimization processing on the target image to obtain the optimized image. In some embodiments, the GPU can perform quality optimization processing on the first, second or target image through image optimization models, such as Al models, deep learning models, etc., to obtain the optimized image.

[0169] In some embodiments, after S504 is completed, proceed to S505 or S506.PC17RU2025 / 000024

[0170] S505: Convert the optimized image to RGB format.

[0171] In some embodiments, if the electronic device has display functionality or a display screen, the GPU can convert the optimized image to RGB format for local screen output on the electronic device, displaying the optimized image on the electronic device.

[0172] In other embodiments, the image converted to RGB format can be sent by the electronic device to other devices for display on other devices without encoding.

[0173] S506: Encode the optimized image into a video stream.

[0174] In some embodiments, the GPU can encode the optimized image into a video stream and send it to other devices, which can then decode the video stream to obtain playable video. In some embodiments, Al models can be used for frame interpolation encoding to enhance the frame rate and presentation quality of the video stream.

[0175] S507: Determine if the YUV subsampled rendering is enabled. If yes, proceed to S508; if no, proceed to S509.

[0176] It can be understood that the GPU can determine whether the application has enabled YUV subsampled mode and determine the rendering method as YUV subsampled rendering or RGB full-sampled rendering based on the determination result. For example, before S501, the application submits rendering commands to the GPU through an API, where the rendering commands indicate whether the application has enabled YUV subsampled rendering. Alternatively, the GPU can obtain system setting information through an API and determine whether the application has enabled YUV subsampled rendering based on the system setting information.

[0177] In some embodiments, the rendering method enable status of all applications can be modified through the system settings of the electronic device, for example, modifying the rendering method enable status of all applications to enable YUV subsampled rendering by click operation, voice operation, gesture operation and so on. In other embodiments, the rendering method enable status of individual applications can be modified.

[0178] In some embodiments, applications default to enabling YUV subsampled rendering or default to not enabling YUV subsampled rendering. For example, preset configurations of applications are added to the application installation package, allowing applications to run according to preset configurations.

[0179] S508: Convert color data and shader code from full-sampled format to subsampled format.

[0180] In some embodiments, if the GPU needs to render the first image in a subsampled format according to the application's requirements, but the application provides full-sampled color data, then the GPU can convert color data from full-sampled format to subsampled format. For example, convert color data in RGB / RGBA format to YUV420 format.

[0181] Also, the shader code for full-sampled format rendering should be converted to shader code for subsampled format color, which means that, determine a shader code corresponding to a shader that is used for YUV subsampled format rendering, instead of using the shader code for full-sampled format.According to an embodiment, any shader code can be determined once the shader code is for subsampled format rendering.

[0182] In other words, if direct YUV subsampled 3D / 2D rendering is enabled in GPU driver settings for this particular application, then inside GPU driver the application input RGB data will be automatically converted to YUV subsampled format and processed by the YUV subsampled rasterization pipeline using the shader code automatically modified inside the GPU driver accordingly. The YUV subsampled output produced could be optionally passed to any image processing software operating in YUV subsampled format - like Al or computational algorithms based image improvement, recognition etc., and finally either converted to RGB for the screen output or passed to the video encoder.

[0183] In some embodiments, after S508 is completed, proceed to S503.

[0184] S509: Render the first image based on the full-sampled format to obtain the target image.

[0185] In some embodiments, the GPU can complete the full-sampled format rendering of the first image through steps such as rasterization, shading, and combining. If direct YUV subsampled 3D / 2D rendering is disabled in GPU driver settings for this application - it may be done if very high pixel precise mage quality is required for the local rendering use case or if some specific case of RGB processing is involved, then SOTA RGB rendering pipeline is being called for this particular application. According to some embodiments the RGB rendering can be replaced with RGBA rendering.

[0186] For example, first rasterize the first image to obtain multiple pixels; then, call the shader based on the shader code and shade the pixels based on the color data; finally, combine the shaded image with the second image to obtain the target image.

[0187] In some embodiments, after S509 is completed, proceed to S510, or S511 and S512.

[0188] S510: Display the target image in RGB format.

[0189] In some embodiments, the target image is in RGB format or RGBA format, and can be displayed by the electronic device or other devices.

[0190] S511 : Convert the target image to subsampled format.

[0191] In some embodiments, the GPU can convert the target image to subsampled format, in order to encode them into a video stream and send it to other devices.

[0192] In some embodiments, Al models or any computational algorithms can be used for frame interpolation encoding to enhance the frame rate and presentation quality of the video stream.

[0193] S512: Encode the target image into a video stream.

[0194] In some embodiments, the video stream can be sent to other devices, and after other devices receive the encoded video stream, they can then decode the video stream to obtain playable video.

[0195] Through embodiments described, applicable rendering methods can be adopted according to the needs of different applications, thereby enabling targeted rendering of images of different qualities. Moreover, for subsampled rendering, the target image can be efficiently optimized for quality, thus rapidly enhancing image quality without affecting rendering efficiency.If the rendering is based on a subsampled format compared to full-sampled format rendering, rendering efficiency can be greatly improved while maintaining essentially the same final image quality. Below, referring to Fig. 7A and Fig. 7B, a comparison of the two rendering methods is provided. Fig 7A corresponds to the embodiment where YUV subsampled rendering pipeline is emulated using existing SOTA GPU driver pipeline while fig 7B shows the corresponding SOTA RGB rendering pipeline followed by YUV conversion that emulates the actual conversion happening in process of video encoding so that the results of YUV subsampled rendering could be compared with the results of SOTA rendering.

[0196] As shown in Fig. 7A, the YUV subsampled rendering pipeline includes a G-buffer (geometry buffer), a luma pass, a chroma pass, and a combining pass.

[0197] For example, in a 3D rendering scenario, the G-buffer can be used to obtain at least one of primitives, vertices, depth information and shadow data of the first image. Wherein depth information and shadow data are used to determine the order of objects in the scene and the lighting effects between object surfaces.

[0198] For example, the luma pass is used for luma shading of the first image.

[0199] For example, the chroma pass is used for chroma shading of the first image. For instance, using YUV422 subsampled format rendering, compared to using YUV444 subsampled format rendering, the size of the chroma pass data can be reduced by 1 / 2.

[0200] For example, the combining pass is used to combine the results of the Luma and Chroma passes for the first image

[0201] As shown in Fig. 7B, the RGB rendering pipeline includes G-buffers, a forward pass, and a YUV subsampled conversion pass. It can be understood that the RGB rendering pipeline inputs the first image and full-sampled color data, and outputs subsampled YUV images.

[0202] For example, in a 3D rendering scenario, the G-buffer can be used to obtain at least one of primitives, vertices, depth information and shadow data of the first image. Wherein depth information and shadow data are used to determine the order of objects in the scene and the lighting effects between object surfaces.

[0203] For example, the forward pass is used for rendering the first image to obtain the rendered RGB image.

[0204] For example, the YUV subsampled conversion pass is used to convert the rendered RGB image into a subsampled YUV image.

[0205] Comparatively, the chroma pass in the YUV subsampled rendering pipeline significantly reduces rendering time for chrominance shading compared to the forward pass in the RGB rendering pipeline, due to the compressed chrominance values in the YUV subsampled format compared to the full-sampled format.PC17RU2025 / 000024

[0206] In the YUV conversion pass of the RGB rendering pipeline, there is a decrease in image quality, resulting in the subsampled YUV images output by the RGB rendering pipeline being of almost the same quality as the subsampled YUV images output by the YUV subsampled rendering pipeline.

[0207] Unlike SOTA RGB rendering pipeline methods using for rendering memory saving and performance improvement ms like super-resolution etc., which lead to visible and measurable quality degradation, the YUV subsampled rendering proposed either keeps image quality the same as original RGB based rendering - for the real time remote rendering usages; or makes it insignificantly (indistinguishably to the human eye) worse - for any local rendering applications not having video encoding stage in their pipeline.

[0208] Additional up to 1.5x performance improvement is expected in rendering results post-processing - from straightforward color adjustments to complicated tasks like Al based super resolution due to smaller number of elements to process in U and V channels.

[0209] In some embodiments, dynamic scenes with complex geometric shapes, including complex physical lighting calculations, are rendered based on the above two rendering pipelines, followed by frame interpolation video encoding using Al to obtain a video stream, thereby assessing the performance of the two rendering pipelines. The assessment results include rendering speed and the image quality presented to users.

[0210] The described assessment basically focuses on VRAM usage and PSNR quality metric of following aspects.

[0211] 1) Simulated direct YUV subsampled rendering - pipeline 7 A

[0212] 2) Encoded / Decoded result of RGB rendering - i.e. the data provided to the end user.

[0213] 3) Encoded / Decoded result of YUV subsampled rendering.

[0214] The results show that the rendering speed of the YUV subsampled rendering pipeline is faster than that of the RGB rendering pipeline, and the image quality presented to users by both pipelines and measured by PSNR is almost the same. Specifically, by using direct YUV subsampled rendering, up to 1.5x VRAM saving (one-third lower usage) vs SOTA and 1.5x faster colors involving calculations are expected due to smaller number of elements to process in U and V channels. Any underlying rendering algorithm is liable to it - traditional rendering, Al based rendering, ray tracing. In its turn it will lead to the whole rendering process performance improvement to some extent depending on the exact rendering algorithm used.

[0215] According to described examples, rendering images in YUV subsampled format not only ensures the picture quality of images or videos viewed by users but also improves rendering speed and reduces the memory of the GPU, thereby saving power and significantly reducing hardware costs.

[0216] An apparatus may be the implemented as described in the above method embodiments. For example the apparatus may include one or more modules, which are configured to perform the method described in the above method embodiments. It should be understood by a person skilled in the art that, the relevant description of the apparatus modules in the embodiments of the present disclosure may beunderstood with reference to the relevant description of the method for frame interpolation in the embodiments of the present disclosure.

[0217] Fig. 8 is a schematic structural diagram of a computing device according to one or more embodiments of the present disclosure. As shown in Fig. 8, the computing device 800 includes a processor 802, an interface 804 for communicating with other devices, a memory 806, and a bus 808. The processor 802, the interface 804, and the memory 806 are in communication with each other via the bus 808. The memory 806 may be stored with computer execution instructions, and the processor 802 executes computer execution instructions stored in the memory 806 to enable the computing device to execute any of the above methods. The computing device 800 can be a server or a terminal device. The number of processors and memories in the computing device 800 is not limited in the embodiments of the present disclosure.

[0218] The bus 808 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, only one line is used in Fig. 8, but it does not mean that there is only one bus or one type of bus. The bus 808 may include pathways for transferring information between various components of the computing device 800 (e.g., the memory 806, the processor 802, or the interface 804).

[0219] The processor 802 can include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP) processor.

[0220] The memory 806 can include volatile memory, such as random access memory (RAM). The processor 802 can also include non-volatile memory, such as read-only memory (ROM), flash memory, a mechanical hard disk (HDD), or a solid state drive (SSD).

[0221] An executable program code is stored in the memory 806, and the processor 802 executes the executable program code to implement the method or the functions of the aforementioned apparatus in the above embodiments. That is, the memory 806 has instructions on it for executing the method described in the above embodiments.

[0222] The structure of the computing device 800 enumerated above is only an exemplary illustration, and the present disclosure is not limited thereto. The computing device 800 of embodiments of the present application includes a variety of hardware found in computer systems in the prior art, e.g., the computing device 800 also includes memory other than the memory 806, such as, for example, a disk memory, and the like. It should be understood by those skilled in the art that the computing device 800 can also include other devices necessary to achieve proper operation. In addition, depending on the specific needs, one of skill in the art should understand that the above-described computing device 800 may also include other hardware devices for realizing other additional functions. In addition, one of skill in the art should understand that the above-described computing device 800 may also include only the devices necessary to realize the embodiments of the present disclosure, and need not include all of the devices shown in Fig. 8.Fig. 9 is a schematic diagram of an architecture of a computing device cluster according to one or more embodiments of the present disclosure. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0223] As shown in Fig. 9, the cluster of computing devices described includes at least one computing device 900. The memory 906 in one or more of the computing devices 900 in the cluster of computing devices can hold the same instructions for performing the method described in the above embodiments.

[0224] In some possible implementations, some of the instructions for performing the method described in the above embodiments can also be separately held in the memory 906 of the one or more computing devices 900 in the cluster of computing devices. In other words, a combination of the one or more computing devices 900 can jointly execute instructions for performing the method described in the above embodiments.

[0225] It is noted that the memories 906 in the different computing devices 900 in the cluster of computing devices can store different instructions for performing some of the functions of the computing devices 900, respectively. That is, the instructions stored in the memories 906 in the different computing devices 900 can realize the functions of one or more of the modules of the apparatus.

[0226] In some possible implementations, one or more computing devices in a cluster of computing devices can be connected via a network. Among other things, the network can be a wide area network or a local area network, etc. Fig. 10 illustrates one possible implementation. Fig. 10 is a schematic diagram of a connection between computing devices 1000 A and 1000B over a network according to one or more embodiments of the present disclosure. The two computing devices 1000 A and 1000B are connected to each other via a network. Specifically, the connection to said network is made through a communication interface in each computing device. In this class of possible implementations, the memory 1006 A in the computing device 1000 A holds instructions for performing the functions of the one or more modules of apparatus provided.

[0227] The connection between the clusters of computing devices shown in Fig. 10 can be such that, considering that the method provided in the present disclosure requires processing a large number of data, it is contemplated that the functions realized by the one or more modules of apparatus can be performed by the computing device 1000B.

[0228] The functions of computing device 1000 A illustrated in Fig. 10 can also be accomplished by multiple computing devices. Similarly, the functions of computing device 1000B can be accomplished by multiple computing devices.

[0229] In some aspects of the present disclosure, there is provided an apparatus for executing any of the above methods. It should be understood that the apparatus can execute the steps in the above method embodiments, which will not be repeated here.In some aspects of the present disclosure, there is provided a computing device cluster including at least one computing device, and the computing device includes a processor and a memory coupled with the processor. The memory is storing instructions which, when executed by the processor, cause the computing device cluster to perform any of the above methods.

[0230] In some aspects of the present disclosure, there is provided a chip, including an input / output (I / O) interface and a processor, where the processor is configured to call and run computer execution instructions stored in a memory, to enable a device installing with the chip to execute any of the above methods.

[0231] In some aspects of the present disclosure, there is provided a computer-readable medium storing computer execution instructions which, when run on a computing device cluster, cause the computing device cluster to execute any of the above methods.

[0232] In some aspects of the present disclosure, there is provided a computer program product including computer execution instructions which, when run on a computing device cluster, cause the computing device cluster to execute any of the above methods.

[0233] In some aspects of the present disclosure, there is provided a computer program including computer execution instructions which, when run on a computing device cluster, cause the computing device cluster to execute any of the above methods.

[0234] It will be appreciated that any module, component, or device disclosed herein that executes instructions may include, or otherwise have access to, a non-transitory computer / processor readable storage medium or media for storage of information, such as computer / processor readable instructions, data structures, program modules and / or other data. A non-exhaustive list of examples of non-transitory computer / processor readable storage media includes magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, optical disks such as compact disc read-only memory (CD-ROM), digital video discs or digital versatile discs (i.e., DVDs), Blu-ray Disc™, or other optical storage, volatile and non-volatile, removable and non-removable media implemented in any method or technology, random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology. Any such non-transitory computer / processor storage media may be part of a device / apparatus or accessible or connectable thereto. Computer / processor readable / executable instructions to implement a method, an application or a module described herein may be stored or otherwise held by such non-transitory computer / processor readable storage media.

[0235] It could be noted that the message in the disclosure could be replaced with information, which may be carried in one single message, or be carried in more than one separate message.

[0236] Without special noting, the terms “apparatus” and “device” are used exchangeable, and the terms “identity” and “identifier” are sued exchangeable.

[0237] In the disclosure, the word “a” or “an” when used in conjunction with the term “comprising” or “including” in the claims and / or the specification may mean “one”, but it is also consistent with themeaning of “one or more”, “at least one”, and “one or more than one” unless the content clearly dictates otherwise. Similarly, the word “another” may mean at least a second or more unless the content clearly dictates otherwise.

[0238] In the disclosure, the words “first”, “second”, etc., when used before a same term (e.g., ED, or an operating step) does not mean an order or a sequence of the term. For example, the “first ED” and the “second ED”, means two different EDs without specially indicated, and similarly, although the present disclosure describes methods and processes with steps in a certain order, one or more steps of the methods and processes may be omitted or altered as appropriate. One or more steps may take place in an order other than that in which they are described, as appropriate. For example, the “first step” and the “second step” means two different operating steps without specially indicated, but does not mean the first step have to happen before the second step. The real order depends on the logic of the two steps.

[0239] The terms “coupled”, “coupling” or “connected” as used herein can have several different meanings depending on the context in which these terms are used. For example, as used herein, the terms coupled, coupling, or connected can indicate that two elements or devices are directly connected to one another or connected to one another through one or more intermediate elements or devices via a mechanical element depending on the particular context.

[0240] Note that the expression “at least one of A or B”, as used herein, is interchangeable with the expression “A and / or B”. It refers to a list in which you may select A or B or both A and B. Similarly, “at least one of A, B, or C”, as used herein, is interchangeable with “A and / or B and / or C” or “A, B, and / or C”. It refers to a list in which you may select: A or B or C, or both A and B, or both A and C, or both B and C, or all of A, B and C. The same principle applies for longer lists having a same format.

[0241] The present disclosure encompasses various embodiments, including not only method embodiments, but also other embodiments such as apparatus embodiments and embodiments related to non-transitory computer readable storage media. Embodiments may incorporate, individually or in combinations, the features disclosed herein.

[0242] The term “receive”, “detect” and “decode” as used herein can have several different meanings depending on the context in which these terms are used. For example, without special note, the term “receive” may indicate that information (e.g., DC1, or MAC-CE, RRC signaling or TB) is received successfully by the receiving node, which means the receiving side correctly detect and decode it. In this scenario, “receive” may cover “detect” and “decode” or may indicates same thing, e.g., “receive paging” means decoding paging correctly and obtaining the paging successfully, accordingly, “the receiving side does not receive paging” means the receiving side does not detect and / or decoding the paging, “paging is not received” means the receiving side tries to detect and / or decoding the paging, but not obtain the paging successfully. The term “receive” may sometimes indicate that a signal arrives at the receiving side, but does not mean the information in the signal is detected and decoded correctly, then the receiving side need perform detecting and decoding on the signal to obtain the informationcarried in the signal. In this scenario, “receive”, “detect” and “decode” may indicate different procedure at receiving side to obtain the information.

[0243] Although this disclosure refers to illustrative embodiments, this is not intended to be construed in a limiting sense. Various modifications and combinations of the illustrative embodiments, as well as other embodiments of the disclosure, will be apparent to persons skilled in the art upon reference to the description. When combining two or more embodiments, not all the features in the embodiments to be combined are necessary for the combination.

[0244] Features disclosed herein in the context of any particular embodiments may also or instead be implemented in other embodiments. Method embodiments, for example, may also or instead be implemented in apparatus, system, and / or computer program product embodiments. In addition, although embodiments are described primarily in the context of methods and apparatus, other embodiments are also contemplated, as instructions stored on one or more non-transitory computer-readable media, for example. Such media could store programming or instructions to perform any of various methods consistent with the present disclosure.

Claims

CLAIMS1. An image rendering method, applied to the GPU of an electronic device, wherein the method comprises:obtaining a first image to be rendered, wherein the first image comprises N pixel fragments, each pixel fragment comprising M pixels; N is a positive integer, M is a positive integer greater than 1 ; obtaining a first color information for rendering the first image, wherein the first color information comprises Q first chrominance information and M first luma information corresponding to each of the N pixel fragments, M is larger than Q;rendering the N pixel fragments based on the first color information to obtain a rendered image of the first image;sending the rendered image to a display device, wherein the rendered image is used for display on the display device;wherein, the rendering the N pixel fragments based on the first color information comprises: rendering M pixels in each of the pixel fragments based on Q first chrominance information corresponding to each of the pixel fragments, and rendering M pixels in each of the pixel fragments based on M first luma information corresponding to each of the pixel fragments.

2. The method of claim 1, wherein the sending the rendered image to a display device comprises: compressing the rendered image to obtain a compressed version of the rendered image, and sending the compressed version of the rendered image to the display device as a video frame in a video stream.

3. The method of claim 1, wherein the ratio of N to M is one of the following ratios: 1:2; 2:4; 1:4.

4. The method of claim 1, wherein the sending the rendered image to a display device comprises: obtaining a third image, wherein the third image and the first image are of the same subsampled format;combining the third image with the first image to obtain a combined image; andsending the combined image to a display device.

5. The method of claim 4, wherein the third image comprises N pixel fragments, each pixel fragment of the third image comprises M pixels; the third image has second color information, the second color information comprises Q second chrominance information and M second luma information corresponding to each of the N pixel fragments;the combining the third image with the first image to obtain a combined image comprises:PCI7RU2025 / 000024combining Q first chrominance information of the i th pixel fragment in the first image with Q second chrominance information of the i th pixel fragment in the third image to obtain the i th combined chrominance information; i is a positive integer greater than or equal to 1 and less than or equal to M;combining M first luma information of the i th pixel fragment in the first image with M second luma information of the i th pixel fragment in the third image to obtain i th set of combined luma information, each set of combined luma information comprising M combined luma information; anddetermining the chrominance of each pixel in the i th pixel fragment of the combined image based on the i th combined chrominance information, and determining the luma of M pixels in the i th pixel fragment of the combined image based on each of the M combined luma information in the i th set of combined luma information, to obtain the combined image.

6. The method of claim 4, wherein the sending the combined image to a display device comprises: inputting the combined image into a deep learning model, to optimize quality parameters of the combined image, and obtain the optimized image, wherein the quality parameters include at least one of the following parameters: resolution, sharpness, color saturation, signal-to-noise ratio;sending the optimized image to a display device.

7. The method of claim 1, wherein the obtaining the second image based on the rendered image comprises:performing a full sampled format conversion on the rendered image to obtain the second image.

8. The method of claim 1, wherein the obtaining the first color information for rendering the first image comprises:obtaining the first color information for rendering the first image when the subsampled rendering mode of the electronic device is configured to be enabled.

9. The method of claim 1, wherein the obtaining the first color information for rendering the first image comprises:obtaining the third color information, the third color information comprising RGB color information corresponding to each pixel of the N pixel fragments; andperforming a subsampled format conversion on the third color information to obtain the first color information for rendering the first image.

10. The method of claim 1, wherein the rendering the N pixel fragments based on the first color information to obtain a rendered image of the first image comprises:rendering the N pixel fragments through a subsampled format shader based on the first color information to obtain a rendered image.

11. An apparatus, comprising:at least one processor couple with at least one memory storing computer program codes; wherein when the computer program codes are executed by the at least one processor, cause the first device to:obtain a first image to be rendered, wherein the first image comprises N pixel fragments, each pixel fragment comprising M pixels; N is a positive integer, M is a positive integer greater than 1;obtain a first color information for rendering the first image, wherein the first color information comprises Q first chrominance information and M first luma information corresponding to each of the N pixel fragments, M is larger than Q;render the N pixel fragments based on the first color information to obtain a rendered image of the first image;send the rendered image to a display device, wherein the rendered image is used for display on the display device;wherein, the render the N pixel fragments based on the first color information comprises: render M pixels in each of the pixel fragments based on Q first chrominance information corresponding to each of the pixel fragments, and rendering M pixels in each of the pixel fragments based on M first luma information corresponding to each of the pixel fragments.

12. The apparatus of claim 1, wherein the send the rendered image to a display device comprises: compress the rendered image to obtain a compressed version of the rendered image, and sending the compressed version of the rendered image to the display device as a video frame in a video stream.

13. The apparatus of claim 1, wherein the ratio ofN to M is one of the following ratios: 1:2; 2:4; 1:4.

14. The apparatus of claim 1, wherein the send the rendered image to a display device comprises: obtain a third image, wherein the third image and the first image are of the same subsampled format; combine the third image with the first image to obtain a combined image; andsend the combined image to a display device.

15. The apparatus of claim 4, wherein the third image comprises N pixel fragments, each pixel fragment of the third image comprises M pixels; the third image has second color information, the second color information comprises Q second chrominance information and M second luma information corresponding to each of the N pixel fragments;the combine the third image with the first image to obtain a combined image comprises:combine Q first chrominance information of the i th pixel fragment in the first image with Q second chrominance information of the i th pixel fragment in the third image to obtain the i th combined chrominance information; i is a positive integer greater than or equal to 1 and less than or equal to M;combine M first luma information of the i th pixel fragment in the first image with M second luma information of the i th pixel fragment in the third image to obtain i th set of combined luma information, each set of combined luma information comprising M combined luma information; anddetermine the chrominance of each pixel in the i th pixel fragment of the combined image based on the i th combined chrominance information, and determine the luma of M pixels in the i th pixel fragment of the combined image based on each of the M combined luma information in the i th set of combined luma information, to obtain the combined image.

16. The apparatus of claim 4, wherein the send the combined image to a display device comprises: input the combined image into a deep learning model, to optimize quality parameters of the combined image, and obtain the optimized image, wherein the quality parameters include at least one of the following parameters: resolution, sharpness, color saturation, signal-to-noise ratio;send the optimized image to a display device.

17. The apparatus of claim 1, wherein the obtain the second image based on the rendered image comprises:perform a full sampled format conversion on the rendered image to obtain the second image.

18. The apparatus of claim 1, wherein the obtain the first color information for rendering the first image comprises:obtain the first color information for rendering the first image when the subsampled rendering mode of the electronic device is configured to be enabled.

19. The apparatus of claim 1, wherein the obtain the first color information for rendering the first image comprises:obtain the third color information, the third color information comprising RGB color information corresponding to each pixel of the N pixel fragments; andperform a subsampled format conversion on the third color information to obtain the first color information for rendering the first image.

20. The apparatus of claim 1, wherein the render the N pixel fragments based on the first color information to obtain a rendered image of the first image comprises:render the N pixel fragments through a subsampled format shader based on the first color information to obtain a rendered image.

21. A computing device cluster, comprising at least one computing device, wherein the computing device comprises at least one processor and a memory coupled with the processor, wherein the memory stores instructions which, when executed by the at least one processor, cause the computing device cluster to execute the method according to any one of claims 1 to 10.

22. A computer-readable medium storing computer execution instructions which, when run on a computing device cluster, cause the computing device cluster to execute the method according to any one of claims 1 to 10.

23. A computer program product comprising computer execution instructions which, when run on a computing device cluster, cause the computing device cluster to execute the method according to any one of claims 1 to 10.

Images