Image processing method, electronic device, and storage medium

By predicting the degree of image quality loss and selecting appropriate image compression algorithms, the problem of increased device load caused by high-load application AI processing is solved, enabling more efficient image processing in electronic devices.

WO2026232914A1PCT designated stage Publication Date: 2026-11-12HONOR DEVICE CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2026/079773
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-08
Filing Date
2026-02-24
Publication Date
2026-11-12

AI Technical Summary

Technical Problem

When AI processes the interface images of high-load applications in electronic devices, it can increase the device's load and affect system performance.

Method used

By predicting the degree of image quality loss, a suitable image compression algorithm is determined, the image is compressed, and AI technology is used to process the compressed image, reducing data transmission volume and device load.

Benefits of technology

This reduces the amount of data transmitted by electronic devices during AI processing, lowers device load, and improves system performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2026079773_12112026_PF_FP_ABST
    Figure CN2026079773_12112026_PF_FP_ABST
Patent Text Reader

Abstract

The present application relates to the technical field of terminals, and provides an image processing method, an electronic device, and a storage medium. The image processing method can be applied to an electronic device. The method comprises: acquiring a first image in a first application program; performing image quality loss degree prediction processing on the first image to obtain a first loss value corresponding to the first image, the first loss value being used for representing the image quality loss degree of the first image; determining an image compression algorithm corresponding to the first loss value; on the basis of the image compression algorithm, performing image compression processing on the first image to obtain a first compressed image; and on the basis of artificial intelligence (AI) technology, performing AI processing on the first compressed image, and then restoring the first compressed image to obtain an AI-processed first image. In the technical solutions shown in the present application, the electronic device determines an appropriate image compression algorithm by predicting an image quality loss degree, to compress the image, and performs AI processing on the compressed image. In this way, during a process in which the electronic device performs AI processing on the image, the amount of data transmitted by the electronic device is reduced, the load on the electronic device is reduced.
Need to check novelty before this filing date? Find Prior Art

Description

An image processing method, an electronic device, and a storage medium

[0001] This application claims priority to Chinese Patent Application No. 202510591779.7, filed with the State Intellectual Property Office of China on May 8, 2025, entitled "An Image Processing Method, Electronic Device and Storage Medium", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of terminal technology, and in particular to an image processing method, electronic device and storage medium. Background Technology

[0003] Artificial intelligence (AI) is a technology that can simulate, extend, and enhance human intelligent behavior. AI technology can achieve autonomous and effective behavior in specific environments or tasks through its perception, reasoning, learning, and decision-making capabilities, thereby solving complex problems and completing specific tasks.

[0004] Electronic devices often use AI models to process the interface images of some applications (such as game applications) to achieve functions such as image quality enhancement, image style conversion, and scene recognition.

[0005] If the application is a high-load application, further processing of its interface images based on AI models will increase the load on the electronic device and affect its system performance. Therefore, it is essential to reduce the load during the process of electronic devices processing interface images based on AI models. Summary of the Invention

[0006] This application provides an image processing method, an electronic device, and a storage medium, which can reduce the amount of data transmitted by the electronic device and reduce the load on the electronic device during the AI ​​processing of images.

[0007] In a first aspect, embodiments of this application provide an image processing method applied to an electronic device, comprising: acquiring a first image in a first application; performing image quality loss prediction processing on the first image to obtain a first loss value corresponding to the first image, the first loss value being used to represent the degree of image quality loss of the first image; determining an image compression algorithm corresponding to the first loss value; performing image compression processing on the first image based on the image compression algorithm to obtain a first compressed image; and performing AI processing on the first compressed image based on artificial intelligence (AI) technology to restore the first compressed image, thereby obtaining the AI-processed first image.

[0008] The image processing method provided in this application determines a suitable image compression algorithm based on the prediction of the degree of image quality loss, compresses the image, and performs AI processing on the compressed image. In this way, the amount of data transmitted by the electronic device is reduced and the load on the electronic device is lowered during the AI ​​processing of the image.

[0009] In one implementation, before acquiring the first image from the first application, the method further includes loading the first application, which includes a game application. Using this implementation, the electronic device can determine a suitable image compression algorithm based on a predicted degree of image quality loss in a game scenario to compress the image, and then perform AI processing on the compressed image. This reduces the amount of data transmitted by the electronic device and lowers its load during the AI ​​image processing.

[0010] In one implementation, a first image is subjected to image quality loss prediction processing to obtain a first loss value corresponding to the first image. This includes: acquiring at least one first pixel in a target region of the first image, wherein the first image includes an RGB image created based on a first color space, and if the first image is an RGB image, the first pixel has an RGB pixel format; mapping the target region from the first color space to a second color space, so that the first pixel with RGB format is converted to a second pixel with YUV format; mapping the target region from the second color space to the first color space, so that the second pixel with YUV format is converted to a third pixel with RGB format; and determining the average Euclidean distance between the first pixel and the third pixel as the first loss value. Using this implementation, the electronic device can predict the degree of image quality loss to determine a suitable image compression algorithm, then compress the image, and perform AI processing on the compressed image. This reduces the data transmission volume and load of the electronic device during the AI ​​processing of the image.

[0011] In one implementation, determining the image compression algorithm corresponding to a first loss value includes: if the first loss value is greater than a first loss threshold, determining that the first loss value corresponds to a first image compression algorithm, which is used to compress the image to a first precision; if the first loss value is less than or equal to the first loss threshold, determining that the first loss value corresponds to a second image compression algorithm, which is used to compress the image to a second precision, where the first precision is greater than the second precision. Using this implementation, the electronic device can select a suitable compression algorithm to compress the image and then perform AI processing on the compressed image. This reduces the amount of data transmitted by the electronic device and lowers its load during the AI ​​processing of the image.

[0012] In one implementation, the image compression algorithm includes a color space conversion algorithm; the first image compression algorithm includes a first color space conversion algorithm corresponding to a first preset standard, which is used for calculations based on floating-point numbers; the second image compression algorithm includes a second color space conversion algorithm corresponding to a second preset standard, which is used for calculations based on integers. This implementation illustrates a specific image compression algorithm for selection by electronic devices. In this way, the electronic device can compress images and perform AI processing on the compressed images, thereby reducing the data transmission volume and load on the electronic device during the AI ​​processing.

[0013] In one implementation, based on an image compression algorithm, a first image is compressed to obtain a first compressed image. This includes: if the image compression algorithm is a color space conversion algorithm, mapping the first image from a first color space to a second color space based on the color space conversion algorithm; and separating the luminance channel of the second color space in the first image to obtain the first compressed image. This implementation illustrates a specific method for an electronic device to compress images. This allows the electronic device to perform AI processing on the compressed image, thereby reducing the data transmission volume and load on the electronic device during the AI ​​image processing process.

[0014] In one implementation, separating the luminance channel of a second color space in a first image to obtain a first compressed image includes: separating the luminance channel and chroma channel of the second color space in the first image, and performing linear interpolation on the chroma channel to obtain the first compressed image. This implementation illustrates a specific method by which an electronic device compresses an image. This allows the electronic device to perform AI processing on the compressed image, thereby reducing the amount of data transmitted by the electronic device and lowering its load during the AI ​​processing.

[0015] In one implementation, after performing AI processing on a first compressed image using artificial intelligence (AI) technology, the first compressed image is restored to obtain an AI-processed first image. This includes: after performing AI processing on the first compressed image using AI technology, mapping the first compressed image from a second color space to a first color space to obtain the AI-processed first image. This implementation illustrates a specific method for restoring a compressed image. Thus, during the AI ​​processing of an image by an electronic device, the electronic device can simultaneously compress and restore the first image, reducing the amount of data transmitted by the electronic device and lowering its load.

[0016] In one implementation, loading a first application includes: the first application invoking a rendering thread to initialize a preset AI model; the first application configuring the initialized AI model in a neural network processor (NPU); after completing the configuration of the AI ​​model, the first application determining the rendering method corresponding to the first image, whereby the rendering method includes forward rendering or deferred rendering; the first application configuring the rendering method in a graphics driver; and the graphics driver issuing a first rendering instruction to a graphics processing unit (GPU) based on the rendering method. This implementation illustrates a specific method for loading a first application into an electronic device. Thus, during the loading process, the electronic device completes the configuration of the AI ​​model, enabling AI processing of compressed images, thereby reducing the data transmission volume and load on the electronic device.

[0017] In one implementation, the first application calls the rendering thread to initialize a preset AI model, including: the first application detecting the runtime environment of the AI ​​model; after the runtime environment detection is complete, the first application loads the preset AI model and performs model compilation processing on the AI ​​model; after the model compilation processing is completed, the first application configures the AI ​​model in memory so that the AI ​​model can be configured on the NPU. This implementation illustrates a specific way for an electronic device to configure an AI model. In this way, the electronic device can perform AI processing on compressed images, and during the AI ​​processing of compressed images, the amount of data transmitted by the electronic device is reduced, thus lowering the load on the electronic device.

[0018] In one implementation, the first application configures the initialized AI model after the neural network processor (NPU), and further includes: the NPU asynchronously launching the AI ​​model. This implementation illustrates a specific method by which an electronic device launches an AI model. In this way, the electronic device can perform AI processing on compressed images, and during the AI ​​processing of compressed images, the amount of data transmitted by the electronic device is reduced, thus lowering the load on the electronic device.

[0019] In one implementation, acquiring a first image from a first application includes: in response to a first rendering instruction, the GPU renders the first image; after rendering the first image, the GPU stores the first image in its buffer. This implementation illustrates a specific method by which an electronic device acquires an image. This allows the electronic device to further determine a suitable image compression algorithm based on a predicted degree of image quality loss, compress the image, and then perform AI processing on the compressed image. During the AI ​​processing of the image by the electronic device, the amount of data transmitted by the electronic device is reduced, thus lowering the load on the electronic device.

[0020] In one implementation, a first image undergoes image quality loss prediction processing to obtain a first loss value. This includes: the GPU acquiring a first pixel in a target region; the GPU mapping the target region from a first color space to a second color space, converting the first pixel (RGB format) into a second pixel (YUV format); the GPU mapping the target region from the second color space back to the first color space, converting the second pixel (YUV format) into a third pixel (RGB format); and the GPU determining the average Euclidean distance between the first and third pixels as the first loss value. Using this implementation, electronic devices can predict the degree of image quality loss to determine a suitable image compression algorithm, compress the image, and then perform AI processing on the compressed image. This reduces the amount of data transmitted by the electronic device and lowers its load during the AI ​​processing.

[0021] In one implementation, determining the image compression algorithm corresponding to the first loss value includes: if the first loss value is greater than a first loss threshold, the GPU determines that the first loss value corresponds to a first image compression algorithm; if the first loss value is less than or equal to the first loss threshold, the GPU determines that the first loss value corresponds to a second image compression algorithm. Using this implementation, the electronic device can select a suitable compression algorithm to compress the image and then perform AI processing on the compressed image. This reduces the amount of data transmitted by the electronic device and lowers its load during the AI ​​processing of the image.

[0022] In one implementation, based on an image compression algorithm, a first compressed image is obtained by compressing the first image. This includes: the GPU mapping the first image from a first color space to a second color space using a color space conversion algorithm; and the GPU separating the luminance channel of the second color space from the first image to obtain the first compressed image. This implementation illustrates a specific method for image compression by an electronic device. This allows the electronic device to perform AI processing on the compressed image, thereby reducing the data transmission volume and load on the electronic device during the AI ​​image processing process.

[0023] In one implementation, the GPU separates the luminance channel of a second color space in a first image to obtain a first compressed image. This includes: the GPU separating the luminance channel and chroma channel of the second color space in the first image, and performing linear interpolation on the chroma channel to obtain the first compressed image. This implementation illustrates a specific method by which an electronic device compresses images. This allows the electronic device to perform AI processing on the compressed image, thereby reducing the amount of data transmitted by the electronic device and lowering its load during the AI ​​processing.

[0024] In one implementation, after AI processing of a first compressed image using artificial intelligence (AI) technology, the first compressed image is restored to obtain an AI-processed first image. This includes: the GPU mapping the first compressed image to shared memory; the NPU accessing the shared memory to use the first compressed image as input to an AI model; the NPU performing AI processing on the first compressed image based on the AI ​​model and writing the output AI-processed first compressed image into the shared memory; and the GPU retrieving the AI-processed first compressed image from the shared memory and mapping it from a second color space to a first color space to obtain the AI-processed first image. This implementation demonstrates a specific method for electronic devices to perform AI processing on images. Thus, during the AI ​​processing of images by the electronic device, the device can compress and restore the first image, reducing the data transmission volume and lowering the load on the electronic device.

[0025] In one implementation, the GPU retrieves the AI-processed first compressed image from shared memory and maps it from a second color space to a first color space to obtain the AI-processed first image. This includes: if the AI-processed first compressed image includes chroma channels based on linear interpolation, after retrieving the AI-processed first compressed image from shared memory, the GPU reconstructs the chroma channels based on linear interpolation to complete the mapping from the second color space to the first color space, thus obtaining the AI-processed first image. This implementation demonstrates a specific method for an electronic device to perform AI processing on an image. In this way, during the AI ​​processing of an image by the electronic device, the device can compress and restore the first image, reducing the amount of data transmitted and lowering the load on the electronic device.

[0026] In one implementation, the AI ​​model includes an AI super-resolution model, which is generated based on an offline training process. Using this implementation, electronic devices can perform AI super-resolution processing on interface images to improve their resolution. The electronic devices support AI processing of images in this scenario to reduce data transmission volume and lower the load on the electronic devices.

[0027] Secondly, embodiments of this application provide an image processing apparatus, which includes an acquisition module for acquiring a first image from a first application; a prediction module for performing image quality loss prediction processing on the first image to obtain a first loss value corresponding to the first image, the first loss value representing the degree of image quality loss of the first image; a determination module for determining an image compression algorithm corresponding to the first loss value; a compression module for performing image compression processing on the first image based on the image compression algorithm to obtain a first compressed image; and a processing module for performing AI processing on the first compressed image based on AI technology and then restoring the first compressed image to obtain an AI-processed first image.

[0028] The image processing apparatus provided in this application determines a suitable image compression algorithm based on the prediction of the degree of image quality loss, compresses the image, and performs AI processing on the compressed image. In this way, the amount of data transmitted by the electronic device is reduced and the load on the electronic device is reduced during the AI ​​processing of the image.

[0029] Thirdly, embodiments of this application provide an electronic device, which includes a touch screen, a memory, and one or more processors; the touch screen, the memory, and the processors are coupled; wherein the memory stores computer program code, which includes computer instructions, and when the computer instructions are executed by the processor, the electronic device performs the image processing method provided by the first aspect and any implementation thereof.

[0030] Fourthly, embodiments of this application provide a computer-readable storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform an image processing method as provided in the first aspect and any implementation thereof.

[0031] Fifthly, embodiments of this application provide a computer program product that, when run on an electronic device, causes the electronic device to execute the image processing method provided in the first aspect and any of its implementations.

[0032] It is understood that the beneficial effects that the technical solutions provided in the third to fifth aspects above can achieve can be referred to the beneficial effects in the first aspect and any of its implementation methods, and will not be repeated here.

[0033] It is understood that the beneficial effects that the technical solutions provided in the third to fifth aspects above can achieve can be referred to the beneficial effects in the first aspect and any of its possible implementations, and will not be repeated here. Attached Figure Description

[0034] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application;

[0036] Figure 2 is a schematic diagram of the software structure of the electronic device provided in an embodiment of this application;

[0037] Figure 3 is a schematic flowchart of the first image processing method provided in an embodiment of this application;

[0038] Figure 4 is a first interactive schematic diagram of the image processing method provided in an embodiment of this application;

[0039] Figure 5 is a schematic diagram of a scenario where an electronic device loads a first application according to an embodiment of this application;

[0040] Figure 6 is a flowchart illustrating the offline training process of the AI ​​super-resolution model provided in the embodiments of this application;

[0041] Figure 7 is a schematic diagram of the interaction between the GPU and the NPU;

[0042] Figure 8 is a schematic diagram of game images with different degrees of image quality loss provided in the embodiments of this application;

[0043] Figure 9 is a schematic diagram of the first target area provided in an embodiment of this application;

[0044] Figure 10 is a schematic diagram of the first scene in which the electronic device provided in the embodiment of this application maps the first target area from the first color space to the second color space;

[0045] Figure 11 is a schematic diagram of a second scene in which the electronic device provided in the embodiment of this application maps the first target area from the first color space to the second color space;

[0046] Figure 12 is a schematic diagram of a third scene in which the electronic device provided in the embodiment of this application maps the first target area from the first color space to the second color space;

[0047] Figure 13 is a scene diagram of the first target image obtained by the electronic device after AI processing according to an embodiment of this application;

[0048] Figure 14 is a second flowchart illustrating the image processing method provided in an embodiment of this application;

[0049] Figure 15 is a second interactive schematic diagram of the image processing method provided in the embodiments of this application;

[0050] Figure 16 is a third flowchart of the image processing method provided in the embodiments of this application;

[0051] Figure 17 is a schematic diagram of the image processing apparatus provided in an embodiment of this application;

[0052] Figure 18 is a schematic diagram of the structure of an image processing apparatus provided in another embodiment of this application;

[0053] Figure 19 is a schematic diagram of the chip system provided in an embodiment of this application. Detailed Implementation

[0054] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are all within the protection scope of this application.

[0055] In the description of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. The "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Furthermore, "at least one" means one or more, and "multiple" means two or more. The terms "first," "second," etc., do not limit the quantity or order of execution, and "first," "second," etc., do not necessarily imply differences.

[0056] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0057] The application scenarios of the embodiments of this application will be described below with reference to the accompanying drawings.

[0058] Electronic devices often use AI models to process the interface images of some applications (such as game applications) to achieve functions such as image quality enhancement, image style conversion, and scene recognition.

[0059] If the application is a high-load application, further processing of its interface images based on AI models will increase the load on the electronic device and affect its system performance. Therefore, it is essential to reduce the load during the process of electronic devices processing interface images based on AI models.

[0060] In order to reduce the amount of data transmitted by electronic devices and reduce the load on electronic devices during AI processing of images, this application provides an image processing method.

[0061] The image processing method provided in this application can be applied to electronic devices. These electronic devices include, but are not limited to, mobile phones, tablets, personal computers, personal digital assistants (PDAs), workstations, large-screen devices (e.g., smart screens, smart TVs), wearable devices (e.g., smart bracelets, smartwatches), handheld game consoles, home game consoles, virtual reality (VR) devices, augmented reality (AR) devices, mixed reality devices, and in-vehicle intelligent terminals. This application does not limit the specific technology or form of the electronic device. The electronic devices involved in this application can be equipped with... Harmony This application does not restrict the use of other operating systems.

[0062] Figure 1 is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application.

[0063] As shown in Figure 1, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a Universal Serial Bus (USB) interface 130, antennas 01 and 02, a mobile communication module 150, a wireless communication module 160, a sensor module 180, a camera 192, a display screen 193, etc. The sensor module 180 may include a touch sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a geomagnetic sensor 180D, an accelerometer sensor 180E, a proximity sensor 180F, a proximity light sensor 180G, etc.

[0064] Processor 110 may include one or more processing units, such as a central processing unit (CPU), application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, video codec, digital signal processor (DSP), baseband processor, neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0065] In this embodiment, the GPU can be used to process graphics rendering and image processing tasks. For example, in a game scene, the GPU can render 3D models and scenes to provide an immersive gaming experience. The GPU can also perform graphics pipeline processing to ensure real-time rendering of game visuals. The GPU can also execute shader programs to control the visual effects of graphics rendering. In this embodiment, the GPU can be used to draw a first image.

[0066] In this embodiment, the NPU can draw inspiration from biological neural network structures, such as the transmission patterns between neurons in the human brain, to rapidly process input information and continuously learn on its own. The NPU can enable intelligent cognitive applications in the electronic device 100, such as image recognition, facial recognition, speech recognition, and text understanding. The NPU in this embodiment can also perform functions such as image quality enhancement, image style conversion, and scene recognition based on AI models.

[0067] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0068] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record video in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0069] DSPs are used to process digital signals. In addition to processing digital image signals, they can also process other digital signals. For example, when electronic device 100 selects a frequency point, a digital signal processor is used to perform Fourier transforms on the frequency point energy, etc.

[0070] In some embodiments, processor 110 may include one or more interfaces. Interfaces may include Inter-Integrated Circuit (I2C) interfaces, Inter-Integrated Circuit Sound (I2S) interfaces, Pulse Code Modulation (PCM) interfaces, Universal Asynchronous Receiver / Transmitter (UART) interfaces, Mobile Industry Processor Interface (MIPI) interfaces, General-Purpose Input / Output (GPIO) interfaces, Subscriber Identity Module (SIM) interfaces, and / or Universal Serial Bus (USB) interfaces, etc. These interfaces are used to connect to other components in electronic device 100.

[0071] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C buses. The processor 110 can couple to the touch sensor 180A, charger, flash, camera 192, etc., through different I2C bus interfaces. For example, the processor 110 can couple to the touch sensor 180A through the I2C interface, enabling the processor 110 and the touch sensor 180A to communicate through the I2C bus interface, thereby realizing the touch function of the electronic device 100.

[0072] The I2S interface can be used for audio communication. In some embodiments, the processor 110 may include multiple I2S buses.

[0073] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals.

[0074] The UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 via the UART interface to implement Bluetooth functionality.

[0075] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display screen 193 and the camera 192. The MIPI interface includes a Camera Serial Interface (CSI) and a Display Serial Interface (DSI). In some embodiments, the processor 110 and the camera 192 communicate via the CSI interface to enable the electronic device 100 to capture images. The processor 110 and the display screen 193 communicate via the DSI interface to enable the electronic device 100 to display images.

[0076] The GPIO interface can be configured via software. It can be configured as a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 to a camera 192, a display screen 193, a wireless communication module 160, a sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0077] USB port 130 is a USB standard compliant interface, specifically a Mini USB port, Micro USB port, USB Type-C port, etc. USB port 130 can be used to connect a charger to charge electronic device 100, and can also be used for data transfer between electronic device 100 and peripheral devices. It can also be used to connect headphones for audio playback. This interface can also be used to connect other electronic devices, such as AR devices.

[0078] It is understood that the interface connection relationships between the modules illustrated in the embodiments of the present invention are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0079] The memory may include internal memory 121. Internal memory 121 may include one or more random access memory (RAM) and one or more non-volatile memory (NVM).

[0080] Random access memory (RAM) can be directly read and written by the processor 110. It can be used to store executable programs (such as machine instructions) of the operating system or other running programs, as well as user and application data. Non-volatile memory can also store executable programs and user and application data, and can be pre-loaded into RAM for direct read and write by the processor 110.

[0081] Random access memory can include static random-access memory (SRAM), dynamic random-access memory (DRAM), synchronous dynamic random-access memory (SDRAM), and double data rate synchronous dynamic random-access memory (DDR SDRAM, such as fifth-generation DDR SDRAM, which is generally called DDR5 SDRAM).

[0082] In this embodiment, the shared memory can be set in DDR SDRAM, for example. Shared memory is a memory area allocated by the operating system, which is used to store currently running programs and data. It has the characteristics of fast read and write and can support efficient memory access.

[0083] The external memory interface 120 can be used to connect to external non-volatile memory, thereby expanding the storage capacity of the electronic device 100. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to perform data storage functions.

[0084] The wireless communication function of electronic device 100 can be implemented through antenna 01, antenna 02, mobile communication module 150, wireless communication module 160, modem processor, and baseband processor.

[0085] Antennas 01 and 02 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 01 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0086] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low-noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 01, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 01. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0087] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs a sound signal through an audio device or displays an image or video through the display screen 193. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 150 or other functional modules.

[0088] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including Wireless Local Area Networks (WLANs) (such as Wireless Fidelity (Wi-Fi) networks), Bluetooth (BT), Global Navigation Satellite System (GNSS), Frequency Modulation (FM), Near Field Communication (NFC), and Infrared (IR) technologies. The wireless communication module 160 receives electromagnetic waves via antenna 02, modulates and filters the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, modulate and amplify them, and then convert them into electromagnetic waves for radiation via antenna 02.

[0089] In some embodiments, the antenna 01 of the electronic device 100 is coupled to the mobile communication module 110, and the antenna 02 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology.

[0090] Electronic device 100 implements display functions through a GPU, a display screen 193, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 193 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0091] The display screen 193 is used to display images, videos, etc. The display screen 193 includes a display panel. In some embodiments, the electronic device may include one or N display screens 193, where N is a positive integer greater than 1.

[0092] In this embodiment of the application, the display screen 193 can be used to display a first image.

[0093] Electronic device 100 can perform shooting functions through ISP, camera 192, video codec, GPU, display 193 and application processor.

[0094] The ISP (Image Signal Processor) is used to process data fed back from the camera 192. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set within the camera 192.

[0095] Camera 192 is used to capture still images or videos. An object passes through the lens to generate an optical image that is projected onto a photosensitive element. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP (Internet Service Provider) for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP (Digital Signal Processor) for processing. The DSP converts the digital image signal into image signals in standard formats such as RGB and YUV. In some embodiments, the electronic device 100 may include one or N cameras 192, where N is a positive integer greater than 1.

[0096] Touch sensor 180A, also known as a "touch device," can be located on display screen 193. The touch sensor 180A and display screen 193 together form a touchscreen, also known as a "touchscreen." Touch sensor 180A is used to detect touch operations applied to or near it. The touch sensor can then transmit the detected touch operation to the application processor to determine the type of touch event.

[0097] The gyroscope sensor 180B can be used to determine the motion attitude of the electronic device 100.

[0098] The 180C barometric pressure sensor is used to measure barometric pressure.

[0099] The geomagnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the geomagnetic sensor 180D to detect the opening and closing of the flip cover.

[0100] The 180E accelerometer can detect the magnitude of acceleration of electronic device 100 in various directions (typically three axes). When electronic device 100 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the posture of electronic devices and applied to applications such as screen orientation switching and pedometers.

[0101] A distance sensor 180F is used to measure distance. Electronic device 100 can measure distance via infrared or laser. In some embodiments, during a shooting scene, electronic device 100 can utilize the distance sensor 180F to measure distance for rapid focusing.

[0102] The proximity light sensor 180G may include, for example, a light-emitting diode and a light detector, such as a photodiode.

[0103] The software system of electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This embodiment of the invention uses the layered architecture Android system as an example to exemplify the software structure of electronic device 100.

[0104] Figure 2 is a schematic diagram of the software structure of the electronic device provided in an embodiment of this application.

[0105] As shown in Figure 2, the layered architecture divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0106] The application layer can include a series of application packages.

[0107] As shown in Figure 2, the application package may include applications such as camera, gallery, calendar, call, map, navigation, music, video, and SMS.

[0108] In this embodiment of the application, the application package may include a game application.

[0109] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0110] As shown in Figure 2, the application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, sensor manager, etc.

[0111] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0112] Content providers store and retrieve data, making that data accessible to applications. This data can include videos, images, audio, phone calls made and received, browsing history and bookmarks, phone books, and more.

[0113] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0114] The phone manager is used to provide communication functions for electronic device 100. For example, it manages call status (including connection and disconnection).

[0115] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0116] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of completed downloads or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating electronic devices, and flashing indicator lights.

[0117] The sensor manager is used to monitor data returned by various sensors in the electronic device 100, such as motion sensor data, proximity sensor data, and temperature sensor data. Using the data returned by each sensor, the electronic device 100 can determine whether it is shaking or whether the display screen 193 is obstructed.

[0118] The Android Runtime consists of core libraries and a virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0119] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0120] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0121] System libraries can include multiple functional modules. For example: Surface Manager, Media Libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0122] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0123] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0124] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0125] A 2D graphics engine is a graphics engine for 2D drawing.

[0126] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0127] The kernel layer can include the base kernel, extended kernel, hardware abstraction layer (HAL), and driver layer.

[0128] The basic kernel can implement core functions such as memory management, task management, inter-process communication, and interrupt management based on the memory (Mem) interface, task (Task) interface, inter-process communication (IPC) interface, and interrupt interface.

[0129] The Hardware Abstraction Layer (HAL) can abstract the hardware operation interface, encapsulating the underlying driver interface into a unified Application Programming Interface (API). This simplifies the complexity of hardware operations for applications.

[0130] It should be noted that the embodiments of this application are only illustrated by taking the HAL layer as part of the kernel layer. In fact, the HAL layer can also be independent of the kernel layer, and the embodiments of this application do not limit this.

[0131] The driver layer includes at least the display driver, camera driver, audio driver, sensor driver, and graphics driver.

[0132] In this embodiment of the application, the graphics driver can be used to manage and control the GPU in the computer system, as well as the associated graphics functions and display devices.

[0133] In this embodiment, the display driver can manage the output of a display device (such as display screen 193) and transfer image frames rendered in the GPU from the GPU to the display device. The display driver can cooperate with the graphics driver to control the output of image frames.

[0134] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0135] Figure 3 is a schematic flowchart of the first image processing method provided in the embodiments of this application.

[0136] Figure 4 is a first interactive schematic diagram of the image processing method provided in the embodiments of this application.

[0137] As shown in Figures 3 and 4, in some embodiments, the method includes the following steps S11-S16.

[0138] In step S11, in response to the user's first operation on the first icon, the electronic device 100 loads the first application.

[0139] The first application can be a game application or a non-game application. This embodiment only uses a game application as an example for illustration. The first icon is the application icon of the first application, such as a game icon. The first operation is, for example, a click operation. This embodiment does not limit the application type of the first application, the icon type of the first icon, or the operation type of the first operation.

[0140] Figure 5 is a schematic diagram of a scenario in which an electronic device loads a first application according to an embodiment of this application.

[0141] As shown in Figure 5, the user can click on the game icon 2 corresponding to the game application on the main screen interface 1 of the electronic device 100. In response to the user's click on the game icon 2, the electronic device 100 loads the game application and displays the game loading page 3.

[0142] During the process of displaying the game loading page 3 on the electronic device 100, the electronic device 100 performs the following steps.

[0143] In one implementation, step S11 includes steps S111-S115.

[0144] Step S111: The first application loads the OpenGL ES thread and initializes the AI ​​model in the OpenGL ES thread.

[0145] The Open Graphics Library (OpenGL) is a cross-language, cross-platform API for rendering 2D and 3D vector graphics. OpenGL for Embedded Systems (OpenGL ES) is a subset of OpenGL designed for embedded systems, mobile devices, and platforms with limited computing power. An OpenGL ES thread is a thread bound to the Embedded Graphics Library (EGL) environment, specifically dedicated to performing OpenGL ES rendering operations. Specifically, an OpenGL ES thread is a rendering thread separate from the main thread; it executes OpenGL ES rendering commands to draw graphics and images. Steps S111-S115 in this application are all implemented based on the OpenGL ES thread.

[0146] AI models are models trained using machine learning and deep learning techniques. These models can simulate human cognitive functions to solve specific problems or perform specific tasks.

[0147] In this embodiment, the AI ​​model can be trained offline and then deployed on the electronic device 100. This way, when the first application loads the OpenGL ES thread, the AI ​​model can be loaded into the OpenGL ES thread, thus achieving compatibility between the AI ​​model and the OpenGL ES thread's runtime environment.

[0148] This application uses an AI model as an example to illustrate an AI super-resolution model used to achieve image quality enhancement. In practice, the AI ​​model can also be an AI scene recognition model for scene recognition, an AI style conversion model for image style conversion, etc. This application does not limit the specific type of AI model.

[0149] AI super-resolution model is an image processing model based on deep learning technology. It is used to extract materials from low-resolution images or videos and synthesize high-resolution images in real time.

[0150] The training process of the AI ​​super-resolution model can be implemented on a server or in the cloud, without relying on the computing power of the electronic device 100. This application embodiment illustrates the offline training process of the AI ​​super-resolution model on a server.

[0151] Figure 6 is a flowchart illustrating the offline training process of the AI ​​super-resolution model provided in the embodiments of this application.

[0152] As shown in Figure 6, in one implementation, the offline training process of the AI ​​super-resolution model includes the following steps S01-S04.

[0153] Step S01: The server constructs a data sample set.

[0154] The data sample set can include multiple image pairs consisting of low-resolution and high-resolution images. Based on these image pairs, the model can be trained to learn the ability to reconstruct image details.

[0155] In step S02, the server inputs the data sample set into the AI ​​super-resolution model, trains the AI ​​super-resolution model, and obtains the trained AI super-resolution model.

[0156] In one implementation, the AI ​​super-resolution model may include at least one of the following: Super-Resolution Convolutional Neural Network (SRCNN) and Enhanced Super-Resolution Generative Adversarial Network (ESRGAN).

[0157] Step S03: The server converts the AI ​​super-resolution model into an AI super-resolution model file using conversion tools.

[0158] Step S04: The server deploys the AI ​​super-resolution model file to the electronic device 100.

[0159] In this way, the AI ​​super-resolution model file can be loaded during the initialization of the AI ​​model in the first application.

[0160] Specifically, step S111 includes steps S1111-S1113.

[0161] Step S1111: The first application detects the operating environment of the AI ​​super-resolution model.

[0162] The first application can detect the hardware compatibility of the AI ​​super-resolution model with the GPU and NPU to ensure that the AI ​​super-resolution model can run normally on the electronic device 100.

[0163] Step S1112: After the runtime environment detection is completed, the first application loads the preset AI super-resolution model and performs model compilation processing on the AI ​​super-resolution model.

[0164] The first application can load pre-deployed AI super-resolution model files and compile them into a format that can run on the NPU. This compilation process may include steps such as optimizing the AI ​​super-resolution model's structure, converting the AI ​​super-resolution model's file format, and compiling the AI ​​super-resolution model's code. After the model compilation process, the AI ​​super-resolution model can run efficiently on the NPU.

[0165] Step S1113: After completing the model compilation process, the first application configures the AI ​​super-resolution model in memory so that the AI ​​super-resolution model can be configured on the NPU.

[0166] The first application can allocate appropriate memory resources to the AI ​​super-resolution model so that the AI ​​super-resolution model has sufficient memory resources available during operation.

[0167] After the initialization process of the AI ​​model is completed, the first application can further process the AI ​​model.

[0168] In step S112, the first application configures the initialized AI model on the NPU.

[0169] Step S113: After completing the configuration of the AI ​​model, the first application determines the rendering method corresponding to the first target image. The rendering method includes forward rendering or deferred rendering.

[0170] Step S114: The first application configures the rendering method in the graphics driver.

[0171] In step S115, the graphics driver issues the first rendering instruction to the graphics processor (GPU) based on the rendering method.

[0172] In step S12, the electronic device 100 acquires the first target image from the first application.

[0173] The first target image can be any image to be displayed in the game interface of the game application.

[0174] In one implementation, step S12 includes steps S121-S122.

[0175] In step S121, in response to the first rendering instruction, the GPU renders the first target image.

[0176] Specifically, in response to the first rendering instruction, the GPU can process the computational task corresponding to the first target image through the computer shader.

[0177] A computation shader is a type of shader that does not rely on a traditional graphics rendering pipeline but is used to perform general-purpose computational tasks. This shader can be used to perform tasks such as image processing, physics simulation, and data analysis, and is not limited to graphics rendering. Computation shaders can directly access and manipulate buffers in GPU memory to process data within them, or store data in buffers.

[0178] In step S122, after the GPU finishes rendering the first target image, it stores the first target image in the GPU's buffer.

[0179] After completing the computation task, the GPU stores the image data of the first target image in the shader storage buffer object (SSBO).

[0180] The image data of the first target image can correspond to a lower resolution.

[0181] Shader store buffers are buffer objects in OpenGL used to store and share data within shader programs. As a key component for implementing complex graphics algorithms and general-purpose computing tasks, shader store buffers allow data to be read and written between different shaders, and between shaders and the CPU.

[0182] In this way, the electronic device 100 acquires the first target image based on the GPU.

[0183] Electronic device 100 needs to perform AI processing on the first target image based on the interaction between the GPU and NPU.

[0184] The following provides an example of how the GPU and NPU interact.

[0185] Figure 7 is a schematic diagram of the interaction between the GPU and the NPU.

[0186] As shown in Figure 7, in one implementation, the GPU and NPU interact based on shared memory.

[0187] Shared memory is a memory region in electronic device 100 that can be accessed by multiple processes or threads. This memory region can be located in main memory (e.g., DDR SDRAM). Specifically, electronic device 100 can allocate a region in DDR memory and map this region to the address space of the relevant processes to create shared memory. In this way, multiple processes can exchange data by accessing this shared memory. In the embodiments of this application, electronic device 100 can map the shared memory to the address space of the OpenGL ES thread and to the address space of the AI ​​computing thread, so that both the GPU and NPU can access the shared memory.

[0188] Typically, after the GPU acquires the first target image, it maps it to shared memory. The NPU then accesses this shared memory, using the first target image as input to the AI ​​model. Following this, the NPU performs AI processing on the first target image based on the AI ​​model and writes the processed image back to the shared memory. Thus, the GPU retrieves the processed first target image from the shared memory, performs post-processing on it, and then displays it.

[0189] During this process, both the GPU mapping the first target image to shared memory and the NPU writing the AI-processed first target image to shared memory generate significant data transfer volumes, leading to high latency and high power consumption in the electronic device 100. Furthermore, this process involves multiple steps, including the CPU exposing shared memory to the GPU and NPU through memory mapping, the CPU scheduling multiple threads, and bandwidth allocation for DDR memory. These steps can also cause high CPU and DDR frequencies in the electronic device 100, further impacting its latency and power consumption.

[0190] Therefore, in order to reduce the amount of data transmitted by the electronic device 100 and reduce the load on the electronic device 100, the embodiments of this application first need to process the first target image before implementing the interaction process between the GPU and the NPU, that is, to execute subsequent steps S13-S15.

[0191] In step S13, the electronic device 100 performs image quality loss prediction processing on the first target image to obtain the first target loss value corresponding to the first target image.

[0192] The first target loss value is used to represent the degree of image quality loss of the first target image.

[0193] Image quality loss refers to a decline in the visual quality of an image. Image quality loss typically manifests as loss of detail, reduced sharpness, color distortion, increased noise, compression artifacts, motion blur, and jagged edges. Image quality loss can occur at any stage of an electronic device's image acquisition, processing, compression, transmission, and display.

[0194] Therefore, the first target image may have a certain degree of image quality loss.

[0195] The following is an example illustration based on two game images with different degrees of image quality loss.

[0196] Figure 8 is a schematic diagram of game images with different degrees of image quality loss provided in the embodiments of this application.

[0197] As shown in Figure 8A, the first game image is the image corresponding to the battle scene of the game application. In this scene, the image texture is relatively rich, and the image quality is more likely to be significantly lost during the acquisition, processing, compression, transmission, and display processes by the electronic device 100. The first target image is, for example, the first game image.

[0198] As shown in Figure 8B, the second game image is the image corresponding to the walking scene of the game application. In this scene, there is a large area of ​​blank space, and the degree of image quality loss during the process of image acquisition, processing, compression, transmission and display by the electronic device 100 is relatively small.

[0199] It should be noted that the two game images shown in Figure 8 are only used as examples to illustrate the degree of image quality loss and do not indicate that the two images have a significant difference in image quality during actual display. The electronic device 100 typically handles the image quality loss during the actual display of the game images.

[0200] The following provides a further explanation of how the electronic device 100 determines the degree of image quality loss of the first target image.

[0201] In one implementation, step S13 includes steps S131-S134.

[0202] In step S131, the electronic device 100 acquires at least one first target pixel in the first target region of the first target image.

[0203] The first target image includes an RGB image created based on a first color space. When the first target image is an RGB image, the pixel format of the first target pixel is RGB format.

[0204] Color space is a mathematical model and specification used to quantify, represent, and transmit colors. It defines how colors are represented, including the composition of colors, the range of colors, and the relationships between colors. In this way, colors can be converted into numerical values ​​that can be processed by electronic devices.

[0205] A color space can define elements such as color gamut, color model, and color depth. Color gamut refers to the range of colors that a color space can represent. A color model defines the basic ways colors can be combined. Color depth refers to the numerical precision corresponding to each color channel. In practice, a color space can define many more elements; however, this application does not limit the specific types of elements defined in the color space.

[0206] Color models include, for example, the RGB color model, which is an additive color model corresponding to red, green, and blue light, and is based on the R, G, and B channels. Another example is the Cyan-Magenta-Yellow-Black (CMYK) color model, which is a subtractive color model obtained by superimposing different proportions of cyan, magenta, yellow, and black ink colors. A third example is the lightness-chromaticity (YUV) color model, which is formed by the lightness (Y) channel and the chromaticity (U and V) channels. Color depth, for example, corresponds to 8 bits of numerical precision for each color channel, or 16 bits of numerical precision for each color channel.

[0207] In this embodiment of the application, the first color space can be a color space with an RGB color model and an 8-bit numerical precision color depth.

[0208] After acquiring a first target image created based on a first color space, the electronic device 100 needs to determine a first target region in the first target image.

[0209] It should be noted that the embodiments of this application can determine the degree of image quality loss based on the entire first target image or only based on the first target region. If the electronic device 100 determines the degree of image quality loss based on the entire first target image, the electronic device 100 can obtain a more accurate degree of image quality loss. If the electronic device 100 determines the degree of image quality loss based on the first target region, the electronic device 100 can reduce the amount of computation, thereby reducing power consumption.

[0210] This application embodiment is only used as an example to illustrate how the electronic device 100 determines the degree of image quality loss based on the first target area.

[0211] Figure 9 is a schematic diagram of the first target area provided in an embodiment of this application.

[0212] As shown in Figure 9, for example, when the electronic device 100 acquires the first target image 4, the electronic device 100 can determine the central region 5 of the first target image 4 as the first target region.

[0213] The electronic device 100 designates the central region 5 as the first target region because the first target image, as an image in the game interface of a game application, typically has a more complex texture and more variations in its central region 5. Furthermore, the central region 5 is usually the area that the human eye focuses on. In contrast, the surrounding region 6 of the first target image 4 is typically the area where the UI controls 7 are located. These areas have fewer variations, and the human eye pays less attention to them. Therefore, the degree of image quality loss has a smaller impact on these areas.

[0214] Furthermore, the electronic device 100 samples in the central region 5 to identify at least one sampling point as the first target pixel.

[0215] For example, electronic device 100 can identify first pixel P1, second pixel P2, third pixel P3 and fourth pixel P4 as first target pixels.

[0216] In one implementation, step S131 includes step S1311.

[0217] Step S1311: The GPU acquires at least one first target pixel in the first target region of the first target image.

[0218] In other words, step S131 can be implemented using a GPU.

[0219] In step S132, the electronic device 100 maps the first target area from the first color space to the second color space, so that the first target pixel with RGB format is converted into the second target pixel with YUV format.

[0220] In this embodiment of the application, the second color space can be a color space with a YUV color model.

[0221] Figure 10 is a schematic diagram of the first scene in which the electronic device provided in the embodiment of this application maps the first target area from the first color space to the second color space.

[0222] As shown in Figure 10, the electronic device 100 maps the first target area from the first color space to the second color space, which is actually a process of converting the pixel format of the first target pixel in the first target area.

[0223] In one implementation, the pixel format of the first target pixel can be converted based on a first standard. The first standard is, for example, the ITU-R BT.601 standard. This standard defines the encoding parameters of the video signal, including parameters such as color space, resolution, frame rate, and chroma sampling.

[0224] Specifically, the electronic device 100 can first acquire the red pixel value R corresponding to each first target pixel in the R channel, the green pixel value G corresponding to each target pixel in the G channel, and the blue pixel value B corresponding to each target pixel in the B channel.

[0225] Furthermore, the electronic device 100 can perform pixel format conversion on the first target pixel based on the following first formula-third formula to obtain the converted second target pixel. The second target pixel has the same coordinates as the first target pixel, but the pixel format is different.

[0226] First formula: Y = 0.299R + 0.587G + 0.114B.

[0227] Second formula: U = -0.168736R - 0.331264G + 0.5B + ​​0.5.

[0228] The third formula is: V = 0.5B - 0.418688R - 0.081312G + 0.5.

[0229] The pixel format conversion process of the first pixel P1 is illustrated by example.

[0230] For example, the first pixel P1 corresponds to (R1, G1, B1).

[0231] If R1 = 100, G1 = 150, and B1 = 200, then based on the first formula minus the third formula, we can obtain: Y1 = 0.299 × 100 + 0.587 × 150 + 0.114 × 200 = 185.65. U1 = -0.14713 × 100 - 0.28886 × 150 + 0.436 × 20 = 29.158. V1 = 0.615 × 200 - 0.51499 × 150 - 0.10001 × 100 = 35.7505.

[0232] Then, the electronic device 100 can obtain the first converted pixel point Q1 (i.e., the second target pixel), which corresponds to (Y1, U1, V1) = (185.65, 29.158, 35.7505).

[0233] It should be noted that the values ​​of R1, G1, and B1 shown in the embodiments of this application are for illustrative purposes only and do not constitute a limitation on the actual values.

[0234] Figure 11 is a schematic diagram of a second scene in which the electronic device provided in the embodiment of this application maps the first target area from the first color space to the second color space.

[0235] As shown in Figure 11, in one implementation, the conversion between the first and second color spaces can also be based on a second standard. The second standard could be, for example, the ITU-R BT.709 standard. Different standards define different parameters. The second target pixel obtained based on the second standard has a wider color gamut and better color consistency compared to the second target pixel obtained based on the first standard.

[0236] The electronic device 100 can perform pixel format conversion on the first target pixel based on the following fourth to sixth formulas to obtain the converted second target pixel.

[0237] Fourth formula: Y = 0.2126R + 0.7152G + 0.0722B.

[0238] Fifth formula: U = -0.09991R - 0.33609G + 0.436B + 0.5.

[0239] Sixth formula: V = 0.5B - 0.168736R - 0.331264G + 0.5.

[0240] The pixel format conversion process of the first pixel P1 is illustrated by example.

[0241] For example, the first pixel P1 corresponds to (R1, G1, B1).

[0242] If R1 = 100, G1 = 150, and B1 = 200, then based on formulas 4 and 6, we can obtain: Y1 = 0.2126 × 100 + 0.7152 × 150 + 0.0722 × 200 = 143.98. U1 = -0.09991 × 100 - 0.33609 × 150 + 0.436 × 200 + 0.5 = 27.2965. V1 = 0.5 × 200 - 0.168736 × 100 - 0.331264 × 150 + 0.5 = 34.0368.

[0243] Then, the electronic device 100 can obtain the first converted pixel point Q1 (i.e., the second target pixel), which corresponds to (Y1, U1, V1) = (143.98, 27.2965, 34.0368).

[0244] Figure 12 is a schematic diagram of a third scenario in which the electronic device provided in the embodiment of this application maps the first target area from the first color space to the second color space.

[0245] As shown in Figure 12, in one implementation, the conversion between the first and second color spaces can also be based on a third standard. The third standard could be, for example, the ITU-R BT.2020 standard. The second target pixel obtained based on the third standard has a wider color gamut and better color consistency compared to the second target pixel obtained based on the second standard.

[0246] The electronic device 100 can perform pixel format conversion on the first target pixel based on the following seventh to ninth formulas to obtain the converted second target pixel.

[0247] Seventh formula: Y = 0.2627R + 0.6780G + 0.0593B.

[0248] Eighth formula: U = -0.1396R - 0.3604G + 0.5B.

[0249] Ninth formula: V = 0.5R - 0.4598G - 0.0402B.

[0250] The pixel format conversion process of the first pixel P1 is illustrated by example.

[0251] For example, the first pixel P1 corresponds to (R1, G1, B1).

[0252] If R1 = 100, G1 = 150, and B1 = 200, then based on formula 7 - formula 9, we can obtain: Y1 = 0.2627 × 100 + 0.6780 × 150 + 0.0593 × 200 = 139.83. U1 = -0.1396 × 100 - 0.3604 × 150 + 0.5 × 200 = 31.98. V1 = 0.5 × 100 - 0.4598 × 150 - 0.0402 × 200 = -27.01.

[0253] Then, the electronic device 100 can obtain the first converted pixel point Q1 (i.e., the second target pixel), which corresponds to (Y1, U1, V1) = (139.83, 31.98, -27.01).

[0254] In summary, the electronic device 100 can convert the pixel format of the first target pixel based on three different standards. The specific method of pixel format conversion of the electronic device 100 in this embodiment of the application is not limited.

[0255] In one implementation, step S132 includes step S1321.

[0256] In step S1321, the GPU maps the first target region from the first color space to the second color space, so that the first target pixel with RGB format is converted into the second target pixel with YUV format.

[0257] In other words, step S132 can be implemented using a GPU.

[0258] In step S133, the electronic device 100 maps the first target region from the second color space to the first color space, so that the second target pixel with YUV format is converted into a third target pixel with RGB format.

[0259] Further based on the aforementioned example, when the pixel format of the first target pixel is converted based on the first standard, the electronic device 100 also reversely converts the second target pixel into the third target pixel based on the first standard.

[0260] Electronic device 100 can perform pixel format conversion on the second target pixel based on the following formulas 10-12 to obtain the converted third target pixel.

[0261] The third target pixel has the same coordinates as the second target pixel, but the pixel format is different.

[0262] Formula 10: R = Y + 1.402(V - 0.5).

[0263] Eleventh formula: G = Y - 0.344136(U - 0.5) - 0.714136(V - 0.5).

[0264] The twelfth formula: B = Y + 1.772(U - 0.5).

[0265] The pixel format conversion process of the first converted pixel Q1 is illustrated by way of example.

[0266] For example, the first transformed pixel Q1 corresponds to (Y1, U1, V1).

[0267] If Y1 = 185.65, U1 = 29.158, and V1 = 35.7505, then based on formulas 10 and 12, we can obtain: R2 = 185.65 + 1.402(35.7505 - 0.5) = 235.11. G2 = 185.65 - 0.344136 × (29.158 - 0.5) - 0.714136 × (35.7505 - 0.5) = 150.60. B2 = 185.65 + 1.772 × (29.158 - 0.5) = 236.42.

[0268] Then, the electronic device 100 can obtain the converted first inverted pixel T1 (i.e., the third target pixel), which corresponds to (R2, G2, B2) = (235.11, 150.60, 236.42).

[0269] When the pixel format of the first target pixel is converted based on the second standard, the electronic device 100 also converts the second target pixel in reverse to the third target pixel based on the second standard.

[0270] The electronic device 100 can perform pixel format conversion on the second target pixel based on the following formulas thirteen to fifteen to obtain the converted third target pixel.

[0271] Formula 13: R = Y + 1.779(V - 0.5).

[0272] Fourteenth formula: G = Y - 0.886(U - 0.5) - 0.213(V - 0.5).

[0273] Formula 15: B = Y + 2.115(U - 0.5).

[0274] The specific method by which the electronic device 100 reversely converts the second target pixel into a third target pixel based on the second standard can be referred to in the foregoing embodiments, and will not be repeated here.

[0275] When the pixel format of the first target pixel is converted based on the third standard, the electronic device 100 also converts the second target pixel in reverse to the third target pixel based on the third standard.

[0276] The electronic device 100 can perform pixel format conversion on the second target pixel based on the following sixteenth to eighteenth formulas to obtain the converted third target pixel.

[0277] The sixteenth formula: R = Y + 1.71665 × (V - 0.09901) - 0.87427 × (U - 0.09901).

[0278] Formula 17: G = Y - 0.18709 × (V - 0.09901) - 0.61356 × (U - 0.09901).

[0279] Formula 18: B = Y + 2.11237 × (V - 0.09901).

[0280] The specific method by which the electronic device 100 reversely converts the second target pixel into a third target pixel based on the third standard can be referred to in the foregoing embodiments, and will not be repeated here.

[0281] In one implementation, step S133 includes step S1331.

[0282] In step S1331, the GPU maps the first target region from the second color space to the first color space, so that the second target pixel with YUV format is converted into a third target pixel with RGB format.

[0283] In other words, step S133 can be implemented using a GPU.

[0284] In step S134, the electronic device 100 determines the average Euclidean distance between the first target pixel and the third target pixel as the first target loss value.

[0285] Electronic device 100 can determine the average Euclidean distance based on the following nineteenth formula.

[0286] Nineteenth Formula:

[0287] G 1i Let B be the green pixel value corresponding to the i-th first target pixel in the G channel. 1i R is the blue pixel value corresponding to the i-th first target pixel in the B channel. 2i G represents the red pixel value corresponding to the i-th third target pixel in the R channel. 2i Let B be the green pixel value corresponding to the i-th third target pixel in the G channel. 2iLet be the blue pixel value corresponding to the i-th third target pixel in the B channel. n is the number of first target pixels, and n is a positive integer. i is a positive integer.

[0288] Further based on the aforementioned example, if the electronic device 100 reverse-converts the second target pixel into a third target pixel based on the first standard, and if the first target pixel is a first pixel point P1, R 11 =100, G 11 =150, B 11 =200, then the first third target pixel is the first inverted pixel T1, R 21 =235.11, G 21 =150.60, B 21 =236.42.

[0289] Therefore, the first Euclidean distance D1 between the first target pixel and the first third target pixel is:

[0290] The electronic device 100 can further acquire a second Euclidean distance D2 between a second first target pixel (e.g., second pixel P2) and a second third target pixel (e.g., the second inverted pixel T2 corresponding to second pixel P2), and a third Euclidean distance D2 between a third first target pixel (e.g., third pixel P3) and a third third target pixel (e.g., the third inverted pixel T3 corresponding to third pixel P3), ..., to acquire n nth Euclidean distances D2. n Then, calculate D1-D n The average Euclidean distance D between them.

[0291] For example, the average Euclidean distance is 150.

[0292] Therefore, the electronic device 100 can determine that the first target loss value is 150. In this way, the electronic device 100 can measure the degree of image quality loss of the first target image based on the first target loss value.

[0293] In one implementation, step S134 includes step S1341.

[0294] In step S1341, the GPU determines the average Euclidean distance between the first target pixel and the third target pixel as the first target loss value.

[0295] In other words, step S134 can be implemented using a GPU.

[0296] In step S14, if the first target loss value is greater than the first loss threshold, the electronic device 100 determines that the first target loss value corresponds to the first image compression algorithm.

[0297] Specifically, if the first target loss value is greater than the preset first loss threshold, it indicates that the image quality of the first target image has a large degree of loss. The electronic device needs to select a high-precision image compression algorithm for the first target image, such as the first image compression algorithm, which is used to compress the image to the first precision.

[0298] The electronic device 100 may include various image compression algorithms, including color space conversion algorithms. The color space conversion algorithms include a first color space conversion algorithm corresponding to a first preset standard. The first preset standard is, for example, the third standard in the aforementioned embodiments, namely the ITU-R BT.2020 standard. Therefore, the first color space conversion algorithm can be implemented based on the seventh to ninth formulas mentioned above. In other words, the first color space conversion algorithm is a floating-point number-based color space conversion algorithm, which can reduce image quality loss and improve conversion accuracy during the color conversion process.

[0299] In one implementation, the first loss threshold can be set to 100. This embodiment of the application is only used as an example to illustrate this value, and the specific value of the first loss threshold is not limited in this embodiment of the application.

[0300] Further based on the aforementioned example, when the first target loss value is 150, the electronic device 100 can determine that the first target loss value is greater than the first loss threshold, and thus determine that the first target loss value corresponds to the first color space conversion algorithm.

[0301] In one implementation, step S14 includes step S141.

[0302] In step S141, the GPU determines the first target loss value and its correspondence with the first image compression algorithm.

[0303] In other words, step S14 can be implemented using a GPU.

[0304] Step S15: Based on the first image compression algorithm, the electronic device 100 performs image compression processing on the first target image to obtain the first target compressed image.

[0305] In one implementation, step S15 includes steps S151-S152.

[0306] Step S151: When the image compression algorithm is the first color space conversion algorithm, the electronic device 100 maps the first target image from the first color space to the second color space based on the first color space conversion algorithm.

[0307] Further based on the aforementioned example, the electronic device 100 maps the first target image from the first color space to the second color space, which is actually a process of pixel format conversion of all pixels in the first target image.

[0308] In other words, the electronic device 100 needs to perform pixel format conversion on each RGB format pixel in the first target image based on the seventh formula to the ninth formula, so as to convert each RGB format pixel into a YUV format pixel.

[0309] The process by which the electronic device 100 converts each RGB format pixel in the first target image to a YUV format pixel can be referred to in the foregoing embodiments of the process by which the electronic device 100 performs pixel format conversion on the first target pixel in the first target region. This application embodiment will not elaborate on this process.

[0310] In one implementation, step S151 includes step S1511.

[0311] In step S1511, the GPU maps the first target image from the first color space to the second color space based on the first color space conversion algorithm.

[0312] In other words, step S151 can be implemented using a GPU.

[0313] Specifically, in the OpenGL thread, the GPU can pass vertex coordinates and texture coordinates based on the vertex shader, and set the RGB to YUV conversion formula based on the fragment shader, so as to map the first target image from the first color space to the second color space.

[0314] In step S152, in the first target image, the electronic device 100 separates the Y channel of the second color space to obtain the first target compressed image.

[0315] The reason why the electronic device 100 separates the Y channel to obtain the first target compressed image is that the human eye has a stronger perception of the Y channel and a weaker perception of the U and V channels. Therefore, in this embodiment, only the Y channel image in the first target image needs to be processed, without processing the U and V channel images, thus achieving the effect of data compression and reducing the amount of data transmission between the GPU and NPU.

[0316] In one implementation, step S152 includes step S1521.

[0317] In step S1521, the GPU separates the Y channel of the second color space in the first target image to obtain the first target compressed image.

[0318] In other words, step S152 can be implemented using a GPU.

[0319] Specifically, after the GPU converts each RGB pixel in the first target image to a YUV pixel, the GPU can create a target texture to store the converted YUV pixels. Furthermore, the GPU can bind a frame buffer object (FBO) to the target texture and then read all the pixels of the Y channel from the FBO to obtain the first target compressed image. That is, the first target compressed image is the image corresponding to the Y channel of the first target image.

[0320] The electronic device 100 can also acquire the first target compressed image using other methods.

[0321] In one implementation, step S153 can also be executed after step S151.

[0322] Step S153: In the first target image, separate the Y channel, U channel and V channel of the second color space, and perform linear interpolation on the U channel and V channel to obtain the first target compressed image.

[0323] Linear interpolation is a mathematical method used to estimate the value of an unknown data point between two known data points. In some scenarios, it can achieve data compression. Specifically, in color space conversion scenarios, data often exhibits certain regularity and continuity. Therefore, if the data changes linearly within a certain interval, storing only the start and end points of that interval can significantly reduce the amount of data stored.

[0324] In other words, this embodiment of the application achieves data compression by processing the Y channel image in the first target image and performing linear interpolation processing on the U channel image and V channel image, thereby reducing the amount of data transmission between the GPU and NPU.

[0325] In one implementation, step S153 includes step S1531.

[0326] In step S1531, in the first target image, the GPU separates the Y channel, U channel and V channel of the second color space, and performs linear interpolation on the U channel and V channel to obtain the first target compressed image.

[0327] In other words, step S153 can be implemented using a GPU.

[0328] Specifically, after the GPU converts each RGB pixel in the first target image to a YUV pixel, the GPU can create a target texture to store the converted YUV pixels. Furthermore, the GPU can bind a frame buffer object (FBO) to the target texture, and then read all the pixels in the Y channel, U channel, and V channel from the FBO. The GPU then performs linear interpolation on the pixels in the U channel and V channel to obtain the first target compressed image.

[0329] Step S16: After compressing the first target compressed image based on AI technology, the electronic device 100 restores the first target compressed image to obtain the AI-processed first target image.

[0330] In one implementation, step S16 includes step S161.

[0331] Step S161: After performing AI processing on the first target compressed image based on AI technology, the electronic device 100 maps the first target compressed image from the second color space to the first color space to obtain the AI-processed first target image.

[0332] Figure 13 is a scene diagram of the first target image obtained by the electronic device provided in the embodiment of this application after AI processing.

[0333] As shown in Figure 13, in one implementation, step S161 includes steps S1611-S1614.

[0334] In step S1611, the GPU maps the first target compressed image to shared memory.

[0335] Further based on the aforementioned example, if the first target compressed image is the first target compressed image corresponding to the Y channel, the GPU will map this first target compressed image to shared memory.

[0336] In one implementation, the OpenGL thread can divide the first target compressed image into multiple image blocks in the GPU, and process the image blocks based on multiple sub-thread blocks in the OpenGL thread, so that the first target compressed image is mapped to shared memory. This application does not limit the specific method by which the GPU maps the first target compressed image to shared memory.

[0337] In this way, compared to the method of the GPU mapping the first target image to shared memory, the embodiments of this application reduce the amount of data transfer by about two-thirds.

[0338] In step S1612, the NPU accesses shared memory to use the first target compressed image as input to the AI ​​model.

[0339] Based on the aforementioned example, the NPU can input the first target compressed image into the AI ​​super-resolution model.

[0340] In step S1613, after the NPU performs AI processing on the first target compressed image based on the AI ​​model, it writes the output AI-processed first target compressed image into shared memory.

[0341] Further based on the aforementioned example, after the NPU performs AI super-resolution processing on the first target compressed image based on the AI ​​super-resolution model, it writes the output AI super-resolution processed first target compressed image into shared memory. The AI ​​super-resolution processed first target compressed image is an image with a relatively high resolution.

[0342] In step S1614, the GPU obtains the AI-processed first target compressed image from shared memory and maps the AI-processed first target compressed image from the second color space to the first color space to obtain the AI-processed first target image.

[0343] Further based on the aforementioned example, the GPU can obtain the first target compressed image after AI super-resolution processing from shared memory, that is, the Y channel image after AI super-resolution processing. In this way, the GPU can convert each pixel in the first target image from YUV format to RGB format based on the Y channel image after AI super-resolution processing, the unprocessed U channel image, and the unprocessed V channel image to obtain the first target image after AI processing.

[0344] In fact, steps S1611-S1614 can also be implemented in other ways.

[0345] Further based on the aforementioned example, when the first target compressed image consists of a Y channel image, a U channel image and a V channel image processed by linear interpolation, after the GPU obtains the AI-processed first compressed image from shared memory, it reconstructs the U channel image and the V channel image based on linear interpolation to complete the mapping of the AI-processed first compressed image from the second color space to the first color space, thereby obtaining the AI-processed first target image.

[0346] It should be noted that in this embodiment, the OpenGL thread and the AI ​​computation thread can be executed asynchronously.

[0347] Specifically, after the GPU receives the first target image processed by AI, it sends a start command to the AI ​​computing thread in the NPU to enable the NPU to process the next input first target image. In this way, the electronic device 100 makes full use of both processors and avoids thread blocking.

[0348] The image processing method provided in this embodiment selects a high-precision image compression algorithm based on the prediction of image quality loss, and achieves adaptive image compression. The compressed image is then processed by AI. In this way, the amount of data transmitted by the electronic device is reduced and the load on the electronic device is lowered during the AI ​​processing of the image.

[0349] Figure 14 is a second flowchart illustrating the image processing method provided in an embodiment of this application.

[0350] Figure 15 is a second interactive schematic diagram of the image processing method provided in the embodiments of this application.

[0351] As shown in Figures 14 and 15, in some embodiments, the method includes the following steps S21-S26.

[0352] In step S21, in response to the user's first operation on the first icon, the electronic device 100 loads the first application.

[0353] In one implementation, step S21 includes steps S211-S215.

[0354] Step S211: The first application loads the OpenGL ES thread and initializes the AI ​​model in the OpenGL ES thread.

[0355] In step S212, the first application configures the initialized AI model on the NPU.

[0356] Step S213: After configuring the AI ​​model, the first application determines the rendering method corresponding to the second target image. The rendering method includes forward rendering or deferred rendering.

[0357] Step S214: The first application configures the rendering method in the graphics driver.

[0358] In step S215, the graphics driver issues the first rendering instruction to the graphics processor (GPU) based on the rendering method.

[0359] In this application embodiment, the specific methods of steps S211-S215 can be referred to the foregoing embodiments, and will not be repeated here.

[0360] In step S22, the electronic device 100 acquires the second target image from the first application.

[0361] The second target image can be any image to be displayed in the game interface of the game application.

[0362] As shown in B of Figure 8, the second target image is, for example, the second game image.

[0363] In one implementation, step S22 includes steps S221-S222.

[0364] In step S221, in response to the first rendering instruction, the GPU renders the second target image.

[0365] In step S222, after the GPU finishes rendering the second target image, it stores the second target image in the GPU's buffer.

[0366] In this application embodiment, the specific implementation of steps S221-S222 can be referred to the foregoing embodiments, and will not be repeated here.

[0367] In step S23, the electronic device 100 performs image quality loss prediction processing on the second target image to obtain the second target loss value corresponding to the second target image.

[0368] The second target loss value is used to represent the degree of image quality loss of the second target image, and the second target loss value is less than the first target loss value.

[0369] For example, electronic device 100 can determine the second target loss value as 80.

[0370] In step S24, if the second target loss value is less than or equal to the first loss threshold, the electronic device 100 determines that the second target loss value corresponds to the second image compression algorithm.

[0371] Specifically, if the second target loss value is less than or equal to the preset first loss threshold, it indicates that the image quality loss of the second target image is relatively small. The electronic device only needs to select a lower precision image compression algorithm for the second target image, such as a second image compression algorithm, which is used to compress the image to a second precision. The first precision is greater than the second precision.

[0372] The color space conversion algorithm may further include a first color space conversion algorithm corresponding to a second preset standard. The second preset standard is, for example, the first standard (i.e., ITU-R BT.601 standard) or the second standard (i.e., ITU-R BT.709 standard) in the aforementioned embodiments. Then, the second color space conversion algorithm can be implemented based on the first-third formula, or based on the fourth-sixth formula. The second color space conversion algorithm is an integer-based color space conversion algorithm, which can reduce system overhead during image color conversion.

[0373] Further based on the aforementioned example, when the second target loss value is 80, the electronic device 100 can determine that the second target loss value is less than the first loss threshold, and thus determine that the second target loss value corresponds to the second color space conversion algorithm.

[0374] In this application embodiment, the specific implementation of step S24 can be referred to the foregoing embodiments, and will not be repeated here.

[0375] Step S25: Based on the second image compression algorithm, the electronic device 100 performs image compression processing on the second target image to obtain the second target compressed image.

[0376] In this embodiment, based on the second color space conversion algorithm, the electronic device 100 performs image compression processing on the second target image to obtain a compressed second target image. The specific implementation of step S25 can be found in the foregoing embodiments, and will not be repeated here.

[0377] Step S26: After compressing the second target compressed image based on AI technology, the electronic device 100 restores the second target compressed image to obtain the AI-processed second target image.

[0378] In one implementation, step S26 includes steps S2611-S2614.

[0379] In step S2611, the GPU maps the second target compressed image to shared memory.

[0380] In step S2612, the NPU accesses shared memory to use the second target compressed image as input to the AI ​​model.

[0381] In step S2613, after the NPU performs AI processing on the second target compressed image based on the AI ​​model, it writes the output AI-processed second target compressed image into shared memory.

[0382] In step S2614, the GPU obtains the AI-processed second target compressed image from shared memory and maps the AI-processed second target compressed image from the second color space to the first color space to obtain the AI-processed second target image.

[0383] Furthermore, the GPU can display the second target image after AI processing.

[0384] The image processing method provided in this embodiment selects a low-precision image compression algorithm based on the prediction of image quality loss, and achieves adaptive image compression. The compressed image is then processed by AI. In this way, the amount of data transmitted by the electronic device is reduced and the load on the electronic device is lowered during the AI ​​processing of the image.

[0385] Figure 16 is the third flowchart of the image processing method provided in the embodiments of this application.

[0386] As shown in Figure 16, the image processing method provided in this application embodiment may include the following steps S31-S35.

[0387] Step S31: The electronic device acquires the first image from the first application.

[0388] In step S32, the electronic device performs image quality loss prediction processing on the first image to obtain a first loss value corresponding to the first image. The first loss value is used to represent the image quality loss degree of the first image.

[0389] Step S33: The electronic device determines the image compression algorithm corresponding to the first loss value.

[0390] Step S34: Based on the image compression algorithm, the electronic device performs image compression processing on the first image to obtain the first compressed image.

[0391] Step S35: The electronic device performs AI processing on the first compressed image based on AI technology, and then restores the first compressed image to obtain the AI-processed first image.

[0392] The image processing method provided in this application determines a suitable image compression algorithm based on the prediction of the degree of image quality loss, compresses the image, and performs AI processing on the compressed image. In this way, the amount of data transmitted by the electronic device is reduced and the load on the electronic device is lowered during the AI ​​processing of the image.

[0393] The foregoing primarily describes the solutions provided in the embodiments of this application from the perspective of electronic devices. It is understood that, in order to achieve the aforementioned functions, the electronic device includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, based on the image processing method steps described in conjunction with the embodiments disclosed in this application, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by software-driven hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0394] This application embodiment can divide the above-described electronic device into functional modules or functional units according to the above method examples. For example, each function can be divided into its own functional modules or functional units, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module or functional unit. The module or unit division in this application embodiment is illustrative and represents only one logical functional division; other division methods may be used in actual implementation.

[0395] Figure 17 is a schematic diagram of the structure of the image processing apparatus provided in the embodiment of this application.

[0396] As shown in Figure 17, the image processing device 200 provided in this application embodiment can be applied to the electronic device in the above embodiment. The image processing device 200 may include a display screen 201, a memory 202, a processor 203, and a communication module 204. These devices can be connected via one or more communication buses 205. The display screen 201 may include a display panel 2011 and a touch sensor 2012. The display panel 2011 displays images, and the touch sensor 2012 transmits detected touch operations to the application processor to determine the touch event type. The display panel 2011 provides visual output related to the touch operation. The processor 203 may include one or more processing units, such as an application processor, a modem processor, a graphics processor, an image signal processor, a controller, a video codec, a digital signal processor, a baseband processor, and / or a neural network processor. Different processing units may be independent devices or integrated into one or more processors. The memory 202 is coupled to the processor 203 and is used to store various software programs and / or computer instructions. The memory 202 may include volatile memory and / or non-volatile memory. When the processor executes computer instructions, the electronic device can perform the various functions or steps performed in the above method embodiments.

[0397] Figure 18 is a schematic diagram of the structure of an image processing apparatus provided in another embodiment of this application.

[0398] As shown in Figure 18, in some other embodiments, this application also provides an image processing device 300, which can be applied to the electronic device in the above embodiments. The image processing device 300 includes an acquisition module 301, a prediction module 302, a determination module 303, a compression module 304, and a processing module 305. The acquisition module 301 is used to acquire a first image from a first application. The prediction module 302 is used to perform image quality loss prediction processing on the first image to obtain a first loss value corresponding to the first image, whereby the first loss value represents the degree of image quality loss of the first image. The determination module 303 is used to determine the image compression algorithm corresponding to the first loss value. The compression module 304 is used to perform image compression processing on the first image based on the image compression algorithm to obtain a first compressed image. The processing module 305 is used to perform AI processing on the first compressed image based on AI technology and then restore the first compressed image to obtain the AI-processed first image.

[0399] The image processing apparatus provided in this application determines a suitable image compression algorithm based on the prediction of the degree of image quality loss, compresses the image, and performs AI processing on the compressed image. In this way, the amount of data transmitted by the electronic device is reduced and the load on the electronic device is reduced during the AI ​​processing of the image.

[0400] This application also provides an electronic device, which includes a processor and a memory coupled together. The memory stores program instructions, and when the program instructions are executed by the processor, the processor performs the various functions or steps as described in the above method embodiments.

[0401] Figure 19 is a schematic diagram of the chip system provided in an embodiment of this application.

[0402] As shown in Figure 19, this application embodiment also provides a chip system 400, such as a SoC, which includes at least one processor 401 and at least one interface circuit 402. The processor 401 and the interface circuit 402 can be interconnected via lines. For example, the interface circuit 402 can be used to receive signals from other devices (e.g., the memory of an electronic device). As another example, the interface circuit 402 can be used to send signals to other devices (e.g., the processor 401 or the touchscreen of an electronic device). Exemplarily, the interface circuit 402 can read instructions stored in the memory and send the instructions to the processor 401. When the instructions are executed by the processor 401, the electronic device can perform the steps in the above embodiments. Of course, the chip system may also include other discrete devices, which are not specifically limited in this application embodiment.

[0403] This application also provides a computer-readable storage medium including computer instructions that, when executed on the electronic device, cause the electronic device to perform various functions or steps performed by the electronic device in the above method embodiments.

[0404] This application also provides a computer program product that, when run on an electronic device, causes the electronic device to perform various functions or steps performed by the electronic device in the above method embodiments.

[0405] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0406] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0407] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0408] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0409] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0410] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized in that, Applied to electronic devices, including: Get the first image from the first application; The first image is subjected to image quality loss prediction processing to obtain a first loss value corresponding to the first image. The first loss value is used to represent the image quality loss degree of the first image. Determine the image compression algorithm corresponding to the first loss value; Based on the image compression algorithm, the first image is compressed to obtain the first compressed image; After performing AI processing on the first compressed image based on artificial intelligence (AI) technology, the first compressed image is restored to obtain the first image after AI processing.

2. The image processing method according to claim 1, characterized in that, Before acquiring the first image from the first application, the process also includes: Load the first application, which includes a game application.

3. The image processing method according to claim 2, characterized in that, The step of performing image quality loss prediction processing on the first image to obtain a first loss value corresponding to the first image includes: At least one first pixel in the target region of the first image is obtained, wherein the first image includes a red-green-blue RGB image created based on a first color space, and when the first image is the RGB image, the pixel format of the first pixel is RGB format; The target area is mapped from the first color space to the second color space, so that the first pixel with the RGB format is converted into a second pixel with the luminance and chrominance YUV format; The target region is mapped from the second color space to the first color space, so that the second pixel having the YUV format is converted into a third pixel having the RGB format; The average Euclidean distance between the first pixel and the third pixel is determined as the first loss value.

4. The image processing method according to claim 3, characterized in that, The image compression algorithm for determining the first loss value includes: If the first loss value is greater than the first loss threshold, it is determined that the first loss value corresponds to the first image compression algorithm, which is used to compress the image to a first precision. If the first loss value is less than or equal to the first loss threshold, the first loss value is determined to correspond to the second image compression algorithm, which is used to compress the image to a second precision, wherein the first precision is greater than the second precision.

5. The image processing method according to claim 4, characterized in that, The image compression algorithm includes a color space conversion algorithm; The first image compression algorithm includes a first color space conversion algorithm corresponding to a first preset standard, wherein the first color space conversion algorithm is used for calculation based on floating-point numbers; The second image compression algorithm includes a second color space conversion algorithm corresponding to a second preset standard, which is used for calculation based on integers.

6. The image processing method according to claim 4, characterized in that, The process of compressing the first image based on the image compression algorithm to obtain a first compressed image includes: When the image compression algorithm is the color space conversion algorithm, the first image is mapped from the first color space to the second color space based on the color space conversion algorithm; In the first image, the luminance channel of the second color space is separated to obtain the first compressed image.

7. The image processing method according to claim 6, characterized in that, The step of separating the luminance channel of the second color space in the first image to obtain the first compressed image includes: In the first image, the luminance channel and chroma channel of the second color space are separated, and the chroma channel is linearly interpolated to obtain the first compressed image.

8. The image processing method according to claim 7, characterized in that, The step of restoring the first compressed image after performing AI processing on the first compressed image based on artificial intelligence (AI) technology to obtain the AI-processed first image includes: After performing AI processing on the first compressed image based on the AI ​​technology, the first compressed image is mapped from the second color space to the first color space to obtain the first image after AI processing.

9. The image processing method according to claim 8, characterized in that, Loading the first application includes: The first application calls the rendering thread to initialize the preset AI model; The first application configures the initialized AI model on the neural network processor (NPU); After configuring the AI ​​model, the first application determines the rendering method corresponding to the first image, which includes forward rendering or deferred rendering. The first application configures the rendering method in the graphics driver; The graphics driver issues the first rendering instruction to the graphics processor (GPU) based on the rendering method.

10. The image processing method according to claim 9, characterized in that, The first application calls the rendering thread to initialize a preset AI model, including: The first application detects the operating environment of the AI ​​model; After the runtime environment detection is completed, the first application loads the preset AI model and performs model compilation processing on the AI ​​model. After completing the model compilation process, the first application configures the AI ​​model in memory so that the AI ​​model can be configured in the NPU.

11. The image processing method according to claim 9, characterized in that, The first application configures the initialized AI model after the neural network processor (NPU), and also includes: The NPU asynchronously starts the AI ​​model.

12. The image processing method according to claim 11, characterized in that, The acquisition of the first image of the first application includes: In response to the first rendering instruction, the GPU renders the first image; After rendering the first image, the GPU stores the first image in the GPU's buffer.

13. The image processing method according to claim 12, characterized in that, The step of performing image quality loss prediction processing on the first image to obtain a first loss value corresponding to the first image includes: The GPU acquires the first pixel in the target region. The GPU maps the target region from the first color space to the second color space, so that the first pixel having the RGB format is converted into the second pixel having the YUV format; The GPU maps the target region from the second color space to the first color space, so that the second pixel having the YUV format is converted into a third pixel having the RGB format; The GPU determines the average Euclidean distance between the first pixel and the third pixel as the first loss value.

14. The image processing method according to claim 13, characterized in that, The image compression algorithm for determining the first loss value includes: If the first loss value is greater than the first loss threshold, the GPU determines that the first loss value corresponds to the first image compression algorithm; If the first loss value is less than or equal to the first loss threshold, the GPU determines that the first loss value corresponds to the second image compression algorithm.

15. The image processing method according to claim 14, characterized in that, The process of compressing the first image based on the image compression algorithm to obtain a first compressed image includes: The GPU maps the first image from the first color space to the second color space based on the color space conversion algorithm. The GPU separates the luminance channel of the second color space from the first image to obtain the first compressed image.

16. The image processing method according to claim 15, characterized in that, The GPU separates the luminance channel of the second color space in the first image to obtain the first compressed image, including: In the first image, the GPU separates the luminance channel and chroma channel of the second color space, and performs linear interpolation on the chroma channel to obtain the first compressed image.

17. The image processing method according to claim 16, characterized in that, The step of restoring the first compressed image after performing AI processing on the first compressed image based on artificial intelligence (AI) technology to obtain the AI-processed first image includes: The GPU maps the first compressed image to shared memory; The NPU accesses the shared memory to use the first compressed image as input to the AI ​​model; After the NPU performs AI processing on the first compressed image based on the AI ​​model, it writes the output AI-processed first compressed image into the shared memory. The GPU obtains the AI-processed first compressed image from the shared memory and maps the AI-processed first compressed image from the second color space to the first color space to obtain the AI-processed first image.

18. The image processing method according to claim 17, characterized in that, The GPU retrieves the AI-processed first compressed image from the shared memory and maps the AI-processed first compressed image from the second color space to the first color space to obtain the AI-processed first image, including: In the case that the first compressed image after AI processing includes a chroma channel based on linear interpolation, after the GPU obtains the first compressed image after AI processing from the shared memory, it reconstructs the chroma channel based on linear interpolation to complete the mapping of the first compressed image after AI processing from the second color space to the first color space, thereby obtaining the first image after AI processing.

19. The image processing method according to any one of claims 10-18, characterized in that, The AI ​​model includes an AI super-resolution model, which is generated based on an offline training process.

20. An electronic device, characterized in that, include: A processor and a memory, the memory storing program instructions that, when executed by the processor, cause the electronic device to perform the image processing method as described in any one of claims 1-19.

21. A computer-readable storage medium, characterized in that, include: Computer instructions that, when executed on an electronic device, cause the electronic device to perform the image processing method as described in any one of claims 1-19.