Image encoding method and apparatus, image decoding method and apparatus, device, storage medium, and program product

Abstract

The present application provides an image encoding method and apparatus, an image decoding method and apparatus, a device, a storage medium, and a program product; The image encoding method comprises: acquiring an image to be encoded; executing at least one image detection mode on the image on the basis of the priorities of multiple image detection modes to obtain an image detection result; respectively setting a quantization parameter for at least one macroblock in the image on the basis of the detection result; encoding each macroblock on the basis of the quantization parameter for each macroblock; and generating image encoding data of the image on the basis of macroblock encoding data obtained after encoding.

Description

Image encoding methods, image decoding methods, devices, equipment, storage media, and software products

[0001] Cross-references to related applications

[0002] This application is based on and claims priority to Chinese Patent Application No. 2024115374006, filed on October 30, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of computer graphics, and more particularly to an image encoding method, an image decoding method, an apparatus, a device, a storage medium, and a program product. Background Technology

[0004] As streaming services continue to proliferate, users are demanding higher and higher video clarity. However, maintaining the clarity of live images at high resolution requires a high encoding bitrate. Insufficient bandwidth can easily lead to buffering during viewing and also puts enormous bandwidth cost pressure on streaming service platforms.

[0005] Related technologies address these issues by replacing the encoding standard with a more advanced one, such as upgrading from H.264 to H.265 or H.266. However, more advanced encoding standards have lower hardware compatibility, increasing the difficulty of promotion and leading to greater computational power consumption, making it difficult to achieve the same image encoding effect at lower encoding bitrates. Summary of the Invention

[0006] This application provides an image encoding method, an image decoding method, an apparatus, a device, a storage medium, and a program product, which can achieve more precise encoding control.

[0007] The technical solution of this application embodiment is implemented as follows:

[0008] This application provides an image encoding method applied to an electronic device, the method comprising:

[0009] Obtain the image to be encoded;

[0010] Based on the priority of multiple image detection methods, at least one of the image detection methods is performed on the image to obtain the detection result of the image;

[0011] Based on the detection results, quantization parameters are set for at least one macroblock in the image;

[0012] Each macroblock is encoded based on the quantization parameters of each macroblock;

[0013] Image encoded data of the image is generated based on the macroblock encoded data obtained after encoding.

[0014] This application provides an image decoding method applied to an electronic device, the method comprising:

[0015] Receive image encoded data, wherein the image encoded data is obtained through the image encoding method provided in the embodiments of this application;

[0016] Based on the quantization parameters of each macroblock, the encoded data of each macroblock is decoded to obtain the macroblock decoded data of each macroblock;

[0017] The macroblock decoding data of each macroblock is combined into a decoded image.

[0018] This application provides an image encoding method applied to an electronic device, the method comprising:

[0019] Obtain the image to be encoded;

[0020] Based on the proportion of pixels in multiple color intervals in the image, the target color interval corresponding to the target area of ​​the target object is determined.

[0021] The proportion of pixels belonging to the target color range in each macroblock of the image is used as the probability that each macroblock includes the target region of the target object;

[0022] Based on the probability that each macroblock includes the target region, quantization parameters are set for at least one macroblock;

[0023] Each macroblock is encoded based on the quantization parameters of each macroblock;

[0024] Image encoded data of the image is generated based on the macroblock encoded data obtained after encoding.

[0025] This application provides an image decoding method applied to an electronic device, the method comprising:

[0026] Receive image encoded data, wherein the image encoded data is obtained through the image encoding method provided in the embodiments of this application;

[0027] Based on the quantization parameters of each macroblock, the encoded data of each macroblock is decoded to obtain the macroblock decoded data of each macroblock;

[0028] The macroblock decoding data of each macroblock is combined into a decoded image.

[0029] This application provides an image encoding device, including:

[0030] The data acquisition module is configured to acquire the image to be encoded.

[0031] The encoding module is configured to perform at least one of the image detection methods on the image according to the priority of multiple image detection methods, so as to obtain the detection result of the image;

[0032] The encoding module is further configured to set quantization parameters for at least one macroblock in the image based on the detection results;

[0033] The encoding module is further configured to encode each macroblock based on the quantization parameters of each macroblock;

[0034] The encoding module is further configured to generate image encoding data of the image based on the macroblock encoding data obtained after encoding.

[0035] This application provides an image decoding device, including:

[0036] The data acquisition module is configured to receive image encoded data, wherein the image encoded data is obtained by the image encoding method provided in the embodiments of this application;

[0037] The decoding module is configured to decode the encoded data of each macroblock based on the quantization parameters of each macroblock to obtain the macroblock decoded data of each macroblock.

[0038] The decoding module is further configured to combine the macroblock decoding data of each macroblock into a decoded image.

[0039] This application provides an image encoding device, including:

[0040] The data acquisition module is configured to acquire the image to be encoded.

[0041] The encoding module is configured to determine the target color range corresponding to the target region of the target object based on the proportion of pixels in multiple color ranges in the image.

[0042] The encoding module is further configured to use the proportion of pixels belonging to the target color range in each macroblock of the image as the probability that each macroblock includes the target region of the target object;

[0043] The encoding module is further configured to set quantization parameters for at least one macroblock based on the probability that each macroblock includes the target region.

[0044] The encoding module is further configured to encode each macroblock based on the quantization parameters of each macroblock;

[0045] The encoding module is further configured to generate image encoding data of the image based on the macroblock encoding data obtained after encoding.

[0046] This application provides an image decoding device, including:

[0047] The data acquisition module is configured to receive image encoded data, wherein the image encoded data is obtained through the image encoding method provided in the embodiments of this application;

[0048] The decoding module is configured to decode the encoded data of each macroblock based on the quantization parameters of each macroblock to obtain the macroblock decoded data of each macroblock.

[0049] The decoding module is further configured to combine the macroblock decoding data of each macroblock into a decoded image.

[0050] This application provides an electronic device, the electronic device comprising:

[0051] Memory is used to store executable instructions or computer programs.

[0052] When a processor executes computer-executable instructions or computer programs stored in the memory, it implements the image encoding method or the image decoding method provided in the embodiments of this application.

[0053] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions, which, when executed by a processor, implement the image encoding method or the image decoding method provided in this application.

[0054] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the image encoding method or the image decoding method provided in this application.

[0055] This application provides a method for storing a video stream, which is used to generate a video stream by performing an image encoding method provided in this application; and to store the video stream.

[0056] This application provides a method for transmitting a video stream, which is used to generate a video stream by performing an image encoding method provided in this application; and to transmit the video stream.

[0057] This application provides a computer-readable storage medium storing a computer program / instructions and a video stream thereon. The computer program / instructions are executed by a processor to implement the steps of the image encoding method provided in this application to generate the video stream.

[0058] The embodiments of this application have the following beneficial effects:

[0059] By prioritizing multiple image detection methods, images are detected, and more suitable quantization parameters are set for macroblocks in the image based on the detection results of at least one image detection method. Compared with related technologies that use a single quantization parameter calculation scheme, the image detection methods used are more diverse, and the priority-based detection ensures that the detection results of the more suitable detection method are prioritized to set more suitable quantization parameters for each macroblock. This results in finer coding control and higher quality images with limited bitrate. Attached Figure Description

[0060] Figure 1 is a schematic diagram of the image encoding and decoding system architecture provided in an embodiment of this application;

[0061] Figure 2A is a first structural schematic diagram of the image encoding device provided in an embodiment of this application;

[0062] Figure 2B is a schematic diagram of the second structure of the image encoding device provided in an embodiment of this application;

[0063] Figure 2C is a first structural schematic diagram of the image decoding device provided in an embodiment of this application;

[0064] Figure 2D is a second structural schematic diagram of the image decoding device provided in an embodiment of this application;

[0065] Figure 3A is a first flowchart of the image encoding method provided in an embodiment of this application;

[0066] Figure 3B is a second flowchart of the image encoding method provided in an embodiment of this application;

[0067] Figure 3C is a schematic diagram of the third process of the image encoding method provided in the embodiments of this application;

[0068] Figure 3D is a schematic diagram of the fourth process of the image encoding method provided in the embodiments of this application;

[0069] Figure 3E is a fifth flowchart illustrating the image encoding method provided in this application embodiment;

[0070] Figure 3F is a schematic diagram of the sixth process of the image encoding method provided in the embodiment of this application;

[0071] Figure 3G is a schematic diagram of the seventh process of the image encoding method provided in the embodiments of this application;

[0072] Figure 3H is a schematic diagram of the eighth process of the image encoding method provided in the embodiment of this application;

[0073] Figure 3I is a ninth flowchart illustrating the image encoding method provided in an embodiment of this application;

[0074] Figure 3J is a schematic diagram of the tenth process of the image encoding method provided in the embodiment of this application;

[0075] Figure 3K is a schematic diagram of the eleventh step of the image encoding method provided in the embodiment of this application;

[0076] Figure 3L is a schematic diagram of the twelfth step of the image encoding method provided in the embodiment of this application;

[0077] Figure 3M is a flowchart illustrating the detection method provided in an embodiment of this application;

[0078] Figure 3N is a schematic diagram of the detection method provided in the embodiment of this application;

[0079] Figure 30 is a flowchart illustrating the process of determining the target color range according to an embodiment of this application;

[0080] Figure 4A is a schematic diagram of the thirteenth step of the image encoding method provided in the embodiment of this application;

[0081] Figure 4B is a schematic diagram of the fourteenth step of the image encoding method provided in the embodiment of this application;

[0082] Figure 4C is a schematic diagram of the fifteenth step of the image encoding method provided in the embodiment of this application;

[0083] Figure 4D is a schematic diagram of the sixteenth step of the image encoding method provided in the embodiment of this application;

[0084] Figure 5A is a first flowchart of the image decoding method provided in an embodiment of this application;

[0085] Figure 5B is a second flowchart of the image decoding method provided in an embodiment of this application;

[0086] Figure 5C is a schematic diagram of the third process of the image decoding method provided in the embodiments of this application;

[0087] Figure 6 is a schematic diagram of the fourth process of the image decoding method provided in the embodiments of this application;

[0088] Figure 7A is a schematic diagram of macroblocks in an image provided in an embodiment of this application;

[0089] Figure 7B is a schematic diagram of the first region provided in an embodiment of this application;

[0090] Figure 7C is a schematic diagram of the second region provided in an embodiment of this application;

[0091] Figure 7D is a schematic diagram of the probability level and flatness level of the image provided in the embodiment of this application;

[0092] Figure 8 is a first flowchart illustrating the application of the image encoding method and image decoding method provided in the embodiments of this application in a live streaming scenario;

[0093] Figure 9 is a second flowchart illustrating the application of the image encoding and image decoding methods provided in this application embodiment in a live streaming scenario.

[0094] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0095] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0096] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0097] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0098] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0099] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0100] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0101] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0102] 1) The target region, also known as the region of interest (ROI), refers to the area in an image that requires focused attention. The target region includes a first region and a second region. The first region is the core, overall area of ​​the target object (including living things or objects) in the image. It is the most identifiable part of the target object, highlighting its main form or identity characteristics, and is a core component of the ROI. For example, when the target object is a living thing, the first region is the face (e.g., a human face); when the target object is an object, the first region is the main body of the object (e.g., the body of a car, excluding non-core peripheral parts such as wheels). The second region is a local feature area of ​​the target object (including living things or objects) in the image. It is a local component of the target object outside the first region, possessing specific functions or detailed attributes, and is a supplementary component of the ROI. For example, when the target object is a living thing, the second region is the skin area of ​​the living thing; when the target object is an object, the second region is a local area of ​​the object (e.g., the headlights, rims, etc., of a car, which are peripheral parts with detailed features).

[0103] Taking a light-sensitive area where the target area will show color difference under different lighting conditions as an example, the following are some common light-sensitive areas:

[0104] Edges and contours: Changes in lighting can significantly affect the visibility of an object's edges and contours. For example, strong light may cause edges to be blurred, while weak light may cause contours to be lost in the background.

[0105] Reflection and Highlights: For objects with reflective or highlighting parts, such as metal, glass, or water surfaces, the angle and intensity of the light directly affect the visual appearance of these areas.

[0106] Shadow areas: In shadow areas, lighting conditions determine the depth and shape of the shadows, thus affecting the three-dimensionality and depth of the image.

[0107] Color and Texture: Lighting affects the perception of color in an image, as well as the visibility and sharpness of texture. For areas with significant color variations, such as flower or landscape photographs, lighting correction can make the colors more realistic.

[0108] Backlit areas: Under backlit conditions, certain areas of an object may appear too dark due to insufficient light, or glare may occur due to strong backlighting. The treatment of these areas requires consideration of how to balance the backlighting and the main subject lighting.

[0109] Human Eyes and Facial Features: In portrait photography, lighting is crucial for capturing the details of the eyes, facial expressions, and skin texture. Correct lighting can highlight these features, while improper lighting can cause facial shadows or loss of detail.

[0110] 2) The quantization parameter (QP) is a key indicator defining the quantization step size. QP determines the interval at which continuous signal values ​​are mapped to discrete levels, i.e., the quantization precision. A smaller QP value results in a smaller quantization step size, finer quantization, and higher image quality, but also a higher bitrate; conversely, a larger QP value results in a larger quantization step size, coarser quantization, lower image quality, but a lower bitrate and higher compression ratio. Therefore, QP is used in image coding (or video coding) to strike a trade-off between compression ratio and image quality.

[0111] In different video coding standards, such as H.264 / AVC and HEVC, setting the quantization parameter (QP) is often a crucial coding decision, requiring a comprehensive consideration of coding efficiency and video quality to achieve optimal coding results. Furthermore, the QP not only affects the compression ratio and quality of the video but is also closely related to the bitrate control strategy. The goal of the bitrate control strategy is to reduce the video bitrate as much as possible while maintaining video quality, thereby increasing the compression ratio. The QP is a key parameter in this strategy, controlling the bitrate by adjusting the quantization coefficients. In summary, the QP plays a vital role in video coding; it is a critical parameter that requires a trade-off between coding efficiency and video quality. By properly setting the QP, efficient video coding can be achieved while simultaneously reducing the bitrate to maintain video quality.

[0112] 3) Bitrate refers to the number of bits used per unit of time (usually measured in seconds) in digital audio and video encoding. It represents the speed of data transmission or storage. A higher bitrate means more information is contained per unit of time, usually resulting in higher audio and video quality, but it also requires more storage space and transmission bandwidth.

[0113] 4) Resolution refers to the size of an image in digital audio and video encoding, usually expressed in pixels. Resolution consists of the number of horizontal pixels and the number of vertical pixels. For example, 1920×1080 (1080p) means that the image width is 1080 pixels and the height is 1920 pixels. The higher the resolution, the richer the image details, but it also requires more computing resources to process.

[0114] 5) Super-resolution is an image post-processing technique that generates images with higher resolution and richer details by processing low-resolution images. It can be implemented using deep learning-based algorithms (such as image super-resolution algorithms based on convolutional neural networks).

[0115] 6) Color enhancement is an image processing technique that adjusts the colors of an image to make them more vibrant and the contrast higher, thereby improving the visual effect of the image. Color enhancement technology can be applied to image restoration, video processing, and other fields.

[0116] 7) Adaptive Quantization is a video coding technique that automatically adjusts quantization parameters based on the content of a video sequence to achieve higher coding efficiency. Adaptive quantization can perform different degrees of compression on different regions of the video (such as dynamic and static regions), thereby reducing the coding bitrate while maintaining video quality.

[0117] 8) Macroblock is a basic concept in video coding or image coding. It refers to dividing the image to be encoded into several blocks of the same size (e.g., 16×16 or 8×8), so that image coding processing is performed using macroblocks as the basic unit.

[0118] 9) Image detection methods refer to the methods used to detect reference data from an image for calculating quantization parameters of macroblocks in the image. Image detection methods include a first detection method and a second detection method. Taking a person as the target object in the image as an example, the first detection method can be face recognition, and the second detection method can be skin color detection.

[0119] 10) A color space, also known as a color chart, is a way of describing color, defining a set of colors and a method of representing them. Different color spaces are suitable for different scenarios. Here are some common color spaces:

[0120] The RGB color space is an additive color model based on red, green, and blue light, and is widely used in monitors, televisions, and photography. The RGB color space can be directly associated with display devices, making it suitable for image editing and display, but not for color printing.

[0121] The YUV / YCbCr color space separates luminance information (Y) from chrominance information (U, V or Cb, Cr) and is commonly used for video compression and transmission. In the YUV color space, the Y component is responsible for luminance, while the U and V components are responsible for color information. It is very common in analog and digital television broadcasting.

[0122] The CMYK color space is the color space used in the printing industry. It is based on a subtractive color model of four inks: cyan, magenta, yellow, and black, and allows for the reproduction of a wide range of colors on paper.

[0123] The HSV / HSB color space is a color space based on human perception of color, where H represents hue, S represents saturation, and V or B represents value or brightness, making color selection and manipulation more intuitive.

[0124] The sRGB color space is a standardized RGB color space designed to ensure color consistency across different devices and lighting conditions, and is widely used in web pages and consumer electronics.

[0125] As live video streaming has become a streaming service that internet users encounter every day, users have increasingly higher requirements for video clarity. However, to maintain the clarity of live images at high resolution, a very high encoding bitrate is required. If the bandwidth is insufficient, it can easily lead to buffering during viewing, and at the same time, it puts a huge bandwidth cost pressure on live streaming platforms.

[0126] Related technologies address these issues by replacing the encoding standard with a more advanced one, such as upgrading from H.264 to H.265 or H.266. However, more advanced encoding standards have lower hardware compatibility, increasing the difficulty of promotion and leading to greater computational power consumption, making it difficult to achieve the same image encoding effect at lower encoding bitrates.

[0127] This application provides an image encoding method, an image decoding method, an apparatus, a device, a computer-readable storage medium, and a computer program product, which can achieve more refined encoding control. The following describes exemplary applications of the electronic devices provided in this application. These electronic devices can be implemented as various types of terminal devices such as laptops, tablets, desktop computers, set-top boxes, smartphones, smart speakers, smartwatches, smart TVs, and vehicle terminals, or as servers. Exemplary applications when the device is implemented as a terminal will be described below.

[0128] Referring to Figure 1, which is a schematic diagram of the image encoding and decoding system architecture provided in an embodiment of this application, Figure 1 involves server 100, terminal device 200-1, terminal device 200-2, and network 300. Terminal device 200-1 and terminal device 200-2 are connected to server 100 through network 300, wherein network 300 can be a wide area network, a local area network, or a combination of both.

[0129] In some embodiments, the embodiments of this application can be implemented collaboratively by a server and a terminal device. For example, terminal device 200-1 encodes a local image to be encoded using the image encoding method provided in this application to obtain image encoding data, and then sends the image encoding data to server 100. Server 100 receives the image encoding data and then sends it to terminal device 200-2. Terminal device 200-2 receives the image encoding data and decodes it using the image decoding method provided in this application to obtain a decoded image. Here, in an instant messaging scenario, server 100 can be a backend server for an instant messaging client; in a live streaming scenario, server 100 can be a backend server for a live streaming platform.

[0130] In other embodiments, the embodiments of this application can be implemented collaboratively by a server and a terminal device. For example, terminal device 200-1 sends a locally acquired image to be encoded to server 100 using the image encoding method provided in the embodiments of this application. After receiving the image to be encoded (e.g., a photo from multiple photos in burst mode saved in an electronic photo album, or a frame from a video), server 100 encodes the image to be encoded using the image encoding method provided in the embodiments of this application to obtain image encoding data, and sends the image encoding data to terminal device 200-2. After receiving the image encoding data, terminal device 200-2 decodes the image encoding data using the image decoding method provided in the embodiments of this application to obtain a decoded image. Here, in an instant messaging scenario, server 100 can be a backend server for an instant messaging client; in a live streaming scenario, server 100 can be a backend server for a live streaming platform.

[0131] In other embodiments, the embodiments of this application can be implemented by a terminal device alone. For example, the terminal device 200-1 runs an electronic photo album application, takes a photo (corresponding to the image to be encoded), and then encodes the photo using the image encoding method provided in the embodiments of this application to obtain the image encoding data of the photo, thereby reducing the space occupied by the image in memory and on the disk. When the user opens the electronic photo album, the image encoding data of the photo is decoded using the image decoding method provided in the embodiments of this application to obtain the photo, which is then displayed on the terminal device 200-1.

[0132] In other embodiments, the embodiments of this application can be implemented by a terminal device alone. For example, when the terminal device 200-1 is recording video, it encodes each video frame (corresponding to the image to be encoded) in real time based on the image encoding method provided in the embodiments of this application, thereby obtaining image encoding data for each video frame, thus reducing the space occupied by the video in memory and disk. When the user clicks to play the video, the image encoding data of each video frame is decoded by the image decoding method provided in the embodiments of this application, resulting in a decoded video, which is then played on the terminal device 200-1.

[0133] Taking an image encoding device for image encoding as an example, referring to Figure 2A, Figure 2A is a first structural schematic diagram of an image encoding device provided in an embodiment of this application. The image encoding device can be implemented as the aforementioned terminal device or server. The image encoding device 400 shown in Figure 2A includes: at least one processor 410, a memory 430, and at least one network interface 420. The various components in the image encoding device 400 are coupled together through a bus system 440. It is understood that the bus system 440 is used to realize the connection and communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus system 440 in Figure 2A.

[0134] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0135] The memory 430 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 430 may optionally include one or more storage devices physically located away from the processor 410.

[0136] The memory 430 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 430 described in this application embodiment is intended to include any suitable type of memory.

[0137] In some embodiments, memory 430 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0138] Operating system 431 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0139] The network communication module 432 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 120 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0140] In some embodiments, the apparatus provided in this application can be implemented in software. FIG2A shows a first image encoding apparatus 433 stored in memory 430, which can be software in the form of programs and plug-ins, including the following software modules: a first data acquisition module 4331 and a first encoding module 4332. These modules are logically related, and therefore can be arbitrarily combined or further split according to the functions they implement. The functions of each module will be described below.

[0141] Taking an image encoding device for image encoding as an example, refer to Figure 2B. Figure 2B is a second structural schematic diagram of the image encoding device provided in this application embodiment. The image encoding device can be implemented as the aforementioned terminal device or server. The image encoding device 500 shown in Figure 2B includes: at least one processor 510, a memory 530, and at least one network interface 520. The various components in the image encoding device 500 are coupled together through a bus system 540. It is understood that the bus system 540 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus system 540 in Figure 2B. For a detailed description of the processor 510 and the memory 530, please refer to the above text, and it will not be repeated here.

[0142] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2B shows a second image encoding apparatus 533 stored in memory 530, which can be software in the form of programs and plug-ins, including the following software modules: a second data acquisition module 5331 and a second encoding module 5332. These modules are logically related and can therefore be arbitrarily combined or further split according to the functions they implement. The functions of each module will be described below.

[0143] Taking an image decoding device for image decoding as an example, referring to Figure 2C, Figure 2C is a first structural schematic diagram of an image decoding device provided in an embodiment of this application. The image decoding device can be implemented as the aforementioned terminal device or server. The image decoding device 600 shown in Figure 2C includes: at least one processor 610, a memory 630, and at least one network interface 620. The various components in the image decoding device 600 are coupled together through a bus system 640. It is understood that the bus system 640 is used to realize the connection and communication between these components. In addition to a data bus, the bus system 640 also includes a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus system 640 in Figure 2C. For a detailed description of the processor 610 and the memory 630, please refer to the above text, and it will not be repeated here.

[0144] In some embodiments, the apparatus provided in this application can be implemented in software. FIG2C shows a first image decoding apparatus 633 stored in memory 630, which can be software in the form of programs and plug-ins, including the following software modules: a first data acquisition module 6331 and a first decoding module 6332. These modules are logically related, and therefore can be arbitrarily combined or further split according to the functions they implement. The functions of each module will be described below.

[0145] Taking an image decoding device for image decoding as an example, referring to Figure 2D, Figure 2D is a second structural schematic diagram of the image decoding device provided in an embodiment of this application. The image decoding device can be implemented as the aforementioned terminal device or server. The image decoding device 700 shown in Figure 2D includes: at least one processor 710, a memory 730, and at least one network interface 720. The various components in the image decoding device 700 are coupled together through a bus system 740. It is understood that the bus system 740 is used to realize the connection and communication between these components. In addition to a data bus, the bus system 740 also includes a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus system 740 in Figure 2D. For a detailed description of the processor 710 and the memory 730, please refer to the above text, and it will not be repeated here.

[0146] In some embodiments, the apparatus provided in this application can be implemented in software. FIG2D shows a second image decoding apparatus 733 stored in memory 730, which can be software in the form of programs and plug-ins, including the following software modules: a second data acquisition module 7331 and a second decoding module 7332. These modules are logically linked and can therefore be arbitrarily combined or further split according to the functions they implement. The functions of each module will be described below.

[0147] In some embodiments, the terminal device or server can implement the image encoding or image decoding method provided in this application by running various computer-executable instructions or computer programs. Taking a terminal device as an example, the computer-executable instructions can be microprogram-level commands, machine instructions, or software instructions. The computer program can be a native program or software module in an operating system; it can be a native application (APP), that is, a program that needs to be installed in the operating system to run, such as a live streaming APP or an instant messaging APP; or it can be a small program that can be embedded in any APP, that is, a program that only needs to be downloaded to a browser environment to run. In summary, the above-mentioned computer-executable instructions can be any form of instruction, and the above-mentioned computer program can be any form of application, module, or plugin.

[0148] In other embodiments, the apparatus provided in this application can be implemented in hardware. As an example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the image encoding method or image decoding method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0149] Taking an image encoding device (terminal device or server) for image encoding as an example, the image encoding method provided in this application embodiment is described below. Referring to Figure 3A, which is a first flowchart of the image encoding method provided in this application embodiment, the steps shown in Figure 3A will be used for explanation.

[0150] In step 101, the image to be encoded is obtained.

[0151] In some embodiments, the image to be encoded may be an already encoded image, such as an image decoded from an already encoded video, such as decoding an MPEG format video file, and using the decoded image as the image to be encoded.

[0152] In some embodiments, the image to be encoded can be an uncompressed raw image. Referring to Figure 3B, step 101 shown in Figure 3A can be implemented by steps 1011 to 1012, which will be described in detail below.

[0153] In step 1011, the uncompressed raw image is obtained.

[0154] In some embodiments, the uncompressed raw image can be a frame from raw video, which refers to video data that has not undergone any compression algorithm processing, also known as uncompressed video. Uncompressed raw video images can be captured directly using an image sensor, or a frame from a local raw video can be read using video processing software or programming libraries (such as FFmpeg, OpenCV, etc.) as the raw image. Uncompressed images can also be uncompressed photos in an electronic photo album, such as one photo from multiple photos taken in burst mode as the raw image. The embodiments of this application do not limit the specific method of obtaining uncompressed raw images.

[0155] In step 1012, the original image is used as the image to be encoded, or the original image is downsampled and the resulting downsampled image is used as the image to be encoded.

[0156] In some embodiments, the original image can be used directly as the image to be encoded, or the original image after downsampling processing (such as the Nearest Interpolation, Bilinear Interpolation, Bicubic Interpolation, etc.) can be used as the image to be encoded. For example, the resolution of the original image is 1920×1080 (1080p), and the resolution of the downsampled image obtained after downsampling processing is 1280×720. The embodiments of this application do not limit the specific implementation of downsampling processing.

[0157] This application embodiment uses downsampling processing to reduce the resolution of the original image while only slightly sacrificing the image clarity. This achieves the beneficial effects of reducing the computational cost of image encoding and the storage space of image encoding data under the same image encoding standard.

[0158] Referring again to Figure 3A, in step 102, at least one image detection method is performed on the image according to the priority of multiple image detection methods to obtain the image detection result.

[0159] In some embodiments, the priority is preset. Based on the descending priority of multiple image detection methods, at least one image detection method is performed on the image to obtain the detection result. For example, according to the preset priority, the first detection method with the first priority is executed first, and then the second detection method with the second priority is executed. When the high-priority (first priority) detection method obtains the corresponding detection result, the low-priority (second priority) detection method is no longer executed. When the high-priority detection method does not obtain the corresponding detection result (or when there are insufficient computing resources to execute the high-priority detection method), the low-priority detection method continues to be executed until the detection result of the corresponding priority is obtained.

[0160] By pre-setting the priorities of multiple image detection methods and executing them in descending order of priority (stopping when a high-priority detection obtains a result, and executing low-priority detection only when no result is obtained or resources are insufficient), the effectiveness of image detection can be guaranteed while giving priority to the advantages of high-priority detection methods (such as accuracy and efficiency), reducing unnecessary execution of low-priority detection, thereby optimizing the detection process, saving computing resources, and improving the overall efficiency and practicality of image detection.

[0161] Referring to Figure 3M, which is a flowchart of the detection method provided in this application embodiment, after inputting an image, it is first determined whether the computing resources are sufficient. For example, it is detected whether the Central Processing Unit (CPU) utilization rate is high for a long time (such as higher than the preset utilization rate). For example, if the CPU utilization rate is close to or reaches 100%, it indicates that the CPU resources are insufficient. When the computing resources are sufficient, the execution is switched to the first detection method with a priority of 1 (high priority). When the computing resources are insufficient, the execution is switched to the second detection method with a priority of 2, and the second detection result is output. When the first detection method is executed, if the detection result includes the first region (corresponding to the execution of the high priority detection method to obtain the corresponding detection result), the first detection result is output. When the first detection method is executed, if the detection result does not include the first region (corresponding to the execution of the high priority detection method not obtaining the corresponding detection result), the execution is switched to the second detection method with a priority of 2, and the second detection result is output.

[0162] In some embodiments, the first region may refer to the visual focal area in the image. For example, in a live streaming scenario, the first region may be the face area of ​​the streamer.

[0163] In some embodiments, the multiple image detection methods include at least two detection methods. For example, the first detection method has a higher priority than the second detection method. When the first detection method (e.g., object detection) is performed on the image, after generating a first detection result representing the image including a first region, the second detection method is no longer performed on the image.

[0164] Taking a person as the target object in the image as an example, see Figure 3N. Figure 3N is a schematic diagram of the principle of the detection method provided in the embodiment of this application. The target object can be a live streamer. The first detection method can be face recognition. Correspondingly, the first region is the face region. The second detection method can be skin color detection, used to detect the second region of the streamer (i.e., the region different from the first region). In some embodiments, the second region refers to other regions of interest in the image besides the first region. Taking a live streaming scenario as an example, the second region can be the streamer's skin region, such as exposed skin. See Figure 3N. For the input image-1, the first detection method corresponding to priority 1 is executed according to the preset priority. The detection result indicates that image-1 includes the first region, then... The first detection result represents the first region in image-1, and the second detection method is not performed on image-1. For the input image-2, the first detection method corresponding to priority 1 is performed according to the preset priority. If the detection result represents that image-2 does not include the first region, the second detection method corresponding to priority 2 is performed according to the preset priority. The probability of each macroblock including the second region is combined into the second detection result. Here, the second detection method can also be a detection method combining skin color detection and flatness detection, used to detect the anchor's skin area and the flatness of each macroblock in the image. The probability of each macroblock including the second region and the flatness of the macroblock are combined into the second detection result (see the explanation of steps 10226 to 10228 below).

[0165] Taking the target object in the image as an example, the target object can be a product in an advertising video. The first detection method can be target recognition, such as YOLO and other target recognition methods. The second detection method can be color value detection, which is used to detect local areas of the product.

[0166] Referring to Figure 3C, step 102 shown in Figure 3A can be implemented through the following steps 1021 to 1022, which will be explained in detail below.

[0167] In step 1021, a first detection method is performed on the image, and if a first region including the target object is detected in the image, a first detection result representing the image including the first region is generated.

[0168] In some embodiments, referring to FIG3D, step 1021 shown in FIG3C can be implemented by steps 10211 to 10212, which are described in detail below.

[0169] In step 10211, a pre-trained detection model is invoked to perform keypoint detection on the image, and keypoint detection results are obtained, wherein the keypoint detection results include bounding boxes and keypoints.

[0170] In some embodiments, keypoint detection is performed on an image using a pre-trained detection model to obtain keypoint detection results, wherein the keypoint detection results include bounding boxes and keypoints.

[0171] As an example, the detection model can be a multi-task cascaded convolutional network (MTCNN), or YOLOv8-Pose based on the You Only Look Once (YOLO) model. MTCNN consists of three parts: a proposal network (P-Net), a refinement network (R-Net), and an output network (O-Net). The following explanation uses MTCNN for keypoint detection as an example.

[0172] For example, keypoint detection using MTCNN includes the following stages: In the first stage, P-Net generates candidate boxes (Proposals). P-Net outputs bounding boxes and confidence scores within the bounding boxes. Candidate boxes with higher confidence scores are selected as input for subsequent processing. In the second stage, R-Net refines the candidate boxes generated by P-Net. R-Net receives the output from P-Net and further adjusts the position and size of the bounding boxes to output more accurate bounding boxes and confidence scores. In the third stage, O-Net performs final processing on the output of R-Net. Specifically, O-Net is used to further refine the bounding boxes and detect keypoint locations, thus outputting the final bounding boxes and keypoint locations.

[0173] Taking a person as an example, the key points can be the center of the two eyes, the two corners of the mouth, and the tip of the nose in the face region.

[0174] Taking an image where the target object is a physical object as an example, the location of key points can be the coordinates of the object that have specific meaning. For example, when the object is a car, the location of key points can be the center of the car's wheel hub, the center of the headlights, etc.

[0175] In step 10212, in response to the number of key points being a preset number of key points, the region corresponding to the bounding box is taken as the first region of the target object, and a first detection result is generated in which the image includes the first region. The preset number of key points is used to determine the minimum number of images that include the first region.

[0176] In some embodiments, when the number of key points is a preset number of key points, the region corresponding to the bounding box is taken as the first region of the target object, and a first detection result of the image including the first region is generated. The preset number of key points is used to determine the minimum number of images including the first region. This application embodiment does not limit the number of key points. Taking the target object as a face as an example, see Figure 7B. Figure 7B is a schematic diagram of the first region provided in this application embodiment. The region corresponding to the bounding box is the first region.

[0177] Continuing from the example above, the target object in the image is a person. The first region is the face region. When the number of key points is a preset number of key points (e.g., 5), the first detection result indicates that the image includes the face region.

[0178] Continuing from the previous example, the target object in the image is an example of an object. The first region is the main body region of the object (e.g., the body of a car). When the number of key points is a preset number of key points (e.g., 3), the first detection result indicates that the image includes the main body region (i.e., the first region).

[0179] Referring again to Figure 3C, in step 1022, if the first region is not detected in the image or there is insufficient computing resources to perform the first detection method, the second detection method is performed on each macroblock in the image to obtain the second detection result.

[0180] In some embodiments, a macroblock is a square combination of adjacent pixels in an image. The size of a macroblock can be 16×16 or 8×8, with the unit being pixels. This application does not limit the specific size of the macroblock.

[0181] For example, referring to Figure 7A, which is a schematic diagram of macroblocks in an image provided in an embodiment of this application, the image is divided into multiple macroblocks according to a preset macroblock size. When the first region of the target object shown in Figure 7B is not detected (e.g., because the first region of the target object is not displayed in the image, or because the first region is not detected due to occlusion, rotation, scale change, motion blur, etc. of the target object in the image) or there is insufficient computing resources to execute the first detection method, the second detection method is executed for each macroblock in the image to obtain the second detection result.

[0182] For example, the lack of sufficient computing resources can be detected in the following ways:

[0183] 1) System resource detection, such as detecting whether the CPU utilization rate is high for a long time (such as higher than the preset utilization rate). For example, if it is close to or reaches 100%, it indicates that the CPU resources are insufficient.

[0184] 2) Detect encoding performance. For example, if the encoding speed is lower than the preset encoding speed, it means that the performance of the encoding algorithm is affected by insufficient computing resources.

[0185] 3) Detect system response time. For example, if the system's task processing time increases significantly (e.g., it exceeds the preset task processing time), it indicates that the task queue is backlogged due to insufficient computing resources.

[0186] 4) Detect temperature and power consumption. For example, detect the temperature of the hardware. If the temperature of the CPU or graphics processing unit (GPU) continues to rise (e.g., above the preset temperature threshold), it indicates that computing resources are insufficient or overused.

[0187] 5) Check error logs, such as the system log, to see if there are any error messages or warnings about insufficient resources, or use detection tools (such as Nagios, Zabbix, etc.) to detect system resources and issue an alarm when computing resources are insufficient.

[0188] In some embodiments, referring to FIG3E, step 1022 shown in FIG3C can be implemented by steps 10221 to 10223, which are described in detail below.

[0189] In step 10221, the target color interval corresponding to the second region of the target object is determined based on the proportion of pixels in multiple color intervals in the image.

[0190] In some embodiments, referring to FIG3F and FIG3E, step 10221 can be implemented by steps 102211 to 102212, which are described in detail below.

[0191] In step 102211, in response to the proportion of pixels in the image belonging to a preset color range being within a preset pixel proportion range, the preset color range is used as the target color range to which the pixels of the second region of the target object belong.

[0192] In some embodiments, in response to the proportion of pixels in the image belonging to a preset color range being within a preset pixel proportion range (e.g., 8% to 32%), that is, the distribution range of the proportion of pixels in the second region in the pattern being greater than or equal to the minimum value in the preset pixel proportion range and less than or equal to the maximum value in the preset pixel proportion range, the preset color range is taken as the target color range to which the pixels of the second region of the target object belong.

[0193] In some embodiments, the preset color range is an empirical color range obtained by statistically analyzing the color values ​​of pixels belonging to the labeled second region in the image sample.

[0194] Taking a human as the target object in the image as an example, a preset color range is used to represent empirical color values ​​for skin tone. Assuming that different colors are represented by combinations of the red, green, and blue channels in the RGB color space (e.g., red in the RGB color space), with each channel having an integer value between 0 and 255, 256 × 256 × 256 = 16,777,216 colors can be represented using the RGB color space. These 16,777,216 colors are then arranged according to color gradients, such as from white to light yellow, from light yellow to yellow, from yellow to orange, and from orange... Colors can range from red to yellow. For example, RGB values ​​([248, 255], [197, 234], [183, 230]) can be used as a preset color range. A color range can also be an interval in which color values ​​in a color space are divided equally or a set of discrete color values. For example, the empirical color values ​​corresponding to skin color are (248, 220, 190) and (248, 201, 160). Then the preset color range can be represented as {(248, 220, 190), (248, 201, 160)}. The embodiments of this application do not limit the specific selection method of the color range.

[0195] Taking the target object in the image as an example, the preset color range is used to represent the color value of a local area of ​​the object. For example, when the object is a car, the preset color range can represent the color of the wheel. For example, if the wheel is black, the preset color range can be represented as RGB values ​​([0, 30], [0, 30], [0, 30]).

[0196] In step 102212, in response to the fact that the proportion of pixels in the image belonging to the preset color range is not within the preset pixel proportion range, a target color range is determined from multiple color ranges of the image, wherein the proportion of pixels in the target color range in the image is within the preset pixel proportion range, and the color difference between the preset color range and the target color range is less than the color difference threshold.

[0197] In some embodiments, referring to FIG30, FIG30 is a flowchart of determining the target color range provided in the embodiments of this application. When the proportion of pixels in the image that belong to the preset color range is not within the preset pixel proportion range, the target color range can be determined from multiple color ranges through the following steps 601 to 604, which are described in detail below.

[0198] In step 601, the color range in which the proportion of pixels in multiple color ranges falls within a preset pixel proportion range is obtained as a candidate color range.

[0199] In some embodiments, in response to the fact that the proportion of pixels in the image belonging to a preset color range is not within the preset pixel proportion range (e.g., less than the minimum value of the preset pixel proportion range, or greater than the maximum value of the preset pixel proportion range), a candidate color range is determined from multiple color ranges of the image, wherein the proportion of pixels in the candidate color range of the image is within the preset pixel proportion range.

[0200] In step 602, it is determined whether the color difference between the candidate color range and the preset color range is less than the preset color difference threshold.

[0201] In some embodiments, in response to a color difference value being less than a preset color difference threshold, the process proceeds to step 603; in response to a color difference value being greater than or equal to a preset color difference threshold, the process proceeds to step 604.

[0202] In step 603, the candidate color range is used as the target color range.

[0203] In some embodiments, candidate color intervals whose color difference with a preset color interval is less than a preset color difference threshold are used as target color intervals. For example, each color interval may include multiple color values ​​that are consecutive, and each interval includes the same number of multiple color values. The difference between the maximum color value of the preset color interval and the maximum color value of the candidate color interval is less than the color difference threshold.

[0204] For example, suppose the preset color range is ([248, 255], [197, 234], [183, 230]), and the preset pixel ratio range is 8% to 32%. In response to a preset color range where the pixel ratio is 40%, which is not within the preset pixel ratio range, candidate color ranges are determined from multiple color ranges in the image. For instance, if the pixel ratio of the color range ([244, 251], [190, 230], [180, 220]) is 30%, which is within the preset pixel ratio range, it is considered a candidate color range. The preset color range ([248, 255], [197, 234], [183, 230]) and the candidate color range ([244, 251], [190, 230], [180, 220]) are compared. If the color difference between the two color intervals is 18 (the difference between the maximum sums of the RGB values ​​of the two color intervals, such as the maximum sum of the RGB values ​​of ([248, 255], [197, 234], [183, 230]) being 255+234+230=719, and the maximum sum of the RGB values ​​of ([244, 251], [190, 230], [180, 220]) being 251+230+220=701), which is less than the preset color difference threshold of 30, then the candidate color interval ([244, 251], [190, 230], [180, 220]) is taken as the target color interval. Here, the calculation of the color difference between the two color intervals is only an example, and the specific calculation method is not limited in the embodiments of this application.

[0205] For example, the color difference threshold can be set in the following ways: 1) by relying on human experience; 2) by determining based on big data statistical results, for example, by calculating the average difference between adjacent color intervals based on different color intervals marked by humans in multiple samples, and using it as the color difference threshold.

[0206] In some embodiments, after determining the target color interval from multiple color intervals of the image, the preset color interval can be updated to the target color interval (corresponding to the candidate color interval in step 603), wherein the updated preset color interval is used to perform a second detection method on the new image to be encoded.

[0207] Taking a person as the target object in the image as an example, such as in a live broadcast scene, the difference in light intensity may cause the color value of the same yellow skin to be different under different lighting conditions. By introducing a color difference threshold, the skin area (second region) under different lighting conditions can be better detected. Suppose that the preset color range is updated in the second detection method of the first frame image, it means that the lighting has changed. For the second frame image, if the second detection method is continued to be used, the second region is detected according to the updated preset color range to adapt to the change in lighting.

[0208] In step 604, the preset color range is used as the target color range.

[0209] In some embodiments, when the color difference between the candidate color range and the preset color range is greater than or equal to the preset color difference threshold, the preset color range is used as the target color range, and subsequent processing steps are performed.

[0210] By introducing a color difference threshold, the influence of light on the detection accuracy of the second region (light-sensitive region) is considered. This allows the use of a new preset color range when performing the second detection method on subsequent images to be encoded. This enables the second region in the video frame to be adaptively matched according to changes in lighting, thereby improving the accuracy of the second region detection.

[0211] Referring again to Figure 3E, in step 10222, the proportion of pixels in each macroblock that belong to the target color range is used as the probability that the macroblock includes the second region.

[0212] In some embodiments, the proportion of pixels belonging to the target color range in each macroblock is obtained as the probability that the macroblock includes the second region. For example, if the proportion of pixels belonging to the target color range in macroblock 1 is 80 / 100 (assuming 100 is the number of pixels included in macroblock 1), then the probability that macroblock 1 includes the second region is 0.8.

[0213] In step 10223, a second detection result is generated based on the probability that each macroblock includes the second region.

[0214] In some embodiments, the probability that each macroblock includes the second region is combined to form a second detection result. For example, the second detection result is represented as "[macroblock 1: 0.01, macroblock 2: 0.10, ..., macroblock N: 0.02]".

[0215] In some embodiments, referring to FIG3G, before performing the second detection method on each macroblock in the image and obtaining the second detection result, the following steps 10224 to 10225 can be performed, which are described in detail below.

[0216] In step 10224, for multiple color intervals, the number of pixels in the image belonging to each color interval is determined, wherein the multiple color intervals are obtained by dividing the color space.

[0217] In some embodiments, the color value of each pixel in the image is determined (e.g., the RGB value of the pixel is obtained and mapped to a color number as the color value, such as mapping the RGB value (255, 255, 255) corresponding to white to 0). For each color value, the color interval to which the color value belongs is determined (e.g., one color interval corresponds to multiple color values). The number of pixels in the image belonging to the color interval is determined. Each color interval may include multiple color values ​​with consecutive values, and the number of multiple color values ​​included in each interval is the same. Multiple color intervals are obtained by dividing the color space.

[0218] In step 10225, for each color interval, the ratio of the number of pixels belonging to the color interval to the total number of pixels in the image is used as the proportion of pixels belonging to the color interval in the image.

[0219] In some embodiments, when the target object is a living organism in the image, the first region is the facial region of the organism (e.g., the facial region of a person), and the second region is the skin region of the organism (see FIG7C, FIG7C is a schematic diagram of the second region provided in the embodiments of this application, for example, the skin regions corresponding to the arms and neck of the target object shown in FIG7C are taken as the second region).

[0220] In some embodiments, when the target object is an image of an object, the first region is the main body region of the object (e.g., the main body of a car (excluding the wheels)) and the second region is a local region of the object (e.g., the headlights, wheel rims, etc. of a car).

[0221] For example, a local region is a distinctive local area within the main body region. It refers to those areas that can significantly represent the characteristics of the object and are crucial for the identification and classification of the object. For example, it can include the following types of regions: 1) Texture feature regions, regions with unique texture patterns, such as patterns on clothing, textures in nature (such as leaves, animal fur, etc.); 2) Color feature regions, specific colors or color combinations that can be used to distinguish different parts of an object, such as the distinctive color of clothing, the color of a vehicle, etc.; 3) Shape feature regions, specific geometric shapes or contours, such as the symmetry of an object, circular or elliptical regions, etc.

[0222] In some embodiments, referring to FIG3H, step 1022 shown in FIG3C can be implemented by steps 10226 to 10228, which are described in detail below.

[0223] In step 10226, the flatness of each macroblock in the image is statistically analyzed.

[0224] In some embodiments, referring to FIG3I, step 10226 shown in FIG3H can be implemented by performing steps 102261 to 102263 for each macroblock, as described in detail below.

[0225] In step 102261, the macroblock is grayscaled to obtain the grayscale value of each pixel in the macroblock.

[0226] In some embodiments, the grayscale value of each pixel can be calculated by a weighted average method, such as by weighting the RGB values ​​of each pixel to obtain the grayscale value of the pixel. For example, the weights of R, G and B are 0.299, 0.587 and 0.114, respectively. The grayscale value of each pixel can also be determined by an average method, that is, by adding the RGB values ​​and dividing by 3, or by using other more complex grayscale methods. The embodiments of this application do not limit the specific grayscale method.

[0227] In step 102262, the average gray value of the macroblock is determined based on the gray value of each pixel.

[0228] In some embodiments, each pixel of the macroblock is traversed, the gray values ​​of all pixels are summed to obtain the gray sum, and the gray sum is divided by the number of pixels in the macroblock (for example, for a 16×16 macroblock, this number is 256) to obtain the gray average of the macroblock.

[0229] In step 102263, the gray-level variance of the macroblock is determined based on the gray-level mean, which is used as the flatness of the macroblock.

[0230] In some embodiments, all pixels in a macroblock are traversed, the square of the difference between the gray value and the mean of each pixel is calculated, these squared differences are accumulated, and then divided by the total number of pixels in the macroblock to obtain the gray variance of the macroblock. The gray variance is used as the flatness of the macroblock. Here, the smaller the value of the gray variance of the macroblock, the higher the flatness of the macroblock.

[0231] Referring again to Figure 3H, in step 10227, the proportion of pixels belonging to the target color range in each macroblock of the image is counted, and the proportion is used as the probability that the macroblock includes the second region of the target object, where the target color range is the color value range of the second region of the target object.

[0232] In some embodiments, the proportion of pixels belonging to the target color range in each macroblock is used as the probability that the macroblock includes the second region. For specific implementation details, please refer to the description of step 10222 above, which will not be repeated here.

[0233] In step 10228, a second detection result is generated based on the flatness and probability of each macroblock.

[0234] In some embodiments, the flatness and probability of each macroblock are statistically analyzed, and the flatness and probability of each macroblock are combined into a second detection result. For example, the second detection result is represented as "[macroblock 1: (flatness 0.90, probability 0.01), macroblock 2: (flatness 0.90, probability 0.10), ..., macroblock N: (flatness 0.80, probability 0.02)]".

[0235] By generating a second detection result based on the flatness of each macroblock and the probability of containing the second region, the flatness reflects the texture features of the macroblock, and the probability reflects the color matching degree. This achieves a comprehensive judgment of the second region from both texture and color dimensions, improving the richness and accuracy of the second detection result.

[0236] In other embodiments, the flatness of each macroblock in the image is statistically analyzed, and the flatness of each macroblock is combined into a second detection result.

[0237] Through steps 1021 and 1022, at least one image detection method is performed on the image according to the preset priority of multiple image detection methods to obtain the image detection result. Specifically, the priority of the first detection method is higher than that of the second detection method. The priority setting mainly considers sharpness and encoding bitrate. Specifically, in order to achieve the technical effect of higher sharpness of target regions (such as the first region and the second region) in the image under low bitrate environment, this technical effect is measured by the bitrate allocated to macroblocks. That is, the higher the bitrate allocated to macroblocks in the target region of the same image, the higher the sharpness of the macroblocks.

[0238] The reason why the first detection method has a higher priority than the second detection method is that the first detection method detects the first region, which has the most significant impact on the perception of clarity (for example, in a live broadcast scenario, the focus is on the streamer's face). Therefore, the first detection method is executed first. If the first region is detected, a quantization parameter setting scheme suitable for the first region is then adopted (see the explanation of steps 1031 to 1032 below) to make the bitrate allocated to the first region higher than that of non-first regions, so as to ensure the clarity of the first region and save bitrate.

[0239] When the first region is not detected or there are insufficient computing resources to execute the first detection method, the second detection method is then used to detect the second region in the image (such as the skin area of ​​the anchor in a live broadcast scene) and the flatness of each macroblock. The probability of each macroblock including the second region is combined with the flatness as the second detection result. The corresponding quantization parameter setting scheme is adopted (see the explanation of steps 1033 to 1035 below) to make the bit rate of the second region higher than that of the non-second region, thereby ensuring the clarity of the second region and saving bit rate.

[0240] By setting quantization parameters in a targeted manner based on the detection results of different detection methods, it is possible to ensure that key regions in the image are allocated a higher bitrate, thereby achieving the beneficial effect of better clarity in key regions under lower coding bitrate conditions.

[0241] Referring again to Figure 3A, in step 103, quantization parameters are set for at least one macroblock in the image based on the detection results.

[0242] In some embodiments, when the detection result is a first detection result and the first detection result characterizes a first region of the target object in the image, referring to FIG3J, step 103 shown in FIG3A can be implemented by steps 1031 to 1032, which are described in detail below.

[0243] In step 1031, macroblocks that at least partially overlap with the first region are identified.

[0244] In some embodiments, macroblocks that intersect with the first region are identified, such as macroblocks that partially overlap with the first region or macroblocks that are entirely located within the first region.

[0245] For example, see Figure 7B, the first macroblock in the first row is a macroblock that partially overlaps with the first region, and the second macroblock in the first row is a macroblock that is completely located within the first region.

[0246] In step 1032, a first quantization parameter is set for macroblocks that intersect with the first region, and a second quantization parameter is set for macroblocks that do not intersect with the first region, wherein the first quantization parameter is smaller than the second quantization parameter.

[0247] In some embodiments, a first quantization parameter is set for macroblocks that intersect with the first region, and a second quantization parameter is set for macroblocks that do not intersect with the first region. The first quantization parameter is less than the second quantization parameter. The quantization parameter (QP) is the sequence number of the quantization step (QStep). When QP takes the minimum value (e.g., 0), it represents the finest quantization, in which most details in the macroblock will be preserved. When QP takes the maximum value (e.g., 51), it represents the coarsest quantization, in which details in the macroblock will be lost. During the encoding process, the encoder can automatically change the value of QP according to the actual situation of the image (keeping the first quantization parameter less than the second quantization parameter during this process), thereby changing the QStep. The embodiments of this application do not limit the specific values ​​of the first quantization parameter and the second quantization parameter.

[0248] For example, the quantization parameters for each macroblock can be determined using Rate Distortion Optimization (RDO). Assume the first quantization parameter corresponds to a first quantization parameter range, and the second quantization parameter corresponds to a second quantization parameter range, where the minimum value of the second quantization parameter range is greater than the maximum value of the first quantization parameter range. During the encoding process, the encoder determines the final quantization parameters for macroblocks that intersect with the first region (selecting the first quantization parameter from the first quantization parameter range) and macroblocks that do not intersect with the first region (selecting the second quantization parameter from the second quantization parameter range), respectively, using the Rate-Distortion Optimization (RDO) algorithm: For each macroblock, it iterates through the candidate quantization parameters within its respective quantization parameter range, calculates the rate-distortion cost (RDcost) corresponding to each candidate quantization parameter value (i.e., the weighted sum of image quality loss and bitrate consumption after macroblock encoding), selects the candidate quantization parameter value with the minimum rate-distortion cost as the final quantization parameter of the macroblock, and ensures that the final quantization parameter (first quantization parameter) of macroblocks that intersect with the first region is always less than the final quantization parameter (second quantization parameter) of macroblocks that do not intersect with the first region, thereby optimizing the overall encoding efficiency while prioritizing the preservation of details in the first region.

[0249] For example, suppose the range of QP is [0, 51], where QP = 0 corresponds to the finest quantization (highest image quality, maximum bitrate) and QP = 51 corresponds to the coarsest quantization (lowest image quality, minimum bitrate). Set the first quantization parameter interval corresponding to the first quantization parameter to [19, 25] and the second quantization parameter interval corresponding to the second quantization parameter to [31, 37]. A macroblock A in the image is a macroblock that intersects with the first region (such as the face region). The encoder traverses each candidate QP value within [19, 25] and calculates the rate-distortion cost corresponding to each candidate value. Suppose that when candidate QP = 19, the image quality loss (measured by mean square error MSE) after encoding macroblock A is 6 and the bitrate consumption is 125 bits. Calculated with a weighting coefficient λ = 0.1, the rate-distortion cost = 6 + 0.1 × 125 = 18.5, and 18.5 is the minimum rate-distortion cost. Then the first quantization parameter of macroblock A is 19. For example, if a macroblock B in an image is a macroblock that does not intersect with the first region, and the encoder traverses each candidate QP value within [31,37], assuming that when candidate QP = 31, the MSE of macroblock B is 22, the bit rate consumption is 42 bits, and the rate distortion cost is 22 + 0.1 × 42 = 26.2, and 26.2 is the minimum rate distortion cost, then the second quantization parameter of macroblock B is 31.

[0250] In some embodiments, where the detection result is a second detection result, and the second detection result characterizes the flatness of each macroblock and the probability that the macroblock includes a second region of the target object, step 103 shown in FIG3K and FIG3A can be implemented by performing steps 1033 to 1035 for each macroblock, as described in detail below.

[0251] In step 1033, the target flatness level of the macroblock is determined among a plurality of preset flatness levels.

[0252] In some embodiments, the flatness of a macroblock is determined to be among a preset plurality of flatness levels, and the target flatness levels are divided into high, medium and low according to the value. For example, the flatness value (the value of grayscale variance) range is [0.005, 0.05) as high flatness level, [0.05, 0.01) as medium flatness level, and [0.01, 0.1) as low flatness level. The higher the flatness level, the flatter the image in the macroblock (for example, the flatness of the background area in the image is higher than the flatness of the face area in the image). The embodiments of this application do not limit the specific number of flatness levels and the division range.

[0253] For example, the pre-defined division of multiple flatness levels can be achieved as follows: statistically analyze the flatness of image samples of the same type (e.g., live video, advertising video) to obtain the flatness value range, divide the flatness value range into multiple sub-intervals, and each sub-interval corresponds to a level.

[0254] In step 1034, the target probability level in which the probability of the macroblock including the second region is located among a preset plurality of probability levels is determined.

[0255] In some embodiments, the probability that a macroblock includes a second region of a target object is determined to be at the target probability level, which is divided into high, medium, low, and middle levels according to a set of preset probability levels.

[0256] For example, suppose the probability value range is [0.1, 0.3) as low probability level, [0.3, 0.6) as medium probability level, and [0.6, 1] as high probability level. The higher the probability level, the more overlapping the macroblock and the second region are. The embodiments of this application do not limit the specific number of probability levels and the division range.

[0257] In step 1035, the quantization parameter table is queried based on the target flatness level and the target probability level to obtain the preset quantization parameters associated with the target flatness level and the target probability level, which are used as the quantization parameters of the macroblock. The quantization parameter table includes the mapping relationship between multiple flatness levels, multiple probability levels and multiple preset quantization parameters. The preset quantization parameters are negatively correlated with the target flatness level and the probability level.

[0258] In some embodiments, a preset quantization parameter associated with the target flatness level and the target probability level is obtained by querying the quantization parameter table through the target flatness level and the target probability level, and used as the quantization parameter of the macroblock.

[0259] Taking an image with two levels each for probability and flatness, where the target object is a live streamer in a live broadcast scene, and the second region corresponds to the streamer's skin, as an example, the quantization parameter table can be represented by Table 1:

[0260] Table 1

[0261] For example, see Figure 7D, which is a schematic diagram of the probability level and flatness level of an image provided in an embodiment of this application. The value range of the quantization parameters in Table 1 can be set according to the application scenario. During the image encoding process, the encoder can automatically change the value of QP according to the actual situation of the image. This embodiment of the application does not restrict the value of the quantization parameters.

[0262] By mapping the flatness of macroblocks to a target flatness level among multiple flatness levels, and mapping the probability of a macroblock containing a second region to a target probability level among multiple probability levels, and by querying a preset quantization parameter table based on the target flatness level and target probability level to determine the corresponding preset quantization parameters, precise allocation of encoding bitrate and differentiated preservation of image details can be achieved. For example, for regions that need to be highlighted (such as the skin area of ​​a streamer in a live broadcast scene, corresponding to a high probability level and a high flatness level), rich details are preserved using low quantization parameters; for non-critical regions with redundant details (such as the streamer's clothing or texture areas, corresponding to a low probability level and a low flatness level), bitrate is reasonably saved using high quantization parameters; at the same time, other regions (such as facial features and desktop backgrounds) are matched with medium quantization parameters to balance details and bitrate consumption. This avoids the problem of loss of key details or waste of bitrate caused by uniform quantization, and can adaptively adjust the encoding strategy according to the characteristics of image content. Ultimately, while ensuring the display quality of core areas, it improves overall encoding efficiency and optimizes bandwidth resource usage.

[0263] In some embodiments, where the detection result is a second detection result, and the second detection result represents the probability that each macroblock includes a second region of the target object, the following processing is performed for each macroblock: generating quantization parameters of the macroblock based on the probability that the macroblock includes a second region of the target object, wherein the probability that the macroblock includes a second region is negatively correlated with the quantization parameters of the macroblock.

[0264] For example, the probability that a macroblock includes the second region of the target object is negatively correlated with the quantization parameter of the macroblock. That is, the higher the probability value of the macroblock, the lower the quantization parameter corresponding to the macroblock. During the image encoding process, the encoder can automatically change the value of QP according to the actual situation of the image. This application embodiment does not restrict the value of the quantization parameter.

[0265] By generating quantization parameters based on the probability that a macroblock contains the second region of the target object, the higher the probability that a macroblock contains the second region, the lower the corresponding quantization parameter value; and the lower the probability that a macroblock contains the second region, the higher the corresponding quantization parameter value. This approach can ensure the image details of macroblocks with high probability while reasonably compressing macroblocks with low probability through high quantization parameters, thus achieving a balance between image quality in the core area and overall coding efficiency.

[0266] In some embodiments, where the detection result is a second detection result and the second detection result characterizes the flatness of each macroblock, the following processing is performed for each macroblock: generating quantization parameters of the macroblock based on the flatness of the macroblock, wherein the flatness of the macroblock is negatively correlated with the quantization parameters of the macroblock.

[0267] For example, the flatness of a macroblock (the gray-level variance of the macroblock) is negatively correlated with the quantization parameter of the macroblock. That is, the higher the gray-level variance of the macroblock (the less flat the image in the macroblock), the lower the quantization coefficient of the macroblock. During the image encoding process, the encoder can automatically change the value of QP according to the actual situation of the image. This application embodiment does not restrict the value of the quantization parameter.

[0268] When the detection result is the second detection result representing the flatness of each macroblock, a quantization parameter is generated for each macroblock based on its flatness. Macroblock flatness and quantization parameter are negatively correlated; that is, the higher the macroblock flatness (the smaller the gray-level variance), the higher the quantization parameter value; the lower the macroblock flatness (the larger the gray-level variance), the lower the quantization parameter value. This achieves the goal of setting low quantization parameters to preserve details for macroblocks with low flatness (rich texture) and setting high quantization parameters to compress redundancy for macroblocks with high flatness (simple texture). While ensuring the quality of key texture areas of the image, it also improves the overall coding efficiency and achieves an optimized balance between image quality and bitrate.

[0269] Through steps 1031 to 1035, more suitable quantization parameters are set for different macroblocks based on the detection results of different image priorities, thereby achieving more refined encoding control. This helps to transmit higher quality images under limited bandwidth and meet users' needs for high-definition images.

[0270] In some embodiments, after setting quantization parameters for at least one macroblock in the image based on the detection results, referring to Figure 3L, the following steps 1036 to 1037 can also be performed for each macroblock, which are described in detail below.

[0271] In step 1036, the amount of change in the quantization parameters of the macroblock relative to the reference quantization coefficients is determined, wherein the reference quantization coefficients are the quantization coefficients of the macroblock located at the same position as the macroblock in the encoded image.

[0272] For example, suppose the quantization parameter of macroblock 1 in the image n+4 to be encoded is 20, and the quantization coefficient of macroblock 1 in the previously encoded image n+3 is 40 (reference quantization coefficient). The change in the quantization parameter of macroblock 1 compared to the reference quantization coefficient is 20-40=-20.

[0273] In step 1037, the product of the change value and the adjustment coefficient is determined, and the quantization parameters of the macroblock are updated based on the sum of the product and the reference quantization coefficient, wherein the adjustment coefficient is positively correlated with the number of times the second region appears consecutively at the position of the macroblock.

[0274] For example, the adjustment factor can be expressed as y = ax, where x represents the number of times the second region appears consecutively at the position of the macroblock, y represents the adjustment factor, and a is a constant greater than zero.

[0275] Continuing with the example above, assuming the second region appears in all images from frame n to frame n+4, and a is 0.02, then the adjustment factor is 0.02 × 5 = 0.1. The product of the change value (-20) between image n+4 and image n+3 and the adjustment factor is -2. The sum of the product and the reference quantization factor is 40 + (-2) = 38. 38 is used as the updated quantization parameter of macroblock 1 in image n+4 to be encoded. Here, the adjustment factor is positively correlated with the number of times the second region appears consecutively at the macroblock position. That is, the more times the second region appears consecutively at the same macroblock position in multiple images to be encoded, the larger the value of the adjustment factor (for example, the maximum value is 1).

[0276] By introducing adjustment coefficients, the quantization parameters of macroblocks at the same position do not change abruptly during the encoding process. In other words, the values ​​of the quantization parameters are smoothly processed, avoiding large changes in the quantization parameters.

[0277] Referring again to Figure 3A, in step 104, each macroblock is encoded based on the quantization parameters of each macroblock.

[0278] Taking the H.264 coding standard as an example, encoding each macroblock based on its quantization parameters can be achieved in the following ways: The image data corresponding to each macroblock is transformed from the pixel domain to the frequency domain using transformation methods such as Discrete Cosine Transform (DCT) to obtain the transform coefficients of each macroblock. The transform coefficients are then quantized using quantization parameters (QP). During quantization, the precision of the transform coefficients is reduced according to the QP, thereby reducing data redundancy. The quantized transform coefficients are then encoded using methods such as Context-based Adaptive Binary Arithmetic Coding (CABAC) or Context-Adaptive Variable Length Coding (CAVLC) to obtain the macroblock encoded data for each macroblock.

[0279] In step 105, image encoding data of the image is generated based on the macroblock encoding data obtained after encoding.

[0280] In some embodiments, after each macroblock is encoded, its macroblock encoded data is combined. This macroblock encoded data may include macroblock header information (such as macroblock type, quantization parameters, etc.) and macroblock compressed data. Specifically, the macroblock encoded data of all macroblocks are combined and encapsulated according to the H.264 syntax structure to form image encoded data of a frame of image to be encoded. Macroblock data can also be encapsulated into Network Abstraction Layer Units (NALUs) for easy storage or transmission.

[0281] Through steps 101 to 105, the image is detected according to the priority of multiple detection methods. Based on the detection results of different priorities, quantization parameters are set for each macroblock in the image. This can more effectively set more suitable quantization parameters for different macroblocks according to the detection results of image detection methods with different priorities, thereby achieving more refined encoding control. This helps to transmit higher quality images under lower encoding bitrate conditions or lower encoding standards, meeting users' needs for high-definition images.

[0282] Taking an image encoding device (terminal device or server) for image encoding as an example, the image encoding method provided in this application embodiment will be described. Referring to FIG4A, FIG4A is a schematic diagram of the thirteenth step of the image encoding method provided in this application embodiment, and the steps shown in FIG4A will be described in conjunction with the steps shown in FIG4A.

[0283] In step 201, the image to be encoded is obtained.

[0284] In some embodiments, an uncompressed original image is obtained (see the description of step 1011 above), and the original image is used as the image to be encoded; or, the original image is downsampled, and the resulting downsampled image is used as the image to be encoded (see the description of step 1012 above).

[0285] In step 202, the target color range corresponding to the target area of ​​the target object is determined based on the proportion of pixels in multiple color ranges in the image.

[0286] In some embodiments, referring to FIG4B, step 202 shown in FIG4A can be implemented by steps 2021 to 2022, which are described in detail below.

[0287] In step 2021, in response to the fact that the proportion of pixels in the image belonging to the preset color range is within the preset pixel proportion range, the preset color range is taken as the target color range to which the target area of ​​the target object in the image belongs.

[0288] In some embodiments, in response to the fact that the proportion of pixels in the image belonging to a preset color range (e.g., the range corresponding to empirical color values ​​representing skin tone) is in a preset pixel proportion range (e.g., 8% to 32%), that is, the distribution range of the proportion of pixels in the second region in the pattern is greater than or equal to the minimum value in the preset pixel proportion range and less than or equal to the maximum value in the preset pixel proportion range, the preset color range is taken as the target color range to which the pixels of the second region of the target object belong.

[0289] In step 2022, in response to the fact that the proportion of pixels in the image belonging to the preset color range is not within the preset pixel proportion range, a target color range is determined from multiple color ranges in the image, wherein the proportion of pixels in the target color range in the image is within the preset pixel proportion range, and the color difference between the preset color range and the target color range is less than the color difference threshold.

[0290] In some embodiments, in response to the fact that the proportion of pixels in the image belonging to a preset color range is not within the preset pixel proportion range (e.g., less than the minimum value of the preset pixel proportion range, or greater than the maximum value of the preset pixel proportion range), a target color range is determined from multiple color ranges of the image, wherein the proportion of pixels in the target color range in the image is within the preset pixel proportion range, and the color difference between the preset color range and the target color range is less than a preset color difference threshold.

[0291] Referring again to Figure 4A, in step 203, the proportion of pixels belonging to the target color range in each macroblock of the image is used as the probability that each macroblock includes the target region of the target object.

[0292] In some embodiments, the proportion of pixels belonging to the target color range in each macroblock is obtained as the probability that the macroblock includes the second region. For example, if the proportion of pixels belonging to the target color range in macroblock 1 is 80 / 100 (assuming 100 is the number of pixels included in macroblock 1), then the probability that macroblock 1 includes the second region is 0.8.

[0293] In step 204, quantization parameters are set for at least one macroblock based on the probability that each macroblock includes the target region.

[0294] In some embodiments, the following processing is performed for each macroblock: quantization parameters of the macroblock are generated based on the probability that the macroblock includes the target region (corresponding to the second region above), wherein the probability that the macroblock includes the target region is negatively correlated with the quantization parameters of the macroblock.

[0295] For example, the probability that a macroblock includes the target region of a target object is negatively correlated with the quantization parameter of the macroblock. That is, the higher the probability value of the macroblock, the lower the quantization parameter corresponding to the macroblock. During the image encoding process, the encoder can automatically change the value of QP according to the actual situation of the image. This application embodiment does not restrict the value of the quantization parameter.

[0296] In some embodiments, referring to Figure 4C, the quantization parameters of a macroblock are generated based on the probability that the macroblock includes the target region. This can be achieved by performing the following steps 2041 to 2043 for each macroblock, as described in detail below.

[0297] In step 2041, the target flatness level of the macroblock is determined among a plurality of preset flatness levels.

[0298] In some embodiments, the flatness of the macroblock is determined among a plurality of preset flatness levels, and the target flatness levels of high, medium and low are divided according to the value. For specific implementation methods, please refer to the description of step 1033 above, which will not be repeated here.

[0299] In step 2042, the target probability level in which the probability that the macroblock includes the target region is located among a preset plurality of probability levels is determined.

[0300] In some embodiments, the probability that the macroblock includes the second region of the target object is determined to be at the target probability level in a preset plurality of probability levels, which are divided into high, medium and low. For specific implementation methods, please refer to the description of step 1034 above, which will not be repeated here.

[0301] In step 2043, the quantization parameter table is queried based on the target flatness level and the target probability level to obtain the preset quantization parameters associated with the target flatness level and the target probability level, which are used as the quantization parameters of the macroblock. The quantization parameter table includes the mapping relationship between multiple flatness levels, multiple probability levels and multiple preset quantization parameters. The preset quantization parameters are negatively correlated with the target flatness level and probability.

[0302] In some embodiments, the preset quantization parameters associated with the target flatness level and the target probability level are obtained by querying the quantization parameter table through the target flatness level and the target probability level, and are used as the quantization parameters of the macroblock. For the setting of the quantization parameter table, please refer to the description of Table 1 above, which will not be repeated here.

[0303] In some embodiments, referring to FIG4D, after setting quantization parameters for at least one macroblock, the following steps 2044 to 2045 may be performed for each macroblock, as described in detail below.

[0304] In step 2044, the amount of change in the quantization parameters of the macroblock relative to the reference quantization coefficients is determined, wherein the reference quantization coefficients are the quantization coefficients of the macroblock located at the same position as the macroblock in the encoded image.

[0305] For example, suppose the quantization parameter of macroblock 1 in the image 2 to be encoded is 40, and the quantization coefficient of macroblock 1 in the encoded image 1 is 20 (reference quantization coefficient). The change in the quantization parameter of macroblock 1 compared to the reference quantization coefficient is 40 - 20 = 20.

[0306] In step 2045, the product of the change value and the adjustment coefficient is determined, and the quantization parameters of the macroblock are updated based on the sum of the product and the reference quantization coefficient, wherein the adjustment coefficient is positively correlated with the number of times the second region appears consecutively at the position of the macroblock.

[0307] Following the example above, assuming the adjustment coefficient is 0.1, the product of the change value (20) and the adjustment coefficient is 2. The sum of the product and the reference quantization coefficient is 20+2=22. 22 is used as the updated quantization parameter of macroblock 1 in the image 2 to be encoded. Here, the adjustment coefficient is positively correlated with the number of times the second region appears consecutively at the position of the macroblock. That is, the more times the second region appears consecutively at the same macroblock position in multiple images to be encoded, the larger the value of the adjustment coefficient (for example, the highest value is 1).

[0308] Referring again to Figure 4A, in step 205, each macroblock is encoded based on the quantization parameters of each macroblock.

[0309] For specific implementation details, please refer to the description in step 104 above, which will not be repeated here.

[0310] In step 206, image encoding data of the image is generated based on the macroblock encoding data obtained after encoding.

[0311] For specific implementation details, please refer to the description in step 105 above, which will not be repeated here.

[0312] Taking an image decoding device (terminal device or server) for image decoding as an example, the image decoding method provided in this application embodiment will be described. Referring to Figure 5A, which is a first flowchart of the image decoding method provided in this application embodiment, the steps shown in Figure 5A will be used for explanation.

[0313] In step 301, image encoded data is received, wherein the image encoded data is obtained through the image encoding method provided in the embodiments of this application.

[0314] In some embodiments, image encoded data is received, wherein the image encoded data is obtained by encoding the image to be encoded in steps 101 to 105.

[0315] In step 302, the macroblock encoded data is decoded based on the quantization parameters of each macroblock to obtain the macroblock decoded data for each macroblock.

[0316] In some embodiments, the image encoded data includes macroblock encoded data for each macroblock (see the description in steps 104 to 105). During decoding, the decoder reads the quantization parameters from each macroblock encoded data, such as macroblock header information (e.g., macroblock type, quantization parameters, etc.) included in the macroblock encoded data. It then uses an inverse quantization process corresponding to the encoder to recover the pixel information of the macroblock. Inverse quantization is the inverse process of quantization; it uses the quantization parameters to recover the original pixel values. Finally, the decoder reassembles the recovered pixel values ​​into macroblocks to obtain the macroblock decoded data for each macroblock.

[0317] In step 303, the macroblock decoding data of each macroblock is combined into a decoded image.

[0318] In some embodiments, the decoded macroblock decoding data is sorted according to the position of the macroblock in the original image, and the sorted macroblock decoding data is spliced ​​together according to its position to form a complete decoded image.

[0319] In some embodiments, referring to FIG5B, after obtaining the decoded image, steps 304 to 306 may also be performed, which are described in detail below.

[0320] In step 304, the decoded image undergoes a first enhancement process to obtain a first enhanced image.

[0321] In some embodiments, feature extraction processing is performed on the decoded image to obtain image features; image reconstruction processing is performed based on the image features to obtain a first enhanced image.

[0322] In some embodiments, the decoded image can be first enhanced by a pre-trained image super-resolution model (such as Super-Resolution Convolutional Neural Network (SRCNN), Super-Resolution Generative Adversarial Network (SRGAN), etc.) to obtain a first enhanced image. The embodiments of this application do not limit the image super-resolution model.

[0323] For example, taking SRCNN for the first enhancement process, the first enhancement process includes: first, preprocessing the decoded image, for example, using a bicubic interpolation algorithm to enlarge the decoded image to the target size; next, inputting the enlarged decoded image into a three-layer convolutional neural network, extracting image features through the convolutional layers, and performing nonlinear mapping to fit the mapping relationship from low resolution to high resolution; finally, outputting the high-resolution image result, which is the first enhanced image.

[0324] In step 305, the target grayscale value of each pixel in the first enhanced image is obtained.

[0325] In some embodiments, referring to FIG5C, step 305 shown in FIG5B can be implemented by steps 3051 to 3054, which are described in detail below.

[0326] In step 3051, the grayscale value of each pixel in the first enhanced image is determined.

[0327] In some embodiments, the grayscale value of each pixel can be calculated by a weighted average method, such as by weighting the RGB values ​​of each pixel to obtain the grayscale value of the pixel. For example, the weights of R, G and B are 0.299, 0.587 and 0.114, respectively. The grayscale value of each pixel can also be determined by an average method, that is, by adding the RGB values ​​and dividing by 3, or by using other more complex grayscale value calculation methods. The embodiments of this application do not limit the specific grayscale value calculation method.

[0328] In step 3052, for each grayscale value, the number of pixels with grayscale values ​​in the first enhanced image is determined, and an original grayscale histogram is constructed based on the number.

[0329] In some embodiments, an original grayscale histogram is constructed based on the grayscale value of each pixel in the first enhanced image.

[0330] In step 3053, grayscale values ​​are converted based on the original grayscale histogram to obtain the target grayscale histogram.

[0331] In some embodiments, the original gray-level histogram can be mapped to a target gray-level histogram using a histogram equalization algorithm. Specifically, the target gray-level histogram (or cumulative histogram) is obtained by calculating the cumulative distribution function (CDF) based on the original gray-level histogram.

[0332] In step 3054, the target gray value of each pixel in the first enhanced image is determined based on the target gray histogram.

[0333] For example, the determination of the target grayscale value (or target brightness value) can be expressed as performing the following processing on each pixel: Target grayscale value = (cumulative histogram value - minimum cumulative histogram value) * (maximum grayscale value - minimum grayscale value) / (total number of pixels - minimum cumulative histogram value) + minimum grayscale value.

[0334] The cumulative histogram value (or target grayscale histogram value) represents the number of pixels in the first enhanced image whose grayscale value is less than or equal to the current pixel's grayscale value. It reflects the "popularity" or "cumulative frequency" of the current pixel's grayscale value in the image. The minimum cumulative histogram value (usually 0) indicates that no pixel has a grayscale value lower than this minimum possible value. Subtracting this minimum value in the formula ensures that the calculation starts from 0, that is, from a reference point to map to the target grayscale value. The maximum and minimum grayscale values ​​define the range of the target grayscale value. For example, in an 8-bit image, the maximum grayscale value is 255 (representing white, the pixel with the highest brightness), and the minimum grayscale value is 0 (representing black, the pixel with the lowest brightness).

[0335] Referring again to Figure 5B, in step 306, the first enhanced image is subjected to a second enhancement process based on the target grayscale value of each pixel to obtain the second enhanced image.

[0336] In some embodiments, the grayscale value of each pixel in the first enhanced image is updated to the target grayscale value corresponding to the pixel to obtain the second enhanced image.

[0337] In some embodiments, the saturation of each pixel in the region other than the second region in the second enhanced image can be enhanced (e.g., the color space of the second enhanced image is converted to the HSL color space, and the value of the S (saturation) channel is directly adjusted to increase the saturation) to obtain a new second enhanced image. For example, in a live streaming scenario, by removing the pixels in the second region during saturation enhancement, the problem of reddish skin tone can be effectively avoided. The optimized new second enhanced image can be both vivid in color and does not appear "fake".

[0338] Through steps 301 to 306, after acquiring the decoded image, on the one hand, the clarity of the decoded image is improved through the first enhancement process (image super-resolution processing), and on the other hand, the color of the decoded image is enhanced through the second enhancement process and saturation enhancement, which can alleviate the loss of image color when the bit rate decreases.

[0339] Taking an image decoding device (terminal device or server) for image decoding as an example, the image decoding method provided in this application embodiment will be described. Referring to Figure 6, which is a fourth flowchart of the image decoding method provided in this application embodiment, the steps shown in Figure 6 will be used for explanation.

[0340] In step 401, image encoded data is received, wherein the image encoded data is obtained by the image encoding method provided in the embodiments of this application.

[0341] In some embodiments, image encoded data is received, wherein the image encoded data is obtained by encoding the image to be encoded in steps 201 to 206.

[0342] In step 402, the macroblock encoded data is decoded based on the quantization parameters of each macroblock to obtain the macroblock decoded data for each macroblock.

[0343] In some embodiments, the image encoded data includes macroblock encoded data for each macroblock (see the description in steps 205 to 206). During decoding, the decoder reads the quantization parameters from each macroblock encoded data, such as macroblock header information (e.g., macroblock type, quantization parameters, etc.) included in the macroblock encoded data. It then uses an inverse quantization process corresponding to the encoder to recover the pixel information of the macroblock. Inverse quantization is the inverse process of quantization; it uses the quantization parameters to recover the original pixel values. Finally, the decoder reassembles the recovered pixel values ​​into macroblocks to obtain the macroblock decoded data for each macroblock.

[0344] In step 403, the macroblock decoding data of each macroblock is combined into a decoded image.

[0345] In some embodiments, the decoded macroblock decoding data is sorted according to the position of the macroblock in the original image, and the sorted macroblock decoding data is spliced ​​together according to its position to form a complete decoded image.

[0346] In some embodiments, after obtaining the decoded image, a first enhancement process can be performed on the decoded image to obtain a first enhanced image (see the description of step 304 above); the target gray value of each pixel in the first enhanced image can be obtained (see the description of step 305 above); and a second enhancement process can be performed on the first enhanced image using the target gray value of each pixel to obtain a second enhanced image (see the description of step 306 above).

[0347] The following describes specific examples of the application of the encoding / decoding method of this application in live streaming scenarios. As live video streaming has become a streaming media service that internet users encounter daily, users' demands for video clarity are also increasing. Currently, most live video streaming platforms have begun to adopt 1080p video resolution. To maintain high clarity at such a high resolution, a high encoding bitrate is required. For example, common live streaming platforms generally use a video bitrate of at least 2Mbps, while live streaming platforms with higher clarity requirements will use 6Mbps or even higher video bitrates. However, such a high bitrate places high demands on the user's downlink bandwidth. Insufficient bandwidth can easily lead to buffering and also puts enormous bandwidth cost pressure on the live streaming platform.

[0348] The image encoding and decoding methods provided in this application can reduce video bitrate without reducing image clarity. For example, in a live broadcast where the content is mainly about people and the "host" is the core of the video, and the host is showcasing products or performing talents in front of the camera, a live stream that was originally 1080p and 2Mbps only needs an encoding bitrate of 1Mbps to achieve the same clarity as before. This significantly reduces the bandwidth requirements of the user's mobile phone for watching live broadcasts, and also significantly reduces the bandwidth costs of the live broadcast platform. This application does not limit the live broadcast scenario or content. For example, this application can also be used for live broadcast scenarios such as game live broadcasts and sports event live broadcasts.

[0349] The following describes an exemplary application of the embodiments of this application in a live streaming scenario. Referring to Figure 8, Figure 8 is a first flowchart illustrating the application of the image encoding method and image decoding method provided in the embodiments of this application in a live streaming scenario. The steps shown in Figure 8 will be used for explanation.

[0350] In step 501, the first terminal device acquires video data.

[0351] In some embodiments, the first terminal device can capture uncompressed raw video streams and acquire video data by using a camera or other video input device connected to a computer.

[0352] In step 502, the first terminal device performs image preprocessing on the video data to obtain the video data to be encoded.

[0353] In some embodiments, the video data includes multiple video frames. The first terminal device performs image preprocessing (corresponding to the downsampling process mentioned above) on each video frame to obtain the video data to be encoded. The downsampling process can employ algorithms such as the Nearest Interpolation, Bilinear Interpolation, and Bicubic Interpolation to reduce the resolution of each video frame image. For example, if the original video frame resolution is 1920×1080 (1080p), the resolution of the video frame obtained after downsampling is 1280×720. The embodiments of this application do not limit the specific implementation of the downsampling process.

[0354] By downsampling, for example, downsampling the resolution from 1080p to 720p, the number of pixels in the video frames that need to be encoded is reduced by 65%, thereby achieving the beneficial effect of saving 30%-50% of the encoding bitrate and reducing the computational cost of video encoding under the same video encoding standard.

[0355] Here, the decrease in image clarity caused by reducing the number of pixels and lowering the encoding bitrate is compensated for by introducing better encoding control technology to the encoder on the broadcast end (corresponding to step 503 below) and using image post-processing technology on the viewer end (corresponding to step 507 below). This ensures that while the resolution of video frames is reduced, the image clarity is only slightly compromised, thus achieving the beneficial effect of reducing the computational cost of video encoding under the same video encoding standard.

[0356] In step 503, the first terminal device performs video encoding on the video data to be encoded based on the image encoding method provided in the embodiments of this application, and obtains encoded video data.

[0357] In some embodiments, video encoding is performed on the video data to be encoded based on the image encoding method provided in this application embodiment using a pre-set encoding standard (e.g., H.264 encoding standard) to obtain encoded video data. The image to be encoded in the image encoding method provided in this application embodiment can correspond to a keyframe in the video data.

[0358] Specifically, video frames are categorized into I-frames (keyframes, independently coded, corresponding to the image to be encoded mentioned above; multiple I-frames can exist in a long video stream), P-frames (forward prediction frames, encoded with reference to the previous I-frame or P-frame), and B-frames (bidirectional prediction frames, encoded with reference to the preceding and following frames). Next, each video frame is divided into multiple macroblocks (see the explanation of Figure 7A above). Pixel value prediction is performed for each macroblock at the same position between video frames. Specifically, I-frames use intra-frame prediction, i.e., predicting the current pixel value based on already encoded pixels; P-frames and... B-frames use intra-frame prediction or inter-frame prediction methods, which reduce redundant information by comparing pixel values ​​of adjacent frames. Next, the residual between the predicted value and the original pixel value is calculated, and the residual is subjected to Discrete Cosine Transform (DCT) and quantization to further reduce the amount of data. Finally, the coefficients and other information obtained after quantization are entropy encoded to generate the final coding bitrate, which is the encoded video data. Here, the quantization process is based on quantization parameters. The encoder can perform adaptive quantization on multiple video frames during the encoding process, automatically adjusting the value of the quantization parameters according to the content of the video sequence.

[0359] For example, the quantization parameter setting for each macroblock of an I-frame (corresponding to the image to be encoded above) can be achieved as follows: At least one image detection method is performed on the I-frame according to the priority of multiple image detection methods to obtain the detection result of the I-frame. Specifically, a first detection method is performed on the I-frame. If a first region (e.g., the anchor's face) including a target object (e.g., a broadcaster) is detected in the I-frame, a first detection result representing the image including the first region is generated (see the description of step 1021 above). If the first region is not detected in the I-frame or there are insufficient computing resources to perform the first detection method, a second detection method is performed on each macroblock in the I-frame to obtain a second detection result (see the description of step 1022 above). Quantization parameters are then set for at least one macroblock in the I-frame based on the detection result (see the description of step 103 above).

[0360] For example, a P-frame is used to record the differences between each macroblock in the current frame and the reference frame (either the previous I-frame or the previous P-frame). These differences include pixel differences and motion vector differences between macroblocks, where the motion vector difference represents the positional offset of a macroblock in the current frame relative to the corresponding macroblock in the reference frame. A DCT transform and quantization process are performed on the differences of each macroblock to obtain quantized coefficients. Entropy coding is then applied to these quantized coefficients to obtain the encoded data corresponding to the P-frame. Here, the quantization process is based on quantization parameters. During encoding, the encoder can determine the quantization parameters of each macroblock according to H.264 or other encoding standards, or it can determine the quantization parameters of each macroblock using the aforementioned scheme for setting quantization parameters for I-frames. The quantization parameter values ​​for each macroblock can be dynamically adjusted.

[0361] For example, a B-frame is used to record the differences between each macroblock in the current frame and the preceding and following reference frames (the preceding I-frame or P-frame and the following P-frame). These differences include pixel differences and motion vector differences between macroblocks, where motion vector differences represent the positional offset of a macroblock in the current frame relative to the corresponding macroblock in the preceding and following reference frames. A DCT transform and quantization process are performed on the differences of each macroblock to obtain quantized coefficients. Entropy coding is then applied to these quantized coefficients to obtain the encoded data corresponding to the B-frame. Here, the quantization process is based on quantization parameters. During encoding, the encoder can determine the quantization parameters of each macroblock according to H.264 or other encoding standards, or it can determine the quantization parameters of each macroblock using the same scheme described above for setting quantization parameters for I-frames. The quantization parameter values ​​for each macroblock can be dynamically adjusted.

[0362] For example, performing the second detection method on each macroblock in the I-frame and obtaining the second detection result can be achieved in the following way: Based on the proportion of pixels in multiple color intervals in the I-frame, determine the target color interval corresponding to the second region of the target object (e.g., the skin region of the anchor) (see the description of step 10221 above); take the proportion of pixels in each macroblock that belong to the target color interval as the probability that the macroblock includes the second region (see the description of step 10222 above); generate the second detection result based on the probability that each macroblock includes the second region (see the description of step 10223 above).

[0363] For example, when the detection result is the first detection result, and the first detection result represents a first region of the target object in the I-frame, the quantization parameters for at least one macroblock in the I-frame are set according to the detection result. This can be achieved by: determining the macroblocks that intersect with the first region (see the description of step 1031 above); setting the first quantization parameter for the macroblocks that intersect with the first region, and setting the second quantization parameter for the macroblocks that do not intersect with the first region, wherein the first quantization parameter is smaller than the second quantization parameter (see the description of step 1032 above).

[0364] For example, when the detection result is a second detection result, and the second detection result represents the flatness of each macroblock and the probability that the macroblock includes a second region of the target object, setting quantization parameters for at least one macroblock in the I-frame according to the detection result can be achieved as follows: determine the target flatness level of the macroblock's flatness among a plurality of preset flatness levels (see the explanation in step 1033 above); determine the target probability level of the probability that the macroblock includes a second region among a plurality of preset probability levels (see the explanation in step 1034 above); query the quantization parameter table based on the target flatness level and the target probability level to obtain the preset quantization parameters associated with the target flatness level and the target probability level, which are used as the quantization parameters of the macroblock. The quantization parameter table includes the mapping relationship between multiple flatness levels, multiple probability levels and multiple preset quantization parameters, and the preset quantization parameters are negatively correlated with the target flatness level and the probability level (see the explanation in step 1035 above).

[0365] In some embodiments, after setting quantization parameters for at least one macroblock in an I-frame based on the detection results, the following processing can be performed for each macroblock: determining the amount of change in the macroblock's quantization parameters relative to a reference quantization coefficient, wherein the reference quantization coefficient is the quantization coefficient of a macroblock in the encoded video frame (corresponding to the image above) that is at the same position as the macroblock (see the description of step 1036 above); determining the product of the change value and the adjustment coefficient, and updating the macroblock's quantization parameters based on the sum of the product and the reference quantization coefficient, wherein the adjustment coefficient is positively correlated with the number of times the second region appears consecutively at the macroblock's position (see the description of step 1037 above).

[0366] In step 504, the first terminal device sends encoded video data to the second terminal device.

[0367] In some embodiments, the first terminal device encapsulates the encoded video data according to a specific protocol (such as Real Time Messaging Protocol (RTMP)) for transmission over the network, and then sends the encapsulated encoded video data to the second terminal device.

[0368] In step 505, the second terminal device receives encoded video data.

[0369] In some embodiments, the second terminal device receives encoded video data via a network and stores the encoded video data in a buffer.

[0370] In step 506, the second terminal device performs video decoding on the encoded video data based on the image decoding method provided in the embodiments of this application to obtain decoded video data.

[0371] In some embodiments, the decoded video data may include multiple decoded video frames, and the video encoded data includes macroblock encoded data for each macroblock of each video frame (see the description in step 104). During decoding, the decoder reads the quantization parameters in each macroblock encoded data, such as macroblock header information (e.g., macroblock type, quantization parameters, etc.) included in the macroblock encoded data. It then uses an inverse quantization process corresponding to the encoder to recover the pixel information of the macroblock. Inverse quantization is the inverse process of quantization; it uses the quantization parameters to recover the original pixel values. Finally, the decoder recombines the recovered pixel values ​​into macroblocks to obtain macroblock decoded data for each macroblock, thereby combining the macroblock decoded data of each macroblock into a decoded video frame.

[0372] For example, an I-frame can be decoded independently without relying on other frames. The decoder reads the quantization parameters from the coded data of each macroblock corresponding to the I-frame, such as macroblock header information (e.g., macroblock type, quantization parameters, etc.). It then uses the inverse quantization process corresponding to the encoder to recover the pixel information of the macroblock. Inverse quantization is the reverse of quantization; it uses the quantization parameters to recover the original pixel values. Finally, the decoder reassembles the recovered pixel values ​​into macroblocks, obtaining the macroblock decoded data for each macroblock, thus combining the macroblock decoded data of each macroblock into a complete I-frame.

[0373] For example, P-frame decoding can be achieved by performing the following processing on each macroblock: extracting the motion vector difference of the corresponding macroblock from the encoded data of the P-frame, the motion vector difference indicating the positional offset between the current macroblock and the corresponding macroblock in the reference frame (the previous I-frame or P-frame); the decoder finding the corresponding macroblock data in the reference frame based on the motion vector difference; using the macroblock data in the reference frame to predict the macroblock of the current P-frame; extracting the prediction error data (corresponding pixel difference) from the encoded data of the P-frame; adding the prediction error data to the predicted macroblock (i.e., summing the pixel difference of the macroblock in the P-frame with the macroblock at the corresponding position in the reference frame); reconstructing the current macroblock in the P-frame; repeating the above steps until all macroblocks have been processed; and then combining all the reconstructed macroblocks into a complete P-frame.

[0374] For example, B-frame decoding can be achieved by performing the following processing on each macroblock: extracting the forward and backward motion vector differences of the corresponding macroblock from the encoded data of the B-frame. The motion vector differences indicate the positional offset between the current macroblock and the corresponding macroblock in the reference frame (the preceding I-frame or P-frame and the following P-frame). The decoder finds the corresponding macroblock data from the preceding I-frame or P-frame and the following P-frame based on the forward and backward motion vector differences, respectively. Using the macroblock data in the reference frame, bidirectional prediction is performed on the macroblock of the current B-frame. Prediction error data (corresponding pixel differences) is extracted from the encoded data of the B-frame. The prediction error data is added to the macroblock of the B-frame to reconstruct the current macroblock in the B-frame. The above steps are repeated until all macroblocks have been processed. Then, all the reconstructed macroblocks are combined into a complete B-frame.

[0375] In step 507, the second terminal device performs image post-processing on the decoded video data to obtain the video data to be displayed.

[0376] In some embodiments, the following processing is performed on each decoded video frame in the decoded video data: a first enhancement processing is performed on the decoded video frame to obtain a first enhanced video frame (see the description of step 304 above); the target grayscale value of each pixel in the first enhanced video frame is obtained (see the description of step 305 above); a second enhancement processing is performed on the first enhanced video frame using the target grayscale value of each pixel to obtain a second enhanced video frame (see the description of step 306 above); and the second enhanced video frames of each decoded video frame are combined in sequence to form video data to be displayed.

[0377] In some embodiments, the saturation of each pixel in the region other than the second region in the second enhanced video frame can be enhanced (for example, the color space of the second enhanced video frame can be converted to the HSL color space, and the value of the S (saturation) channel can be directly adjusted to increase the saturation) to obtain a new second enhanced video frame. By removing the pixels in the second region during saturation enhancement, the problem of reddish skin tone can be effectively avoided. The optimized new second enhanced video frame can be both vivid in color and does not appear "fake".

[0378] In other embodiments, after the first terminal device (broadcaster) acquires video data (corresponding to step 501 above), the following processing can also be performed: in response to the device performance of the first terminal device reaching a preset performance threshold, a second enhancement processing is performed on each video frame in the video data; the first terminal device performs image preprocessing on the video data after the second enhancement processing to obtain video data to be encoded (corresponding to step 502 above); the first terminal device encodes the video data to be encoded to obtain video encoded data and sends it to the second terminal device (viewer), wherein the video encoded data includes a processing marker, which is used to indicate that the first terminal device has performed the second enhancement processing on the video data, and the second terminal device only needs to perform the first enhancement processing; after receiving the video encoded data with the processing marker, the second terminal device performs decoding processing to obtain decoded video data; the second terminal device performs the first enhancement processing on the decoded video data according to the processing marker to obtain video data to be displayed.

[0379] For example, when the broadcaster's device performance reaches a preset performance threshold (e.g., the device's screen resolution, refresh rate, processor model, etc., reach the corresponding threshold), color enhancement processing (second enhancement processing) is enabled on the broadcaster's end. The enhanced video data is then encoded, and processing markers are added to the encoded video frames. For example, supplemental enhancement information (SEI) is used for marking. The meaning of the marker is "the broadcaster has performed second enhancement processing, and the viewer only needs to perform first enhancement processing." When the viewer receives and decodes the encoded video data, they read this processing marker and only need to perform first enhancement processing, without needing to perform second enhancement processing, thus obtaining the video data to be displayed.

[0380] For example, if the performance of the broadcaster's device does not reach the preset performance threshold, a processing mark can be added to the video frame to be encoded: "The viewer needs to perform the first enhancement processing. When the viewer's device rendering performance reaches the preset rendering performance threshold, the second enhancement processing will be performed." Then, the viewer can selectively perform the second enhancement processing based on the rendering performance. For example, when the viewer's rendering frame rate reaches the preset rendering performance threshold (such as 30 frames / second), the second enhancement processing will be enabled. If the rendering frame rate is detected to be insufficient, the second enhancement processing will be disabled.

[0381] For example, referring to Figure 9, Figure 9 is a second flowchart illustrating the application of the image encoding method and image decoding method provided in this application embodiment in a live streaming scenario. After the broadcaster (corresponding to the first terminal device) captures video (corresponding to step 501 above), the video is processed by the image preprocessing module (corresponding to step 502 above). The preprocessed video is then encoded (corresponding to step 503 above) to obtain video encoded data, which is transmitted to the backend server of the live streaming platform or video cloud via the network. The backend server of the live streaming platform or video cloud sends the video encoded data to multiple viewer terminals (corresponding to the second terminal device). After receiving the video encoded data, the viewer terminal performs video buffering, that is, caches the video encoded data in the buffer (corresponding to step 505 above). Next, the video encoded data is decoded to obtain decoded video data (corresponding to step 506 above). Finally, the decoded video data is post-processed by the image postprocessing module to obtain the video data to be displayed (corresponding to step 507 above). The video data to be displayed is then displayed to the viewer terminal through video rendering.

[0382] Through steps 501 to 507, the "video encoding" on the broadcaster's end is optimized. By downsampling (image preprocessing), the encoding efficiency of the encoder is improved after the encoding bitrate is reduced. At the same time, the image encoding method provided in this application embodiment is used to improve the rationality of the encoder's quantization parameter settings. On the viewer's end, "image postprocessing" is added. On the one hand, the clarity is improved through the first enhancement processing (image super-resolution processing). On the other hand, the color enhancement of video frames is achieved through the second enhancement processing and saturation enhancement, which alleviates the loss of color in the video picture after the bitrate is reduced. In summary, through the improvement of encoding technology on the broadcaster's end and the support of image enhancement technology on the viewer's end, a "1+1>2" effect is achieved. For example, a live stream that was originally 1080p and 2Mbps only needs an encoding bitrate of 1Mbps to obtain the same clarity as before, saving nearly 50% of the video bitrate.

[0383] The following continues to describe an exemplary structure of the first image encoding device 433 provided in the embodiments of this application as a software module. In some embodiments, as shown in FIG2A, the software module stored in the first image encoding device 433 in the memory 430 may include:

[0384] The first data acquisition module 4331 is configured to acquire the image to be encoded.

[0385] The first encoding module 4332 is configured to perform at least one of the image detection methods on the image according to the priority of multiple image detection methods, so as to obtain the detection result of the image.

[0386] In some embodiments, the first encoding module 4332 is further configured to set quantization parameters for at least one macroblock in the image based on the detection result.

[0387] In some embodiments, the first encoding module 4332 is further configured to encode each macroblock based on the quantization parameters of each macroblock;

[0388] In some embodiments, the first encoding module 4332 is further configured to generate image encoding data of the image based on the macroblock encoding data obtained after encoding.

[0389] In some embodiments, the multiple image detection methods include a first detection method and a second detection method, wherein the first detection method has a higher priority than the second detection method; the first encoding module 4332 is further configured to perform the first detection method on the image, and if a first region including a target object is detected in the image, generate a first detection result characterizing that the image includes the first region; if the first region is not detected in the image or there is insufficient computing resources to perform the first detection method, perform the second detection method on each macroblock in the image to obtain a second detection result.

[0390] In some embodiments, the first encoding module 4332 is further configured to determine the target color interval corresponding to the second region of the target object based on the proportion of pixels in multiple color intervals in the image; use the proportion of pixels belonging to the target color interval in each macroblock as the probability that the macroblock includes the second region; and generate a second detection result based on the probability that each macroblock includes the second region.

[0391] In some embodiments, the first encoding module 4332 is further configured to, in response to the proportion of pixels in the image belonging to a preset color range being within a preset pixel proportion range, use the preset color range as the target color range to which the pixels of the second region of the target object belong; and in response to the proportion of pixels in the image belonging to the preset color range not being within the preset pixel proportion range, determine a target color range from multiple color ranges of the image, wherein the proportion of pixels in the target color range in the image is within the preset pixel proportion range, and the color difference between the preset color range and the target color range is less than a color difference threshold.

[0392] In some embodiments, the first encoding module 4332 is further configured to update the preset color range to the target color range, wherein the updated preset color range is used to perform the second detection method on the new image to be encoded.

[0393] In some embodiments, the first encoding module 4332 is further configured to determine the number of pixels in the image belonging to each of the plurality of color intervals, wherein the plurality of color intervals are obtained by dividing a color space; and for each color interval, the ratio of the number of pixels in the image belonging to the color interval to the total number of pixels in the image is used as the proportion of pixels in the color interval in the image.

[0394] In some embodiments, the first encoding module 4332 is further configured such that when the target object is a living organism imaged in the image, the first region is the facial region of the living organism and the second region is the skin region of the living organism; when the target object is an object imaged in the image, the first region is the main body region of the object and the second region is a local region of the object.

[0395] In some embodiments, the first encoding module 4332 is further configured to: calculate the flatness of each macroblock in the image; calculate the proportion of pixels belonging to the target color range in each macroblock in the image, and use the proportion as the probability that the macroblock includes the second region of the target object, wherein the target color range is the color value range of the second region of the target object; and generate a second detection result based on the flatness and the probability of each macroblock.

[0396] In some embodiments, the first encoding module 4332 is further configured to calculate the flatness of each macroblock in the image and generate a second detection result based on the flatness of each macroblock.

[0397] In some embodiments, the first encoding module 4332 is further configured to perform the following processing for each macroblock: grayscale the macroblock to obtain the grayscale value of each pixel in the macroblock; determine the grayscale mean of the macroblock based on the grayscale value of each pixel; and determine the grayscale variance of the macroblock based on the grayscale mean, as the flatness of the macroblock.

[0398] In some embodiments, the first encoding module 4332 is further configured to call a pre-trained detection model to perform keypoint detection on the image and obtain keypoint detection results, wherein the keypoint detection results include bounding boxes and keypoints; in response to the number of keypoints being a preset number of keypoints, the region corresponding to the bounding box is taken as the first region of the target object, and a first detection result is generated in which the image includes the first region, wherein the preset number of keypoints is used to determine the minimum number of the image including the first region.

[0399] In some embodiments, where the detection result is a first detection result and the first detection result characterizes the image as including a first region of a target object, the first encoding module 4332 is further configured to determine macroblocks that at least partially overlap with the first region; set a first quantization parameter for macroblocks that intersect with the first region, and set a second quantization parameter for macroblocks that do not intersect with the first region, wherein the first quantization parameter is smaller than the second quantization parameter.

[0400] In some embodiments, where the detection result is a second detection result, and the second detection result characterizes the flatness of each macroblock and the probability that the macroblock includes a second region of the target object, the first encoding module 4332 is further configured to perform the following processing for each macroblock: determining the target flatness level of the macroblock among a preset plurality of flatness levels; determining the target probability level of the probability that the macroblock includes the second region among a preset plurality of probability levels; querying a quantization parameter table based on the target flatness level and the target probability level to obtain a preset quantization parameter associated with the target flatness level and the target probability level, as a quantization parameter for the macroblock, wherein the quantization parameter table includes a mapping relationship between the plurality of flatness levels, the plurality of probability levels and the plurality of preset quantization parameters, and the preset quantization parameter is negatively correlated with the target flatness level and the probability level.

[0401] In some embodiments, where the detection result is a second detection result, and the second detection result characterizes the probability that each macroblock includes a second region of the target object, the first encoding module 4332 is further configured to perform the following processing for each macroblock: generating a quantization parameter of the macroblock based on the probability that the macroblock includes the second region of the target object, wherein the probability that the macroblock includes the second region is negatively correlated with the quantization parameter of the macroblock.

[0402] In some embodiments, the first encoding module 4332 is further configured to perform the following processing for each macroblock: determining the amount of change of the quantization parameter of the macroblock relative to a reference quantization coefficient, wherein the reference quantization coefficient is the quantization coefficient of a macroblock in the encoded image that is at the same position as the macroblock; determining the product of the change value and an adjustment coefficient; and updating the quantization parameter of the macroblock based on the sum of the product and the reference quantization coefficient, wherein the adjustment coefficient is positively correlated with the number of times the second region appears consecutively at the position of the macroblock.

[0403] In some embodiments, where the detection result is a second detection result and the second detection result characterizes the flatness of each macroblock, the first encoding module 4332 is further configured to perform the following processing for each macroblock: generating quantization parameters of the macroblock based on the flatness of the macroblock, wherein the flatness of the macroblock is negatively correlated with the quantization parameters of the macroblock.

[0404] In some embodiments, the first data acquisition module 4331 is further configured to acquire an uncompressed raw image; use the raw image as the image to be encoded, or to downsample the raw image and use the resulting downsampled image as the image to be encoded.

[0405] The following continues to describe the exemplary structure of the second image encoding device 533 provided in the embodiments of this application as a software module. In some embodiments, as shown in FIG2B, the software module stored in the second image encoding device 533 in the memory 530 may include:

[0406] The second data acquisition module 5331 is configured to acquire the image to be encoded.

[0407] The second encoding module 5332 is configured to determine the target color range corresponding to the target region of the target object based on the proportion of pixels in multiple color ranges in the image.

[0408] In some embodiments, the second encoding module 5332 is further configured to use the proportion of pixels belonging to the target color range in each macroblock of the image as the probability that each macroblock includes the target region of the target object.

[0409] In some embodiments, the second encoding module 5332 is further configured to set quantization parameters for at least one macroblock based on the probability that each macroblock includes the target region.

[0410] In some embodiments, the second encoding module 5332 is further configured to encode each macroblock based on the quantization parameters of each macroblock;

[0411] In some embodiments, the second encoding module 5332 is further configured to generate image encoding data of the image based on the macroblock encoding data obtained after encoding.

[0412] In some embodiments, the second encoding module 5332 is further configured to: in response to the proportion of pixels in the image belonging to a preset color range being within a preset pixel proportion range, use the preset color range as the target color range to which the target region of the target object in the image belongs; in response to the proportion of pixels in the image belonging to the preset color range not being within the preset pixel proportion range, determine a target color range from multiple color ranges in the image, wherein the proportion of pixels in the target color range in the image is within the preset pixel proportion range, and the color difference between the preset color range and the target color range is less than a color difference threshold.

[0413] In some embodiments, the second encoding module 5332 is further configured to perform the following processing for each macroblock: generating quantization parameters of the macroblock based on the probability that the macroblock includes the target region, wherein the probability that the macroblock includes the target region is negatively correlated with the quantization parameters of the macroblock.

[0414] In some embodiments, the second encoding module 5332 is further configured to perform the following processing for each macroblock: determining the target flatness level of the macroblock's flatness among a plurality of preset flatness levels; determining the target probability level of the probability that the macroblock includes the target region among a plurality of preset probability levels; querying a quantization parameter table based on the target flatness level and the target probability level to obtain preset quantization parameters associated with the target flatness level and the target probability level, as quantization parameters for the macroblock, wherein the quantization parameter table includes a mapping relationship between the plurality of flatness levels, the plurality of probability levels and the plurality of preset quantization parameters, and the preset quantization parameters are negatively correlated with the target flatness level and the probability.

[0415] In some embodiments, the second encoding module 5332 is further configured to perform the following processing for each macroblock: determining the amount of change of the quantization parameter of the macroblock relative to a reference quantization coefficient, wherein the reference quantization coefficient is the quantization coefficient of a macroblock in the encoded image that is at the same position as the macroblock; determining the product of the change value and an adjustment coefficient; and updating the quantization parameter of the macroblock based on the sum of the product and the reference quantization coefficient, wherein the adjustment coefficient is positively correlated with the number of times the second region appears consecutively at the position of the macroblock.

[0416] The following continues to describe an exemplary structure of the first image decoding device 633 provided in the embodiments of this application as a software module. In some embodiments, as shown in FIG2C, the software module stored in the first image decoding device 633 in the memory 630 may include:

[0417] The first data acquisition module 6331 is configured to receive image encoded data, wherein the image encoded data is obtained through the image encoding method provided in the embodiments of this application.

[0418] The first decoding module 6332 is configured to decode the macroblock encoded data based on the quantization parameters of each macroblock to obtain macroblock decoded data for each macroblock.

[0419] In some embodiments, the first decoding module 6332 is further configured to combine the macroblock decoding data of each macroblock into a decoded image.

[0420] In some embodiments, the first decoding module 6332 is further configured to perform a first enhancement process on the decoded image to obtain a first enhanced image; obtain the target grayscale value of each pixel in the first enhanced image; and perform a second enhancement process on the first enhanced image using the target grayscale value of each pixel to obtain a second enhanced image.

[0421] In some embodiments, the first decoding module 6332 is further configured to: determine the grayscale value of each pixel in the first enhanced image; for each grayscale value, determine the number of pixels in the first enhanced image that have the grayscale value, and construct an original grayscale histogram based on the number; perform grayscale value conversion based on the original grayscale histogram to obtain a target grayscale histogram; and determine the target grayscale value of each pixel in the first enhanced image according to the target grayscale histogram.

[0422] In some embodiments, the first decoding module 6332 is further configured to update the grayscale value of each pixel in the first enhanced image to the target grayscale value corresponding to the pixel, thereby obtaining a second enhanced image.

[0423] In some embodiments, the first decoding module 6332 is further configured to perform feature extraction processing on the decoded image to obtain image features; and perform image reconstruction processing based on the image features to obtain a first enhanced image.

[0424] The following continues to describe an exemplary structure of the second image decoding device 733 provided in the embodiments of this application as a software module. In some embodiments, as shown in FIG2D, the software module stored in the second image decoding device 733 in the memory 730 may include:

[0425] The second data acquisition module 7331 is configured to receive image encoded data, wherein the image encoded data is obtained through the image encoding method provided in the embodiments of this application.

[0426] The second decoding module 7332 is configured to decode the macroblock encoded data based on the quantization parameters of each macroblock to obtain macroblock decoded data for each macroblock.

[0427] In some embodiments, the second decoding module 7332 is further configured to combine the macroblock decoding data of each macroblock into a decoded image.

[0428] This application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the image encoding and image decoding methods described in this application.

[0429] This application provides a computer-readable storage medium storing computer-executable instructions or computer programs. When the computer-executable instructions or computer programs are executed by a processor, the processor will execute the image encoding method and image decoding method provided in this application. For example, the image encoding method is shown in FIG3A, the image encoding method is shown in FIG4A, the image decoding method is shown in FIG5A, and the image decoding method is shown in FIG6.

[0430] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0431] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0432] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0433] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0434] This application provides a method for storing a video stream, which is used to generate a video stream by performing an image encoding method provided in this application; and to store the video stream.

[0435] This application provides a method for transmitting a video stream, which is used to generate a video stream by performing an image encoding method provided in this application; and to transmit the video stream.

[0436] This application provides a computer-readable storage medium storing a computer program / instructions and a video stream thereon. The computer program / instructions are executed by a processor to implement the steps of the image encoding method provided in this application to generate the video stream.

[0437] Some implementations may involve systems, methods, and / or computer-readable media at any possible level of integration technical detail. The computer-readable medium may include a computer-readable non-transitory storage medium (or multiple media) having computer-readable program instructions on it for causing a processor to perform operations, and may also include storage of a video stream generated according to the image encoding method described above. When executed by a processor, the computer program / instructions may implement the steps of the image encoding method to generate the video stream, or implement the steps of the image decoding method to decode the video stream.

[0438] In summary, through the embodiments of this application, images are detected according to the priority of multiple image detection methods. Based on the detection results of at least one image detection method, more suitable quantization parameters are set for macroblocks in the image. Compared with related technologies that use a single quantization parameter calculation scheme, the image detection methods used are more diverse, and the detection by priority can ensure that the detection results of the more suitable detection method are prioritized to set more suitable quantization parameters for each macroblock. This achieves more refined encoding control and higher quality images under limited bitrate.

[0439] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. An image encoding method applied to an electronic device, the method comprising: Obtain the image to be encoded; Based on the priority of multiple image detection methods, at least one of the image detection methods is performed on the image to obtain the detection result of the image; Based on the detection results, quantization parameters are set for at least one macroblock in the image; Each macroblock is encoded based on the quantization parameters of each macroblock; Image encoded data of the image is generated based on the macroblock encoded data obtained after encoding.

2. The method according to claim 1, wherein, The multiple image detection methods include a first detection method and a second detection method, wherein the first detection method has a higher priority than the second detection method; The step of performing at least one of the image detection methods on the image according to the priority of multiple image detection methods to obtain the detection result of the image includes: The first detection method is performed on the image, and if a first region including a target object is detected from the image, a first detection result characterizing that the image includes the first region is generated; If the first region is not detected in the image or there are insufficient computing resources to perform the first detection method, the second detection method is performed on each macroblock in the image to obtain a second detection result.

3. The method according to any one of claims 1 to 2, wherein performing the second detection method on each macroblock in the image to obtain a second detection result includes: Based on the proportion of pixels in multiple color intervals in the image, the target color interval corresponding to the second region of the target object is determined; The proportion of pixels belonging to the target color range in each macroblock is used as the probability that the macroblock includes the second region; A second detection result is generated based on the probability that each macroblock includes the second region.

4. The method according to any one of claims 1 to 3, wherein, Determining the target color range corresponding to the second region of the target object based on the proportion of pixels in multiple color ranges in the image includes: In response to the fact that the proportion of pixels in the image belonging to a preset color range is within a preset pixel proportion range, the preset color range is used as the target color range to which the pixels of the second region of the target object belong; In response to the fact that the proportion of pixels in the image belonging to a preset color range is not within the preset pixel proportion range, a target color range is determined from multiple color ranges of the image, wherein the proportion of pixels in the target color range in the image is within the preset pixel proportion range, and the color difference between the preset color range and the target color range is less than a color difference threshold.

5. The method according to any one of claims 1 to 4, wherein, After determining the target color range from multiple color ranges of the image, the method further includes: The preset color range is updated to the target color range, wherein the updated preset color range is used to perform the second detection method on the new image to be encoded.

6. The method according to any one of claims 1 to 5, wherein, Before performing the second detection method on each macroblock in the image to obtain the second detection result, the method further includes: For the multiple color intervals, determine the number of pixels in the image belonging to each color interval, wherein the multiple color intervals are obtained by dividing the color space; For each color range, the ratio of the number of pixels belonging to that color range to the total number of pixels in the image is taken as the proportion of pixels belonging to that color range in the image.

7. The method according to any one of claims 1 to 6, wherein, When the target object is a living organism imaged in the image, the first region is the facial region of the living organism, and the second region is the skin region of the living organism; When the target object is an object imaged in the image, the first region is the main body region of the object, and the second region is a local region of the object.

8. The method according to any one of claims 1 to 7, wherein, The step of performing the second detection method on each macroblock in the image to obtain a second detection result includes: The flatness of each macroblock in the image is calculated. The proportion of pixels belonging to the target color range in each macroblock of the image is statistically analyzed, and the proportion is used as the probability that the macroblock includes the second region of the target object, wherein the target color range is the color value range of the second region of the target object; A second detection result is generated based on the flatness and probability of each macroblock.

9. The method according to any one of claims 1 to 8, wherein the plurality of image detection methods includes a first detection method and a second detection method, wherein the first detection method has a higher priority than the second detection method, and wherein, The step of performing the second detection method on each macroblock in the image to obtain a second detection result includes: The flatness of each macroblock in the image is calculated, and a second detection result is generated based on the flatness of each macroblock.

10. The method according to any one of claims 1 to 9, wherein, The method of calculating the flatness of each macroblock in the image includes: For each of the macroblocks, the following processing is performed: The macroblock is converted to grayscale to obtain the grayscale value of each pixel in the macroblock; The average gray value of the macroblock is determined based on the gray value of each pixel. The gray-level variance of the macroblock is determined based on the gray-level mean, and is used as the flatness of the macroblock.

11. The method according to any one of claims 1 to 10, wherein, Performing the first detection method on the image includes: A pre-trained detection model is invoked to perform keypoint detection on the image, and keypoint detection results are obtained, wherein the keypoint detection results include bounding boxes and keypoints; In response to the number of key points being a preset number of key points, the region corresponding to the bounding box is taken as the first region of the target object, and a first detection result is generated in which the image includes the first region, wherein the preset number of key points is used to determine the minimum number of the image including the first region.

12. The method according to any one of claims 1 to 11, wherein, When the detection result is a first detection result, and the first detection result characterizes the image as including a first region of the target object, the step of setting quantization parameters for at least one macroblock in the image based on the detection result includes: Identify macroblocks that at least partially overlap with the first region; A first quantization parameter is set for macroblocks that intersect with the first region, and a second quantization parameter is set for macroblocks that do not intersect with the first region, wherein the first quantization parameter is smaller than the second quantization parameter.

13. The method according to any one of claims 1 to 12, wherein, When the detection result is a second detection result, and the second detection result characterizes the flatness of each macroblock and the probability that the macroblock includes a second region of the target object, the step of setting quantization parameters for at least one macroblock in the image based on the detection result includes: For each of the macroblocks, the following processing is performed: Determine the target flatness level of the macroblock among a set of preset flatness levels; Determine the target probability level among a preset plurality of probability levels in which the probability that the macroblock includes the second region; Based on the target flatness level and the target probability level, a quantization parameter table is queried to obtain a preset quantization parameter associated with the target flatness level and the target probability level, which is used as the quantization parameter of the macroblock. The quantization parameter table includes the mapping relationship between the plurality of flatness levels, the plurality of probability levels and the plurality of preset quantization parameters, and the preset quantization parameter is negatively correlated with the target flatness level and the probability level.

14. The method according to any one of claims 1 to 13, wherein, When the detection result is a second detection result, and the second detection result represents the probability that each macroblock includes a second region of the target object, the step of setting quantization parameters for at least one macroblock in the image based on the detection result includes: For each of the macroblocks, the following processing is performed: The quantization parameters of the macroblock are generated based on the probability that the macroblock includes the second region of the target object, wherein the probability that the macroblock includes the second region is negatively correlated with the quantization parameters of the macroblock.

15. The method according to any one of claims 1 to 14, wherein, When the detection result is a second detection result, and the second detection result characterizes the flatness of each macroblock, setting quantization parameters for at least one macroblock in the image based on the detection result includes: For each of the macroblocks, the following processing is performed: The quantization parameters of the macroblock are generated based on the flatness of the macroblock, wherein the flatness of the macroblock is negatively correlated with the quantization parameters of the macroblock.

16. The method according to any one of claims 1 to 15, wherein, After setting quantization parameters for at least one macroblock in the image based on the detection results, the method further includes: For each of the macroblocks, the following processing is performed: Determine the amount of change in the quantization parameter of the macroblock relative to a reference quantization coefficient, wherein the reference quantization coefficient is the quantization coefficient of a macroblock in the encoded image that is at the same position as the macroblock; The product of the change value and the adjustment coefficient is determined, and the quantization parameters of the macroblock are updated based on the sum of the product and the reference quantization coefficient, wherein the adjustment coefficient is positively correlated with the number of times the second region appears consecutively at the position of the macroblock.

17. The method according to any one of claims 1 to 16, wherein, The process of acquiring the image to be encoded includes: Obtain the uncompressed raw image; The original image can be used as the image to be encoded, or the original image can be downsampled and the resulting downsampled image can be used as the image to be encoded.

18. An image encoding method applied to an electronic device, the method comprising: Obtain the image to be encoded; Based on the proportion of pixels in multiple color intervals in the image, the target color interval corresponding to the target area of ​​the target object is determined. The proportion of pixels belonging to the target color range in each macroblock of the image is used as the probability that each macroblock includes the target region of the target object; Based on the probability that each macroblock includes the target region, quantization parameters are set for at least one macroblock; Each macroblock is encoded based on the quantization parameters of each macroblock; Image encoded data of the image is generated based on the macroblock encoded data obtained after encoding.

19. The method according to claim 18, wherein, Determining the target color range corresponding to the target region of the target object based on the proportion of pixels in multiple color ranges in the image includes: In response to the fact that the proportion of pixels in the image belonging to a preset color range is within a preset pixel proportion range, the preset color range is taken as the target color range to which the target region of the target object in the image belongs. In response to the fact that the proportion of pixels in the image belonging to a preset color range is not within the preset pixel proportion range, a target color range is determined from multiple color ranges of the image, wherein the proportion of pixels in the target color range in the image is within the preset pixel proportion range, and the color difference between the preset color range and the target color range is less than a color difference threshold.

20. The method according to any one of claims 18 to 19, wherein, The step of setting quantization parameters for at least one macroblock based on the probability that each macroblock includes the target region includes: For each of the macroblocks, the following processing is performed: The quantization parameters of the macroblock are generated based on the probability that the macroblock includes the target region, wherein the probability that the macroblock includes the target region is negatively correlated with the quantization parameters of the macroblock.

21. The method according to any one of claims 18 to 20, wherein, The step of setting quantization parameters for at least one macroblock based on the probability that each macroblock includes the target region includes: For each of the macroblocks, the following processing is performed: Determine the target flatness level of the macroblock among a set of preset flatness levels; Determine the target probability level among a preset plurality of probability levels in which the probability that the macroblock includes the target region is located; Based on the target flatness level and the target probability level, a quantization parameter table is queried to obtain a preset quantization parameter associated with the target flatness level and the target probability level, which is used as the quantization parameter of the macroblock. The quantization parameter table includes the mapping relationship between the plurality of flatness levels, the plurality of probability levels and the plurality of preset quantization parameters, and the preset quantization parameter is negatively correlated with the target flatness level and the probability.

22. The method according to any one of claims 18 to 21, wherein, After setting quantization parameters for at least one of the macroblocks, the method further includes: For each of the macroblocks, the following processing is performed: Determine the amount of change in the quantization parameter of the macroblock relative to a reference quantization coefficient, wherein the reference quantization coefficient is the quantization coefficient of a macroblock in the encoded image that is at the same position as the macroblock; The product of the change value and the adjustment coefficient is determined, and the quantization parameters of the macroblock are updated based on the sum of the product and the reference quantization coefficient, wherein the adjustment coefficient is positively correlated with the number of times the second region appears consecutively at the position of the macroblock.

23. An image decoding method applied to an electronic device, the method comprising: Receive image encoded data, wherein the image encoded data is obtained by the image encoding method according to any one of claims 1 to 17; The macroblock encoded data is decoded based on the quantization parameters of each macroblock to obtain the macroblock decoded data of each macroblock; The macroblock decoding data of each macroblock is combined into a decoded image.

24. The method according to claim 23, wherein, After obtaining the decoded image, the method further includes: The decoded image is subjected to a first enhancement process to obtain a first enhanced image; Obtain the target grayscale value of each pixel in the first enhanced image; The first enhanced image is subjected to a second enhancement process based on the target grayscale value of each pixel to obtain a second enhanced image.

25. The method according to any one of claims 23 to 24, wherein, The step of obtaining the target grayscale value of each pixel in the first enhanced image includes: Determine the grayscale value of each pixel in the first enhanced image; For each gray value, determine the number of pixels in the first enhanced image that have the gray value, and construct an original gray-level histogram based on the number. Based on the original grayscale histogram, grayscale values ​​are converted to obtain the target grayscale histogram; Based on the target grayscale histogram, the target grayscale value of each pixel in the first enhanced image is determined.

26. The method according to any one of claims 23 to 25, wherein, The step of performing a second enhancement process on the first enhanced image using the target grayscale value of each pixel to obtain a second enhanced image includes: The grayscale value of each pixel in the first enhanced image is updated to the target grayscale value corresponding to the pixel to obtain the second enhanced image; The first enhancement process on the decoded image to obtain the first enhanced image includes: The decoded image is subjected to feature extraction processing to obtain image features; Based on the image features, image reconstruction processing is performed to obtain the first enhanced image.

27. An image decoding method applied to an electronic device, the method comprising: Receive image encoded data, wherein the image encoded data is obtained by the image encoding method according to any one of claims 18 to 22; The macroblock encoded data is decoded based on the quantization parameters of each macroblock to obtain the macroblock decoded data of each macroblock; The macroblock decoding data of each macroblock is combined into a decoded image.

28. An image encoding apparatus, comprising: The data acquisition module is configured to acquire the image to be encoded. The encoding module is configured to perform at least one of the image detection methods on the image according to the priority of multiple image detection methods, so as to obtain the detection result of the image; The encoding module is further configured to set quantization parameters for at least one macroblock in the image based on the detection results; The encoding module is further configured to encode each macroblock based on the quantization parameters of each macroblock; The encoding module is further configured to generate image encoding data of the image based on the macroblock encoding data obtained after encoding.

29. An image encoding apparatus, comprising: The data acquisition module is configured to acquire the image to be encoded. The encoding module is configured to determine the target color range corresponding to the target region of the target object based on the proportion of pixels in multiple color ranges in the image. The encoding module is further configured to use the proportion of pixels belonging to the target color range in each macroblock of the image as the probability that each macroblock includes the target region of the target object; The encoding module is further configured to set quantization parameters for at least one macroblock based on the probability that each macroblock includes the target region. The encoding module is further configured to encode each macroblock based on the quantization parameters of each macroblock; The encoding module is further configured to generate image encoding data of the image based on the macroblock encoding data obtained after encoding.

30. An image decoding apparatus, comprising: A data acquisition module is configured to receive image encoded data, wherein the image encoded data is obtained by the image encoding method according to any one of claims 1 to 17; The decoding module is configured to decode the macroblock encoded data based on the quantization parameters of each macroblock to obtain macroblock decoded data for each macroblock. The decoding module is further configured to combine the macroblock decoding data of each macroblock into a decoded image.

31. An image decoding apparatus, comprising: A data acquisition module is configured to receive image encoded data, wherein the image encoded data is obtained by the image encoding method according to any one of claims 18 to 22; The decoding module is configured to decode the encoded data of each macroblock based on the quantization parameters of each macroblock to obtain the macroblock decoded data of each macroblock. The decoding module is further configured to combine the macroblock decoding data of each macroblock into a decoded image.

32. An electronic device, comprising: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the image encoding method according to any one of claims 1 to 17, or implements the image decoding method according to any one of claims 18 to 22.

33. An electronic device, comprising: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the image decoding method according to any one of claims 23 to 26 or 27.

34. A computer-readable storage medium storing computer-executable instructions or a computer program, wherein when executed by a processor, the computer-executable instructions or the computer program implement the image encoding method according to any one of claims 1 to 17 or claims 18 to 22, or implement the image decoding method according to any one of claims 23 to 26 or claim 27.

35. A computer program product comprising computer-executable instructions or a computer program, wherein when executed by a processor, the computer-executable instructions or the computer program implement the image encoding method according to any one of claims 1 to 17 or 18 to 22, or implement the image decoding method according to any one of claims 23 to 26 or 27.

36. A method for storing a video stream, comprising performing an image encoding method according to any one of claims 1 to 17 or 18 to 22 to generate a video stream; and storing the video stream.

37. A method for transmitting a video stream, comprising performing an image encoding method according to any one of claims 1 to 17 or 18 to 22 to generate a video stream; and transmitting the video stream.

38. A computer-readable storage medium having stored thereon a computer program / instructions and a video stream, the computer program / instructions being executed by a processor to implement the steps of the image encoding method according to any one of claims 1 to 17 or 18 to 22 to generate the video stream.

Images