Encoding method, signal processing method, and related device

Abstract

A signal processing method provided in this application includes: obtaining a first baseline image, a first gain map, and metadata, where the first baseline image corresponds to a first dynamic range; processing the first basline image based on the metadata to obtain a second baseline image; and obtaining a target image based on the second baseline image and the first gain map, where the target image corresponds to second dynamic range, and the second dynamic range is differnt from the first dynamic range. The solutions provided in this application are compatible with image encoding and processing in a plurality of formats, and comparatively good image quality can be achieved on the systems with different support capabilities.

Description

Coding method, signal processing method and related equipment

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application No. 202311032268.9 filed on August 15, 2023, Chinese patent application No. 202311158678.8 filed on September 6, 2023, and Chinese patent application No. 202311294767.5 filed on September 28, 2023, the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The present application relates to the field of coding and decoding, and more specifically to a coding method, a signal processing method and related devices. Background Art

[0004] Dynamic range is used in many fields to express the ratio of the maximum value to the minimum value of a variable. In digital images, dynamic range represents the ratio between the maximum grayscale value and the minimum grayscale value within the displayable range of the image. The dynamic range in nature is quite large. The brightness of the night scene under the starry sky is about 0.001cd / m2, and the brightness of the sun itself is as high as 1,000,000,000cd / m2. Such a dynamic range reaches the order of 1,000,000,000 / 0.001=1013. However, in real scenes in nature, the brightness of the sun and the brightness of starlight will not be obtained at the same time. For natural scenes in the real world, the dynamic range is between 10 -3 to 10 6 In most color digital images, each channel of red (R), green (G), and blue (B) is stored in an 8-bit byte. That is, the range of each channel is 0 to 255 gray levels. Here, 0 to 255 is the dynamic range of the image. In the real world, the dynamic range of the same scene is between 10 -3 to 10 6 The dynamic range of an ordinary picture or video is called low dynamic range (LDR).

[0005] Summary of the Invention

[0006] In existing operating systems such as Android, the decoding module of the terminal device (calling hardware or software decoding), the graphics processing module (calling GPU or software for image processing), and the display module (calling display hardware) are usually independent modules. There is usually only one image buffer between these three modules, and the accompanying metadata is often ignored and not passed, resulting in the inability to correctly process HDR data and achieve the desired display effect. The present application provides a coding method, a signal processing method, and related equipment to solve the above problems. The subject matter of protection of this application is defined by the claims.

[0007] In a first aspect, the present application provides a signal processing method, comprising: acquiring a first baseline image, a first gain map, and metadata, wherein the first baseline image corresponds to a first dynamic range; processing the first baseline image according to the metadata to obtain a second baseline image; and obtaining a target image according to the second baseline image and the first gain map, wherein the target image corresponds to a second dynamic range, and the second dynamic range is different from the first dynamic range.

[0008] By using a dual-layer code stream, this application can merge the images obtained from the two image buffers into one image after decoding for systems that can handle dual-layer code streams. In this way, the decoded image can be correctly transmitted to the display module using the existing interface. When displayed, tone mapping is performed based on the display-related information transmitted by the metadata, so that the correct HDR display effect can be obtained on the terminal. For systems that cannot use dual-layer code streams, only the basic image can be used, that is, the decoded basic image is transmitted to the display module through the existing interface, so that the correct basic display effect can be obtained on the terminal, thereby achieving compatibility with formats supported by different systems.

[0009] In a possible implementation of the first aspect, processing the first base image based on the metadata includes: upsampling or downsampling the first base image based on the metadata to obtain the second base image, where the resolution of the second base image is the first target resolution. By using upsampling or downsampling, compatibility with image encoding and processing in various formats is achieved, relatively good image quality can be achieved on systems with different support capabilities, and file size can be significantly reduced.

[0010] In a possible implementation manner of the first aspect, the first target resolution is a resolution of the first gain map.

[0011] In a possible implementation of the first aspect, the first target resolution is a first preset resolution. The preset resolution may be included in metadata, or the preset resolution is related to display characteristics of a display device, but is determined for a specific display device.

[0012] In a possible implementation manner of the first aspect, the metadata includes first sampling phase information, and upsampling or downsampling the first basic image according to the metadata includes: upsampling or downsampling the first basic image according to the first sampling phase information.

[0013] In a possible implementation manner of the first aspect, the first sampling phase information is used to indicate a relative positional relationship between pixels of the first base image and the second base image.

[0014] In a possible implementation manner of the first aspect, the first sampling phase information includes pixel coordinates of a first preset position of the first basic image, and the first sampling phase information also includes pixel coordinates of a second preset position of the second basic image.

[0015] In a possible implementation manner of the first aspect, the first sampling phase information includes an offset between pixel coordinates of the second basic image and pixel coordinates of the first basic image.

[0016] In a possible implementation of the first aspect, the first sampling phase information includes first luminance sampling phase information, and upsampling or downsampling the first basic image according to the first sampling phase information includes: upsampling or downsampling the luminance component of the first basic image according to the first luminance sampling phase information; or, the first sampling phase information includes first chrominance sampling phase information, and upsampling or downsampling the first basic image according to the first sampling phase information includes: upsampling or downsampling the chrominance component of the first basic image according to the first chrominance sampling phase information; or, the first sampling phase information includes first luminance sampling phase information and first chrominance sampling phase information, and upsampling or downsampling the first basic image according to the first sampling phase information includes: upsampling or downsampling the luminance component of the first basic image according to the first luminance sampling phase information, and upsampling or downsampling the chrominance component of the first basic image according to the first chrominance sampling phase information.

[0017] In a possible implementation of the first aspect, the metadata includes indication information of a first target sampling algorithm, and upsampling or downsampling the first basic image according to the metadata includes: upsampling or downsampling the first basic image using the first target sampling algorithm according to the indication information of the first target sampling algorithm.

[0018] In a possible implementation manner of the first aspect, the indication information of the first target sampling algorithm includes an index value used to indicate determination of the first target sampling algorithm from multiple preset sampling algorithms.

[0019] In a possible implementation manner of the first aspect, the indication information of the first target sampling algorithm includes a name or other specific identifier of the first target sampling algorithm.

[0020] In a possible implementation of the first aspect, when the metadata includes first sampling phase information and indication information of the first target sampling algorithm, upsampling or downsampling the first basic image according to the metadata includes: upsampling or downsampling the first basic image using the first target sampling algorithm according to the first sampling phase information and the indication information of the first target sampling algorithm.

[0021] In a possible implementation of the first aspect, obtaining a target image based on the second base image and the first gain map includes: processing the first gain map according to the metadata to obtain a second gain map; and obtaining the target image based on the second base image and the second gain map.

[0022] In a possible implementation of the first aspect, processing the first gain map according to the metadata includes: upsampling or downsampling the first gain map according to the metadata to obtain the second gain map, where a resolution of the second gain map is a second target resolution.

[0023] In a possible implementation manner of the first aspect, the second target resolution is a resolution of the first basic image.

[0024] In a possible implementation manner of the first aspect, the second target resolution is a second preset resolution.

[0025] In a possible implementation of the first aspect, the metadata includes second sampling phase information, and upsampling or downsampling the first gain map according to the metadata includes: upsampling or downsampling the first gain map according to the second sampling phase information.

[0026] In a possible implementation manner of the first aspect, the second sampling phase information is used to indicate a relative positional relationship between pixels of the first gain map and the second gain map.

[0027] In a possible implementation manner of the first aspect, the second sampling phase information includes pixel coordinates of a first preset position of the first gain map, and the second sampling phase information further includes pixel coordinates of a second preset position of the second gain map.

[0028] In a possible implementation manner of the first aspect, the second sampling phase information includes an offset between pixel coordinates of the second gain map and pixel coordinates of the first gain map.

[0029] In a possible implementation of the first aspect, the second sampling phase information includes second luminance sampling phase information, and upsampling or downsampling the first gain map according to the second sampling phase information includes: upsampling or downsampling the luminance component of the first gain map according to the second luminance sampling phase information; or, the second sampling phase information includes second chrominance sampling phase information, and upsampling or downsampling the first gain map according to the second sampling phase information includes: upsampling or downsampling the chrominance component of the first gain map according to the second chrominance sampling phase information; or, the second sampling phase information includes second luminance sampling phase information and second chrominance sampling phase information, and upsampling or downsampling the first gain map according to the second sampling phase information includes: upsampling or downsampling the luminance component of the first gain map according to the second luminance sampling phase information, and upsampling or downsampling the chrominance component of the first gain map according to the second chrominance sampling phase information.

[0030] In a possible implementation of the first aspect, the metadata includes indication information of a second target sampling algorithm, and upsampling or downsampling the first gain map according to the metadata includes: upsampling or downsampling the first gain map using the second target sampling algorithm according to the indication information of the second target sampling algorithm.

[0031] In a possible implementation manner of the first aspect, the indication information of the second target sampling algorithm includes an index value used to indicate determination of the second target sampling algorithm from a plurality of preset sampling algorithms.

[0032] In a possible implementation manner of the first aspect, the indication information of the second target sampling algorithm includes a name or other specific identifier of the second target sampling algorithm.

[0033] In a possible implementation of the first aspect, when the metadata includes second sampling phase information and indication information of the second target sampling algorithm, upsampling or downsampling the first gain map according to the metadata includes: upsampling or downsampling the first gain map using the second target sampling algorithm according to the second sampling phase information and the indication information of the second target sampling algorithm.

[0034] In a possible implementation of the first aspect, the first dynamic range is a standard dynamic range SDR, and the second dynamic range is a high dynamic range HDR.

[0035] In a possible implementation of the first aspect, the first dynamic range is a high dynamic range (HDR), and the second dynamic range is a standard dynamic range (SDR).

[0036] In a possible implementation of the first aspect, the second basic image corresponds to a first transfer function, the first gain map corresponds to a second transfer function, the target image corresponds to a third transfer function, and the first transfer function, the second transfer function, and the third transfer function are photoelectric transfer functions or electro-optical transfer functions.

[0037] In a possible implementation of the first aspect, the first transfer function, the second transfer function, and the third transfer function are all one of a linear function, a log function, a hybrid log-gamma HLG function, and a perceptual quantization PQ function; the first transfer function, the second transfer function, and the third transfer function are different from each other.

[0038] In a possible implementation of the first aspect, obtaining the target image based on the second basis image and the first gain map includes: converting the second basis image, wherein the converted second basis image corresponds to the second transfer function; and obtaining the target image based on the converted second basis image and the first gain map.

[0039] In a possible implementation of the first aspect, obtaining the target image based on the second basic image and the first gain map includes: converting the first gain map, the converted first gain map corresponding to the first transfer function; and obtaining the target image based on the converted first gain map and the second basic image.

[0040] In a possible implementation of the first aspect, obtaining the target image based on the second basis image and the first gain map includes: converting the second basis image, wherein the converted second basis image corresponds to the third transfer function; converting the first gain map, wherein the converted first gain map corresponds to the third transfer function; and obtaining the target image based on the converted first gain map and the converted second basis image.

[0041] In a possible implementation of the first aspect, obtaining the target image based on the second basic image and the first gain map includes: converting the second basic image, wherein the converted second basic image corresponds to a fourth transfer function, and the fourth transfer function is different from the third transfer function; converting the first gain map, wherein the converted first gain map corresponds to the fourth transfer function; and obtaining the target image based on the converted first gain map and the converted second basic image.

[0042] In a possible implementation of the first aspect, the metadata further includes information for indicating metadata attributes corresponding to the gain map, where the information is used to indicate that the metadata corresponding to the gain map is applicable to one or more of the luminance component (Y component) or chrominance components (U and / or V components) of the gain map; or, the information is used to indicate that the metadata corresponding to the gain map is applicable to one or more of the R component, G component, or B component of the gain map.

[0043] In a possible implementation of the first aspect, the metadata also includes information for indicating metadata attributes corresponding to the base image, where the information is used to indicate that the metadata corresponding to the base image is applicable to one or more of the luminance component (Y component) or chrominance components (U and / or V components) of the base image; or, the information is used to indicate that the metadata corresponding to the base image is applicable to one or more of the R component, G component, or B component of the base image.

[0044] In a second aspect, the present application provides an encoding method, characterized in that it includes: obtaining a first base image, a first gain map and metadata based on a first image, wherein the first base image corresponds to a first dynamic range, the first image corresponds to a second dynamic range, and the second dynamic range is different from the first dynamic range; encoding the first base image; encoding the first gain map; encoding the metadata.

[0045] In a possible implementation manner of the second aspect, the metadata includes first sampling phase information, where the first sampling phase information is used to indicate a relative positional relationship between pixels of the first base image and the second base image.

[0046] In a possible implementation manner of the second aspect, the first sampling phase information includes pixel coordinates of a first preset position of the first basic image, and the first sampling phase information also includes pixel coordinates of a second preset position of the second basic image.

[0047] In a possible implementation manner of the second aspect, the first sampling phase information includes an offset between pixel coordinates of the second basic image and pixel coordinates of the first basic image.

[0048] In a possible implementation manner of the second aspect, the first sampling phase information includes first luminance sampling phase information and / or first chrominance sampling phase information.

[0049] In a possible implementation manner of the second aspect, the first sampling phase information includes one or more of first red R channel sampling phase information, first green G channel sampling phase information, and first blue B channel sampling phase information.

[0050] In a possible implementation manner of the second aspect, the metadata includes indication information of the first target sampling algorithm.

[0051] In a possible implementation manner of the second aspect, the indication information of the first target sampling algorithm includes an index value used to indicate determination of the first target sampling algorithm from multiple preset sampling algorithms.

[0052] In a possible implementation manner of the second aspect, the indication information of the first target sampling algorithm includes a name or other specific identifier of the first target sampling algorithm.

[0053] In a possible implementation manner of the second aspect, the metadata includes a first target resolution.

[0054] In a possible implementation manner of the second aspect, the metadata includes second sampling phase information, where the second sampling phase information is used to indicate a relative positional relationship between pixels of the first gain map and the second gain map.

[0055] In a possible implementation manner of the second aspect, the second sampling phase information includes pixel coordinates of a first preset position of the first gain map, and the second sampling phase information further includes pixel coordinates of a second preset position of the second gain map.

[0056] In a possible implementation manner of the second aspect, the second sampling phase information includes an offset between pixel coordinates of the second gain map and pixel coordinates of the first gain map.

[0057] In a possible implementation manner of the second aspect, the second sampling phase information includes second luminance sampling phase information and / or second chrominance sampling phase information.

[0058] In a possible implementation manner of the second aspect, the second sampling phase information includes one or more of second red R channel sampling phase information, second green G channel sampling phase information, and second blue B channel sampling phase information.

[0059] In a possible implementation manner of the second aspect, the metadata includes indication information of the second target sampling algorithm.

[0060] In a possible implementation manner of the second aspect, the indication information of the second target sampling algorithm includes an index value used to indicate determination of the second target sampling algorithm from multiple preset sampling algorithms.

[0061] In a possible implementation manner of the second aspect, the indication information of the second target sampling algorithm includes a name or other specific identifier of the second target sampling algorithm.

[0062] In a possible implementation manner of the second aspect, the metadata includes a second target resolution, where the second target resolution is the same as or different from the first target resolution.

[0063] In a third aspect, the present application provides an electronic device comprising a unit for implementing the first aspect or any possible implementation of the first aspect, such as a decoding module and a processing module. The decoding module is configured to decode a base image, a gain map, and metadata from a bitstream, and the processing module is configured to process the base image based on the metadata, and then obtain a target image based on the processed base image and the gain map. Optionally, the electronic device further comprises a display module for displaying the obtained target image. Optionally, the electronic device further comprises a receiving module for receiving a bitstream.

[0064] In a fourth aspect, the present application provides an electronic device, comprising a unit for implementing the second aspect or any possible implementation of the second aspect, for example, a processing module and an encoding module, the processing module being configured to obtain a first base image, a first gain map, and metadata based on a first image, wherein the first base image corresponds to a first dynamic range, the first image corresponds to a second dynamic range, and the second dynamic range is different from the first dynamic range; the encoding module being configured to encode the first base image, the first gain map, and the metadata according to the second aspect or any possible implementation of the second aspect to obtain a code stream. Optionally, the electronic device further comprises a sending module configured to send the code stream.

[0065] In a fifth aspect, the present application provides an electronic device, comprising a processor, the processor being configured to couple with a memory, read and execute instructions and / or program codes in the memory, so as to execute the first aspect or any possible implementation of the first aspect. In one possible implementation, the electronic device further comprises a display or can be connected to an external display, the display being configured to display a target image, the target image being obtained by the first aspect or any possible implementation of the first aspect. In one possible implementation, the electronic device further comprises a communication interface, the communication interface being configured to receive an image or video signal to be processed, the image or video signal to be processed being a code stream or a bit stream.

[0066] In a sixth aspect, the present application provides an electronic device, comprising a processor, the processor being configured to couple with a memory, read and execute instructions and / or program code in the memory, and execute the second aspect or any possible implementation of the second aspect. In one possible implementation, the electronic device further comprises a communication interface, the communication interface being configured to transmit a code stream, the code stream being obtained according to the second aspect or any possible implementation of the second aspect.

[0067] In a seventh aspect, the present application provides a chip system comprising a logic circuit, which is used to couple with an input / output interface and transmit data through the input / output interface to execute the first aspect or any possible implementation of the first aspect.

[0068] In an eighth aspect, the present application provides a chip system comprising a logic circuit, wherein the logic circuit is used to couple with an input / output interface and transmit data through the input / output interface to execute the second aspect or any possible implementation of the second aspect.

[0069] In a ninth aspect, the present application provides a computer-readable storage medium storing a program code. When the program code stored in the computer storage medium runs on a computer, the computer executes the first aspect or any possible implementation of the first aspect.

[0070] In the tenth aspect, the present application provides a computer-readable storage medium, which stores program code. When the program code stored in the computer storage medium is run on a computer, the computer executes the second aspect or any possible implementation of the second aspect.

[0071] In an eleventh aspect, the present application provides a computer program product, comprising: a computer program code, which, when executed on a computer, enables the computer to execute the first aspect or any possible implementation of the first aspect.

[0072] In a twelfth aspect, the present application provides a computer program product, which includes: a computer program code, which, when running on a computer, enables the computer to execute the second aspect or any possible implementation of the second aspect.

[0073] In a thirteenth aspect, the present application provides a code stream obtained by encoding an image according to the second aspect or any possible implementation method of the second aspect.

[0074] In a fourteenth aspect, the present application provides a code stream, which includes a first base image, a first gain map and metadata, and the metadata includes information required to process the first base image and the first gain map to obtain a target image.

[0075] In a possible implementation of the thirteenth aspect or the fourteenth aspect, the metadata includes the content included in the metadata in any possible implementation of the second aspect.

[0076] In a fifteenth aspect, the present application provides a streaming media system comprising a client device and a content delivery network (CDN). The client device is configured to implement the first aspect or any possible implementation of the first aspect to obtain a target image for display; the content delivery network is configured to send a code stream comprising a base image, a gain map, and metadata to the client device. Optionally, the system further comprises a cloud server configured to receive content requests from client devices, make decisions, respond to the client, and send the client the address of the client-requested content on the CDN.

[0077] In a sixteenth aspect, the present application provides a code stream storage device, comprising a communication interface and a memory, the communication interface being configured to receive a code stream, the code stream being as described in the thirteenth or fourteenth aspect, or any possible implementation of the thirteenth or fourteenth aspect, and the memory being configured to store the code stream. In one possible implementation, the communication interface is further configured to transmit the code stream to a client device based on an instruction or user request. Optionally, the code stream storage device further comprises a processor configured to transcode the received code stream to transmit the transcoded code stream to different terminal devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] FIG1 is a schematic diagram illustrating an exemplary dynamic range mapping;

[0079] FIG2 is a schematic diagram showing an exemplary PQ photoelectric transfer function;

[0080] FIG3 is a schematic diagram showing an exemplary HLG photoelectric transfer function;

[0081] FIG4 is a schematic diagram illustrating an exemplary SLF photoelectric transfer function;

[0082] FIG5 is an end-to-end schematic diagram illustrating an exemplary HDR image / video from production to display;

[0083] FIG6A is a diagram showing an exemplary embodiment of a terminal device;

[0084] FIG6 is a schematic diagram of a method for processing a digital signal according to an embodiment of the present application;

[0085] FIG6B is a schematic diagram of a system architecture or scenario of an application provided according to an embodiment of the present application;

[0086] FIG7A is a schematic flowchart of an encoding method provided according to an embodiment of the present application;

[0087] FIG7 is a schematic flow chart of another encoding method provided according to an embodiment of the present application;

[0088] FIG8 is a schematic flowchart of a method for processing a digital signal according to an embodiment of the present application;

[0089] FIG9A is a flow chart of a signal processing method according to an embodiment of the present application;

[0090] FIG9 is a schematic diagram of a method for processing a digital signal according to an embodiment of the present application;

[0091] FIG10 is a schematic flowchart of another method for processing a digital signal according to an embodiment of the present application;

[0092] FIG11 is a schematic flowchart of another method for processing a digital signal according to an embodiment of the present application;

[0093] FIG12 is a schematic flow chart of another method for processing a digital signal according to an embodiment of the present application;

[0094] FIG1A is a schematic block diagram of a coding and decoding system used in an embodiment of the present application;

[0095] FIG2A is a block diagram of a content providing system for implementing a content distribution service according to an embodiment of the present application;

[0096] FIG2B is a schematic diagram of an example structure of the terminal device 2106 in FIG2A ;

[0097] FIG3A is a schematic diagram of a workflow of a streaming media system used in an embodiment of the present application;

[0098] FIG3B is a schematic diagram of a streaming media system architecture used in an embodiment of the present application;

[0099] FIG4A is a schematic diagram of a possible system architecture applicable to an embodiment of the present application;

[0100] FIG4B is a schematic diagram of the structure of an image processing system provided in an embodiment of the present application;

[0101] FIG13 is an exemplary diagram of an image encoding device provided in an embodiment of the present application;

[0102] FIG13A is an exemplary diagram of an electronic device provided in an embodiment of the present application;

[0103] FIG14 is an exemplary diagram of an electronic device provided in an embodiment of the present application;

[0104] FIG15 is an exemplary diagram of an electronic device provided in an embodiment of the present application;

[0105] FIG16 is a schematic diagram showing the positional relationship between current image pixels and sampled image pixels according to an embodiment of the present application;

[0106] FIG17 is a schematic diagram of phase information of luminance pixels and chrominance pixels provided in an embodiment of the present application. DETAILED DESCRIPTION

[0107] In order to make the purpose, technical solutions and advantages of this application clearer, this application will be further described in detail below with reference to the accompanying drawings.

[0108] The term "at least one" used in this application refers to one or more than one, including one, two, three and more; "plurality" refers to two or more than two, including two, three and more.

[0109] The term "at least one" in the embodiments of the present application refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where any of a, b, c, ab, ac, bc, or abc can include a single a, a single b, a single c, and can also include multiple a, multiple b, and multiple c.

[0110] Furthermore, unless otherwise indicated, ordinal numbers such as "first" and "second" in the embodiments of this application are used to distinguish between multiple objects and are not used to define the order, timing, priority, or importance of multiple objects. For example, the first priority criterion and the second priority criterion are only used to distinguish different criteria and do not indicate differences in the content, priority, or importance of the two criteria.

[0111] In addition, the terms "including" and "having" in the embodiments, claims, and drawings of this application are not exclusive. For example, a process, method, system, product, or device that includes a series of steps or modules / units is not limited to the listed steps or modules and may also include steps or modules / units that are not listed.

[0112] HDR images or HDR videos may need to be transcoded during distribution or transmission. The current HDR transcoding solution is to directly transcode HDR images or HDR videos. After decoding, the HDR image or HDR video obtains HDR pixel values ​​and HDR format identifiers and information. HDR pixel values ​​usually need to be processed (such as image up / down sampling, enhancement, etc.). The HDR format identifier and information, as well as the processed HDR pixel values, are then sent to the encoder for re-encoding. The re-encoded HDR image or HDR video is distributed to other devices.

[0113] This solution of directly transcoding HDR images or HDR videos (which can be called a "single-layer transcoding solution") may cause anomalies in the transcoded HDR images or HDR videos, affecting the user experience. Because when a single-layer encoded HDR stream is transcoded (decoded and then re-encoded), it depends on the transcoding system to correctly send all the HDR information obtained by the decoder to the encoder. If the information is lost, the transcoding system will fail to re-encode in the correct format, resulting in an erroneous bitstream, which will cause abnormal visual effects when the user watches it, seriously affecting the user experience. Because many of the distribution and transcoding systems on the market have not been upgraded to support the correct transmission of HDR information from the decoding to the encoding end, mainstream mobile phone and application manufacturers have received many user complaints in this regard.

[0114] First, in order to facilitate understanding of the embodiments of the present application, some concepts or terms involved in the embodiments of the present application are explained.

[0115] Color value: A numerical value corresponding to a specific image primary color component (such as R, G, B, or Y).

[0116] Digital code value: A digital representation of an image signal. Digital code value is used to represent nonlinear primary color values.

[0117] Linear color value: A linear color value that is proportional to the light intensity. In an optional case, its value should be normalized to [0,1], referred to as E.

[0118] Nonlinear primary color value: A nonlinear primary color value is a normalized digital expression of image information, which is proportional to the digital coding value. In an optional case, its value should be normalized to [0,1], referred to as E′.

[0119] Electro-optical transfer function (EOTF): A conversion relationship from nonlinear primary color values ​​to linear primary color values.

[0120] Metadata: Data that describes the video source information carried in the video signal.

[0121] Dynamic metadata: metadata associated with each frame of an image that changes from frame to frame.

[0122] Static metadata: Metadata associated with an image sequence that remains unchanged within the image sequence.

[0123] Luminance signal (luma): represents the combination of nonlinear primary color signals, symbolized by Y'.

[0124] Luminance mapping: The luminance of the source image is mapped to the luminance of the target system.

[0125] Display adaptation: Processing of the video signal to adapt it to the display characteristics of the target display.

[0126] Source picture: The image input to the HDR pre-processing stage.

[0127] Mastering display: A reference display used when editing and producing video signals to determine the effects of video editing and production.

[0128] Linear scene light signal: In HDR video technology, the HDR video signal with content as scene light refers to the scene light captured by the camera / camera sensor, which is generally a relative value; the linear scene light signal is obtained after HLG encoding to obtain the HLG signal, which is a scene light signal and is nonlinear; the scene light signal generally needs to be converted into a display light signal through OOTF and displayed on the display device.

[0129] Linear display light signal: In HDR video technology, the HDR video signal with content as display light refers to the display light emitted by the display device, generally expressed as an absolute value in nits. The linear display light signal is PQ-encoded to obtain a PQ signal, which is a nonlinear display light signal. The display light signal is generally displayed on the display device according to its absolute brightness.

[0130] Optical-to-optical conversion factor (OOTF): A curve used in video technology to convert one optical signal into another.

[0131] Dynamic range: The ratio of the maximum brightness to the minimum brightness in a video or image signal.

[0132] High dynamic range (HDR) images have a dynamic range between 0.001 and 10,000 nits, where nit is a unit of illumination.

[0133] Standard dynamic range (SDR) images, also known as low dynamic range, are images with a dynamic range generally between 1 nit and 100 nit.

[0134] Luma-chroma-chroma (LCC) separates the three components of a video signal into luminance and chrominance.

[0135] The optical-electro transfer function (OETF) represents the conversion relationship between linear signals and nonlinear signals of image pixels. Currently, the commonly used OETFs include the following three:

[0136] The three OTFs are Perceptual Quantizer (PQ), Hybrid Log-Gamma (HLG), and Scene Luminance Fidelity (SLF). These three OTFs are specified by the Audio Video Coding Standard (AVS).

[0137] Dynamic range is used in many fields to express the ratio of the maximum value to the minimum value of a variable. In digital images, dynamic range represents the ratio between the maximum grayscale value and the minimum grayscale value within the displayable range of the image. The dynamic range in nature is quite large. The brightness of the night scene under the starry sky is about 0.001cd / m2, and the brightness of the sun itself is as high as 1,000,000,000cd / m2. Such a dynamic range reaches the order of 1,000,000,000 / 0.001=1013. However, in real scenes in nature, the brightness of the sun and the brightness of starlight will not be obtained at the same time. For natural scenes in the real world, the dynamic range is between 10 -3 to 10 6 In most color digital images, each channel of R, G, and B is stored in an 8-bit byte. That is, the range of each channel is 0 to 255 gray levels. Here, 0 to 255 is the dynamic range of the image. In the real world, the dynamic range of the same scene is between 10 and 255. -3 to 10 6 The range of real-world dynamic range is called high dynamic range (HDR), while the dynamic range of ordinary images is called low dynamic range (LDR). The imaging process of a digital camera is actually a mapping from the high dynamic range of the real world to the low dynamic range of the photograph. This is often a nonlinear process.

[0138] FIG1 is a schematic diagram of dynamic range mapping.

[0139] The PQ photoelectric transfer function (PQPTF) is a perceptually quantized PQPTF proposed based on the human eye's brightness perception model. See Figure 2, which shows an image of the PQPTF.

[0140] The PQ photoelectric transfer function represents the conversion relationship between the linear signal value of the image pixel and the nonlinear signal value in the PQ domain. The PQ photoelectric transfer function can be expressed as formula (1):

[0141] The calculation of each parameter in formula (1) is as follows:

[0142] Where L represents the linear signal value, which is normalized to [0, 1]. L' represents the nonlinear signal value, which ranges from [0, 1]. m1, m2, c1, c2, and c3 are PQ photoelectric transfer coefficients, and m1, m2, c1, c2, and c3 satisfy the following relationship:

[0143] The HLG photoelectric transfer function is an improvement on the traditional Gamma curve. See Figure 3, which shows an image of the HLG photoelectric transfer function.

[0144] The HLG photoelectric transfer function applies the traditional Gamma curve in the low range and supplements the log curve in the high range. The HLG photoelectric transfer function represents the conversion relationship between the linear signal value of the image pixel and the nonlinear signal value in the HLG domain. The HLG photoelectric transfer function can be expressed as formula (3):

[0145] Wherein, L represents the linear signal value, and its value range is [0, 12]; L' represents the nonlinear signal value, and its value range is [0, 1]; a, b, and c are HLG photoelectric transfer coefficients, a=0.17883277, b=0.28466892, and c=0.55991073.

[0146] The SLF photoelectric transfer function (SLF) is an optimal curve obtained based on the brightness distribution of an HDR scene, while satisfying the optical characteristics of the human eye. See Figure 4, which shows an image of the SLF photoelectric transfer function.

[0147] The SLF photoelectric transfer curve represents the conversion relationship between the linear signal value of an image pixel and the nonlinear signal value in the SLF domain. The conversion relationship between the linear signal value of an image pixel and the nonlinear signal value in the SLF domain is shown in formula (3):

[0148] Among them, the SLF photoelectric transfer function can be expressed as formula (5):

[0149] Where L represents the linear signal value, which is normalized to [0, 1]. L' represents the nonlinear signal value, which ranges from [0, 1]. p, m, a, and b are the SLF photoelectric transfer coefficients, where p = 2.3, m = 0.14, a = 1.12762, and b = -0.12762.

[0150] Linear space: In this application, linear space refers to the space where linear light signals are located.

[0151] Nonlinear space: In this application, nonlinear space refers to the space resulting from the conversion of linear light signals using nonlinear curves. Commonly used nonlinear curves for HDR include the PQ EOTF-1 curve and the HLG OETF curve, while common nonlinear curves for SDR include the gamma curve. It is generally believed that linear light signals encoded using these nonlinear curves are visually linear relative to the human eye. It should be understood that nonlinear space can be considered a visually linear space.

[0152] Gamma correction: Gamma correction is a method of nonlinearly editing the tonal range of an image. It detects dark and light portions of the image signal and increases their ratio, thereby improving image contrast. The photoelectric conversion characteristics of current display screens, photographic film, and many electronic cameras can be nonlinear. The relationship between the output and input of these nonlinear components can be expressed as a power function: output = (input)γ.

[0153] The nonlinear conversion of the color values ​​output by the device is due to the nonlinear nature of the human visual system. Humans perceive visual stimuli through comparison. External stimuli are intensified at a certain rate, resulting in a uniform increase in the stimulus. Therefore, physical quantities that increase in geometric progression appear uniform to human perception. To display input colors in accordance with human visual principles, the aforementioned nonlinear conversion using the power function is necessary, converting linear color values ​​to nonlinear ones. The gamma value, γ, can be determined based on the photoelectric conversion curve in the color space.

[0154] Color space: Color can be the different perceptions of the eye for light of different frequencies, or it can represent objectively existing light of different frequencies. Color space is a color range defined by a coordinate system established by people to represent colors. The color gamut and color model together define a color space. Among them, the color model is an abstract mathematical model that represents color with a set of color components. The color model may include, for example, the three primary color light model (red, green, blue, RGB) and the four-color printing model (cyan, magenta, yellow key plate, CMYK). The color gamut refers to the sum of the colors that a system can produce. For example, Adobe RGB and sRGB are two different color spaces based on the RGB model.

[0155] Each device, such as a monitor or printer, has its own color space and can only produce colors within its gamut. When you move an image from one device to another, the colors may change on different devices because each device converts and displays RGB or CMYK according to its own color space.

[0156] RGB space refers to a space that uses the brightness of the three primary colors of red, green, and blue to quantitatively represent video signals; YCC space is a color space that represents the separation of brightness and color. The three components of the YCC space video signal represent brightness, chroma, and chroma, respectively. Common YCC space video signals include YUV, YCbCr, ICtCp, etc.

[0157] To obtain images with a higher dynamic range, the image bit width is usually greater than or equal to 10 bits. Common coding standards that support HDR include H.266, H.265, and the High Efficiency Image Format (HEIF). The common Joint Photographic Experts Group (JPEG) and H.264 only support 8-bit encoding, so they cannot well support HDR video and HDR images.

[0158] HDR images and videos are an end-to-end process for achieving a better experience. Figure 4 shows the end-to-end process for HDR video.

[0159] Figure 5 is an end-to-end schematic diagram of HDR images or videos from production to display. As shown in Figure 5, the original video file (also called a master file) is obtained through material production (such as shooting videos, making computer graphics (CG) videos, etc.), editing, color grading and other procedures. Then, based on the original video file, the corresponding dynamic metadata is obtained. After encoding the original video file and the dynamic metadata, a compressed video is obtained. The compressed video is distributed / transmitted (the distribution end is such as a streaming media server, CDN, etc., and the distribution end often needs to be transcoded, that is, the process of decoding first and then encoding) to reach the terminal device (for example, a computer, set-top box, mobile phone, tablet computer, etc.). The terminal device decodes the compressed video to obtain a decompressed video, and then displays it to the user through a display device (for example, a monitor, TV, etc.).

[0160] For ease of description, the term "HDR object" is used in some embodiments of this application. An HDR object can be a static HDR image (also known as an HDR image, HDR photo, HDR picture, etc.), an HDR video, or other types of dynamic HDR images, or a frame within an HDR video or dynamic HDR image.

[0161] It is understood that for ease of description, some embodiments of the present application use HDR images as an example to introduce the technical solutions of the present application. However, it is understood that these embodiments can be applied not only to HDR images, but also to other HDR objects, such as HDR videos, dynamic HDR images, and a frame of an HDR video or dynamic HDR image.

[0162] Figure 6A is an exemplary diagram of a terminal device. As shown in Figure 6A, the terminal device 600 includes a decoding module and a processing module. The decoding module is configured to decode a base image, metadata, and a gain map from a bitstream. The processing module is configured to process the base image based on the metadata to obtain a processed base image, and then obtain a target image based on the processed base image and the gain map. Optionally, the terminal device 600 also includes a display module for displaying the obtained target image.

[0163] The process of the decoding module includes decoding the received code stream through any decoder (HEVC, JPEG, ProRes, HEIF) to obtain the base image, metadata and gain map. The base image and gain map obtained in this process contain image data in any color space form such as RGB or YUV. It should be noted that this application does not limit the code stream format. In terms of color space, it can be YUV or RGB. In terms of the bit width of the data, the code stream can be 8 bits, 10 bits, or 12 bits, etc. In some embodiments, the decoding module can obtain the metadata from the SEI of HEVC or VVC, the user-defined NAL unit or the reserved packet unit, the JFIF-encapsulated APP extension information, or the MP4-encapsulated data segment; it can also obtain the metadata from other file locations, such as the position after the EOI (end of image) of the complete JPEG file.

[0164] It should be noted that metadata mainly includes data such as the source data format, region division information, region traversal order information, or image features and curve parameters, as well as one or more metadata information units. Metadata information units include data such as coordinate information, image features, and curve parameters.

[0165] In some embodiments, the decoding module can obtain the gain map from the SEI of HEVC or VVC, a user-defined NAL unit or a reserved packet unit, or the APP extension information encapsulated in JFIF, or the data segment encapsulated in MP4; or the gain map can be obtained from other file locations indicated in the metadata, such as the position after the EOI (end of image) of the complete JPEG file.

[0166] The gain map may be: one pixel value corresponds to one gain map, or multiple pixel values ​​correspond to one gain map, wherein the gain map may contain multiple values, and the gain map corresponding to each pixel contains the same number of values.

[0167] Figure 6 is a schematic diagram of a method for processing digital signals according to an embodiment of the present application. As shown in Figure 6, the method for processing digital signals provided in an embodiment of the present application can be applied at a decoding end.

[0168] The decoding module (also called decoder) can obtain base layer data, enhancement layer data and metadata from the received bitstream. Then, the decoding module can synthesize the base layer data and enhancement layer data to obtain an HDR image.

[0169] The graphics processing module can process the HDR image. The graphics processing module can also process the base layer data. The graphics processing module can send the processed HDR image and base layer data to the display module.

[0170] The display module may present an HDR image based on the received data (ie, the processed HDR image and the processed base layer data).

[0171] For the sake of convenience of description, the HDR image obtained by the decoding module based on the basic layer data and the enhancement layer data can be called HDR image 1, the HDR image obtained after the graphics processing module processes the HDR image 1 can be called HDR image 2, and the HDR image displayed by the display module can be called HDR image 3.

[0172] The base layer data, which may also be referred to as a base layer image, basic data, a base image or a basic image, may be an SDR image or an HDR image with a lower dynamic range.

[0173] The enhancement layer data, also known as enhancement data, enhancement layer image, or enhancement image, may include some image detail information. The image detail information contained in the enhancement layer data is then used to supplement the base layer data, thereby synthesizing an HDR image with better contrast (i.e., HDR image 1 mentioned above).

[0174] The dynamic range of the base layer data is smaller than the dynamic range of the HDR image (ie, HDR image 1) determined based on the base layer data and the enhancement layer data.

[0175] For example, in some embodiments, the base layer data may be an SDR image, and the HDR image 1 may be an HDR image. The HDR image and the SDR image may have at least one of a different photoelectric conversion function, a different color gamut, and a different dynamic range.

[0176] For another example, in some other embodiments, the base layer data may be an HDR image, but the dynamic range of the HDR image is smaller than the dynamic range of the HDR image 1 .

[0177] For example, assuming the dynamic range of the base layer data is DR x1 to DR x2 , the dynamic range of HDR image 1 is DR y1 to DR y2 In some embodiments, DR y1 Less than DR x1 , DR y2 Greater than DR x2 In other embodiments, DR y1 Less than DR x1 , DR y2 Equal to DR x2 In other embodiments, DR y1 Equal to DR x1 , DR y2 Greater than DR x2 In other words, the base layer data can be considered to be an image with a smaller dynamic range than the HDR image 1.

[0178] In some embodiments, the resolution of the base layer data may be 720P, 1080P, 2K, 4K, or 8K, etc.

[0179] In some embodiments, the resolution of the enhancement layer data may be 720P, 1080P, 2K, 4K, or 8K, etc.

[0180] In some embodiments, the resolution of the HDR image 1 may be 720P, 1080P, 2K, 4K, or 8K, etc.

[0181] In some embodiments, the resolution of the HDR image 2 may be 720P, 1080P, 2K, 4K, or 8K, etc.

[0182] In some embodiments, the resolution of the HDR image 3 may be 720P, 1080P, 2K, 4K, or 8K, etc.

[0183] In some embodiments, the resolution of the base layer data may be the same as the resolution of the enhancement layer data, for example, the resolution of both the base layer data and the enhancement layer data may be 4K.

[0184] In other embodiments, the resolution of the base layer data may be different from the resolution of the enhancement layer data. For example, in some embodiments, the resolution of the base layer data may be greater than the resolution of the enhancement layer data. For example, the resolution of the base layer data may be 4K, and the resolution of the enhancement layer data may be 2K or 1080P. For another example, in other embodiments, the resolution of the base layer data may be smaller than the resolution of the enhancement layer data. For example, the resolution of the base layer data may be 1080P or 2K, and the resolution of the enhancement layer data may be 4K.

[0185] Figure 6B is a system architecture or scenario diagram of an application provided according to an embodiment of the present application. Exemplarily, the system is implemented by terminal device 600B. The system first decodes / acquires basic data (for example, the basic image in Figure 6A), metadata and enhanced data (for example, the gain map in Figure 6A), and then processes the basic data and enhanced data separately based on the metadata, and then synthesizes the processed basic data and enhanced data to obtain an HDR image. The relevant limitations on basic layer data, enhanced layer data, metadata, and HDR images in the above-mentioned embodiment of Figure 6 also apply to the embodiment of Figure 6B and will not be repeated here.

[0186] FIG7A is a schematic flowchart of an encoding method provided according to an embodiment of the present application, including:

[0187] 0701. Obtain a first base image, a first gain map, and metadata based on a first image, wherein the first base image corresponds to a first dynamic range, the first image corresponds to a second dynamic range, and the second dynamic range is different from the first dynamic range.

[0188] 0702, encode the first basic image;

[0189] 0703, encode the first gain map;

[0190] 0703, encode the metadata.

[0191] Optionally, in some embodiments, the first image is a high dynamic range (HDR) image. The acquisition of the HDR image is described in other embodiments of the present application and will not be repeated here.

[0192] Optionally, the process of obtaining the first basic image according to the first image refers to the description of other embodiments of the present application and will not be repeated here. The present application does not limit the data format of the basic image. In terms of color space, the basic image can be YUV or RGB. In terms of the bit width of the data, the basic image can be 8 bits, 10 bits, or 12 bits, etc. The photoelectric conversion characteristics / functions corresponding to the basic image can be PQ, gamma, log, HLG, etc.

[0193] Optionally, encoding the first base image may include: step A, obtaining an intermediate base image; step B, processing the intermediate base image to obtain a second base image, and obtaining metadata corresponding to the base image; and step C, encoding the first base image, the intermediate base image, or the second base image. The encoding of the first base image may refer to the encoding process of the base layer image base[i] in other embodiments of the present application and will not be repeated here.

[0194] It should be noted that, in this application, the base image may also be referred to as base layer image / base layer data / base layer image data / base layer image / base layer data / base layer image data, and the gain map may also be referred to as enhanced data.

[0195] Optionally, encoding the first gain map may include: step A, obtaining an intermediate gain map based on the first image (which may be an HDR image or an SDR image) and the second basic image; step B, downsampling the intermediate enhancement data to obtain second intermediate enhancement data enhance; step C, processing the second intermediate enhancement data enhance to obtain final enhancement data; step D, downsampling the base layer image data; step E, encoding the final enhancement data and metadata corresponding to the gain map.

[0196] Step A, obtaining an intermediate gain map based on the first image (which may be an HDR image or an SDR image) and the second base image, can be implemented by the following scheme:

[0197] Solution 1: Intermediate enhancement data = high dynamic range data / f(baseAfter[i]), where f() is a conversion function of the numerical domain, such as log, OETF or EOTF, piecewise curve, etc., and may also include processing of the second basic layer data (obtaining the data range (minimum value and / or maximum value) of the second basic layer data, mapping the minimum value of the second basic layer data to 0 or a preset value, and / or mapping the maximum value to 1.0 or a preset value, and / or mapping the intermediate value to a certain intermediate value based on the mapping relationship between the maximum value and / or the minimum value.)

[0198] Solution 2: Intermediate enhanced data = high dynamic range data - f(baseAfter[i]), where f() is a conversion function of the numerical domain, such as log, OETF or EOTF, segmented curve, etc., and may also include processing of the second basic layer data (obtaining the data range (minimum value and / or maximum value) of the second basic layer data, mapping the minimum value of the second basic layer data to 0 or a preset value, and / or mapping the maximum value to 1.0 or a preset value, and / or mapping the intermediate value to a certain intermediate value according to the mapping relationship between the maximum value and / or the minimum value).

[0199] Step B, downsampling the intermediate enhanced data to obtain the second intermediate enhanced data enhance, can be implemented by the following scheme:

[0200] Solution 1: No downsampling, direct: Second intermediate enhanced data = intermediate enhanced data

[0201] Solution 2: Pass an index or pass an index for each area, and select a better interpolation method from a preset set. The preset set can be GL_NEAREST (nearest neighbor) and GL_LINEAR (nearest neighbor) of OPENGL texture interpolation methods; it can also be NEAREST (nearest neighbor) or LINEAR (nearest neighbor) of Vulkan texture interpolation methods; it can also be passing multiple sets of filters, or a method containing multiple directional interpolation methods, or bicubic splines; it can also be a full set or a subset of all the above methods. Select the method that minimizes the difference between the second intermediate enhancement data after downsampling and upsampling using this method and the original intermediate enhancement data, and add this method to the enhancement metadata (that is, the metadata corresponding to the enhancement data or gain map).

[0202] Option 3: Select a default texture interpolation method from OpenGL, Vulkan, or Matl. This can be GL_NEAREST (nearest neighbor) or GL_LINEAR (nearest neighbor) from OpenGL, or NEAREST (nearest neighbor) or LINEAR (nearest neighbor) from Vulkan. Select the method that minimizes the difference between the second intermediate enhancement data after downsampling and upsampling using this method and the original intermediate enhancement data, and add this method to the enhancement metadata.

[0203] Solution 4: Use a pre-set filter or a combination of directional interpolation methods, such as bicubic splines. Select the method that minimizes the difference between the second intermediate enhancement data after downsampling and upsampling using this method and the original intermediate enhancement data, and add this method to the enhancement metadata.

[0204] Step C, processing the second intermediate enhanced data enhance to obtain final enhanced data, can be implemented by the following scheme:

[0205] Solution 1: Obtain the data range (minimum value, sum, or maximum value) of the second intermediate enhanced data, map the minimum value of the second intermediate enhanced data to 0 or a preset value, and / or map the maximum value to 1.0 or a preset value, and then map the intermediate value to a certain intermediate value according to the mapping relationship between the maximum value and / or the minimum value, thereby obtaining the final enhanced data.

[0206] Solution 2: Get the histogram of the second intermediate enhanced data, use the histogram equalization method to obtain the mapping relationship TME(), and then use the mapping function enhanceAfter[i]=TMB(enhance[i]) or enhanceAfter[i]=TMB()*enhance[i].

[0207] The mapping relationship has various forms, such as Sigmoid, cubic spline, gamma, straight line, etc. or their inverse function forms, which are not limited in the present invention. The following curves can be used:

[0208] It should be noted that L and L' can be normalized optical signals or electrical signals, which is not limited in the present invention.

[0209] It should be noted that the present invention does not limit the format of metadata. Metadata can include histogram information and tone-mapping curve parameter information as in ST2094-40, or can include tone-mapping curve parameter information as in ST2094-10.

[0210] It should be noted that the present invention does not limit enhanceAfter[i] and enhance[i] to any domain, which can be a linear domain, PQ domain, log domain, etc. The present invention also does not limit the color space of enhanceAfter[i] and enhance[i], which can be YUV, RGB, Lab, HSV, etc.

[0211] It should be noted that color gamut mapping can be added before or after processing to convert from the current color gamut to the target color gamut. The current color gamut and the target color gamut include but are not limited to BT.2020, BT.709, DCI-P3, sRGB, etc.

[0212] Step D, downsampling the base layer image data, can be achieved by the following scheme:

[0213] Traverse multiple preset scaling ratios, traverse multiple preset downsampling phases, traverse multiple preset up / downsampling algorithms, adopt the same algorithm for downsampling, encoding, decoding, and then upsampling the base layer image, and select the optimal parameters. Specifically:

[0214] Iterate through multiple preset zoom levels

[0215] The preset multiple scaling ratios P ​​include: for example, if P is 1 / 2, then 1 / 2 in the horizontal direction, 1 / 2 in the vertical direction, and 1 / 2 in both the vertical and horizontal directions.

[0216] It is also possible to use a specific preset zoom ratio.

[0217] Iterate through multiple preset downsampling phases

[0218] The preset downsampling phase includes dividing the preset downsampling scaling ratio P into O positions, such as 0, P / O, 2*P / O, . . . , (O-1)*P / O.

[0219] It is also possible to use a specific preset downsampling phase.

[0220] Traverse multiple preset up / down sampling algorithms A and downsample the enhanced data

[0221] The preset algorithm is selected from OpenGL, Vulkan, or matel texture interpolation methods. This can be GL_NEAREST (nearest neighbor) or GL_LINEAR (nearest neighbor) from OpenGL, or NEAREST (nearest neighbor) or LINEAR (nearest neighbor) from Vulkan. Alternatively, it can be a preset filter (similar to the eight-tap filter in H.265's MC module or the eight-tap filter used for upsampling / downsampling between SHVC layers), or a combination of directional interpolation methods, or even bicubic splines.

[0222] It is also possible to use a specific preset up / down sampling algorithm.

[0223] Encode, decode, and then upsample

[0224] 1. Use the pre-selected codec algorithm to encode and decode the enhanced data first;

[0225] 2. Then, the decoded reconstructed data is upsampled using the upsampling algorithm corresponding to downsampling algorithm A. If the upsampling algorithm is the preset fixed upsampling algorithm B, upsampling algorithm B is used for upsampling.

[0226] Select the optimal parameters for subsequent encoding process

[0227] This can be:

[0228] 1. Method 1: Calculate the distortion and bit rate of the original enhanced data and the reconstructed enhanced data, and select the one with better rate-distortion (distortion + lamda coefficient * bit rate) performance.

[0229] 2. Method 2: Combine the reconstructed enhanced data with the basic data to obtain the reconstructed high dynamic range data, calculate the distortion between the original high dynamic range data and the reconstructed high dynamic range data, and use the bit rate of the enhanced data encoding to select the one with better rate-distortion (distortion + lamda coefficient * bit rate) performance.

[0230] Step E, encoding the metadata corresponding to the final enhanced data and the gain map, can be achieved by the following scheme:

[0231] For encoding the enhanced data, codecs such as JPEG, HEIF, H.264, and HEVC can be used, and this invention does not limit this.

[0232] Enhancement metadata can be placed in non-coded pixel fields such as SEI and APPN.

[0233] This application provides an image encoding method that is compatible with multiple formats, can achieve relatively good image quality on systems with different support capabilities, and provides compatibility with systems with different capabilities. Among them, upsampling or downsampling can significantly reduce file size.

[0234] FIG7 is a schematic flowchart of an encoding method provided according to an embodiment of the present application.

[0235] 701. Acquire a first HDR image and first base layer data corresponding to the first HDR image.

[0236] 702, encode the first base layer data;

[0237] 703. Determine first enhancement layer data according to the first base layer data and the first HDR image.

[0238] 704, encode the first enhancement layer data;

[0239] 705 : Encode first metadata, where the first metadata includes metadata of the first base layer data and metadata of the first enhancement layer data.

[0240] Optionally, in some embodiments, determining the first enhancement layer data based on the first base layer data and the first HDR image includes: determining the second base layer data based on the first base layer data; determining the first enhancement layer data based on the second base layer data and the first HDR image, and adjusting, based on the concept of rate-distortion optimization, a method for converting the first enhancement layer data determined based on the second base layer data and the first HDR image, so that the encoding of the first enhancement layer data meets the rate-distortion performance target.

[0241] Optionally, in some embodiments, determining the second basic layer data based on the first basic layer data includes: determining intermediate basic layer data based on the first basic layer data; and determining the second basic layer data based on the intermediate basic layer data.

[0242] Optionally, in some embodiments, determining the intermediate basic layer data based on the first basic layer data includes: determining the intermediate basic layer data to be the first basic layer data.

[0243] Optionally, in some embodiments, the intermediate base layer data is determined based on the first base layer data, including: encoding the first base layer data to obtain first base layer encoding information; decoding the first base layer encoding information result to obtain first base layer decoding information, and the intermediate base layer data is the first base layer decoding information.

[0244] Optionally, in some embodiments, determining the second basic layer data based on the intermediate basic layer data includes: determining the second basic layer data to be the intermediate basic layer data.

[0245] Optionally, in some embodiments, the second basic layer data is determined based on the intermediate basic layer data, including: mapping at least one basic layer characteristic value to at least one basic layer reference value respectively to obtain the second basic layer data, the second basic layer data including the at least one basic layer reference value and the values ​​of pixels in the intermediate basic layer data except the at least one basic layer characteristic value, the at least one basic layer characteristic value including at least one of the following values: the maximum value, the minimum value, or the intermediate value of the pixels of the intermediate basic layer data.

[0246] Optionally, in some embodiments, the second basic layer data is determined based on the intermediate basic layer data, including: determining at least one characteristic brightness value based on the first HDR image; determining at least one pixel from the intermediate basic layer data, the at least one pixel corresponding one-to-one to the at least one characteristic brightness value, and the position of each pixel in the at least one pixel is the same as the position of the corresponding characteristic brightness value; fitting function parameters according to a preset or selected mapping function based on the at least one characteristic brightness value and the value of the at least one pixel, to determine a first mapping relationship; and converting the intermediate basic layer data into the second basic layer data according to the first mapping relationship.

[0247] Optionally, in some embodiments, determining at least one characteristic brightness value based on the first HDR image includes: determining a brightness value at a peak position in a histogram of the first HDR image as the characteristic brightness value.

[0248] Optionally, in some embodiments, determining at least one characteristic brightness value based on the first HDR image includes: determining at least one reference area in the first HDR image; determining a reference brightness value of each reference area in the at least one reference area as the characteristic brightness value, the reference brightness value of each reference area being the average brightness value or the maximum brightness value of each reference area.

[0249] Optionally, in some embodiments, converting the intermediate base layer data into the second base layer data according to the first mapping relationship includes: the intermediate base layer data and the second base layer data satisfy the following relationship: baseAfter[i]=TMB1(R base[i]_1 ),

[0250] Wherein, baseAfter[i] represents the value of the i-th pixel in the second basic layer data, R base[i]_1 represents the first reference value of the i-th pixel determined according to the value of the i-th pixel in the intermediate basic layer data, and TMB1() represents the first mapping relationship.

[0251] Optionally, in some embodiments, converting the intermediate base layer data into the second base layer data according to the first mapping relationship includes: the intermediate base layer data and the second base layer data satisfy the following relationship: baseAfter[i]=TMB1(R base[i]_2 )×R base[i]_3 ,

[0252] Wherein, baseAfter[i] represents the value of the i-th pixel in the second basic layer data, TMB1() represents the first mapping relationship, R base[i]_2 is the second reference value of the ith pixel determined according to the value of the ith pixel in the intermediate base layer data, R base[i]_3 It is the third reference value of the i-th pixel determined according to the value of the i-th pixel in the intermediate base layer data.

[0253] Optionally, in some embodiments, the second basic layer data and the first HDR image determine the first enhancement layer data, including: determining first intermediate enhancement data based on the second basic layer data and the first HDR image; determining second intermediate enhancement data based on the first intermediate enhancement data; and determining the first enhancement layer data based on the second intermediate enhancement data.

[0254] Optionally, in some embodiments, determining the first intermediate enhancement data based on the second basic layer data and the first HDR image includes: obtaining a target conversion result based on the second basic layer data and a target numerical conversion function; and determining the first intermediate enhancement layer data based on the first HDR image and the target conversion result.

[0255] Optionally, in some embodiments, the first intermediate enhancement layer data is the quotient of the data of the first HDR image and the target conversion result.

[0256] Optionally, in some embodiments, the first intermediate enhancement layer data is the difference between the data of the first HDR image and the target conversion result.

[0257] Optionally, in some embodiments, determining the second intermediate enhancement data based on the first intermediate enhancement data includes: determining the second intermediate enhancement data to be the first intermediate enhancement data.

[0258] Optionally, in some embodiments, determining the second intermediate enhancement data based on the first intermediate enhancement data includes: determining a target interpolation method; and downsampling the first intermediate enhancement data using the target interpolation method to obtain the second intermediate enhancement data.

[0259] Optionally, in some embodiments, determining the target interpolation method includes: determining the target interpolation method from multiple interpolation methods, wherein the difference between the recovered data obtained after upsampling the second intermediate enhancement data using the target interpolation method and the first intermediate enhancement data is smaller than the difference between the recovered data obtained after upsampling the reference intermediate enhancement data using the reference interpolation method and the first intermediate enhancement data, wherein the first reference intermediate enhancement data is the intermediate enhancement data obtained after downsampling the first intermediate enhancement data using the reference interpolation method, and the reference interpolation method is any interpolation method among the multiple interpolation methods except the target interpolation method.

[0260] Optionally, in some embodiments, the first metadata includes interpolation mode indication information, and the interpolation mode indication information is used to indicate the target interpolation mode.

[0261] Optionally, in some embodiments, the first enhancement layer data is determined based on the second intermediate enhancement data, including: mapping at least one enhancement layer characteristic value to at least one enhancement layer reference value respectively to obtain the first enhancement layer data, the first enhancement layer data including the at least one enhancement layer reference value and the values ​​of pixels in the second intermediate enhancement data other than the at least one enhancement layer characteristic value, the at least one enhancement layer characteristic value including at least one of the following values: the maximum value, the minimum value, or the intermediate value of the pixels of the intermediate basic layer data.

[0262] Optionally, in some embodiments, determining the first enhancement layer data based on the second intermediate enhancement data includes: determining a histogram of the second intermediate enhancement data; performing equalization on the histogram of the second intermediate enhancement data to obtain a balanced histogram of the second intermediate enhancement data; determining a second mapping relationship based on the histogram of the second intermediate enhancement data and the balanced histogram of the second intermediate enhancement data; converting the second intermediate enhancement data into the first enhancement layer data based on the second mapping relationship, and iteratively optimizing and adjusting the second mapping relationship so that the difference between the inverse transformed first enhancement data obtained by inverse transforming the first enhancement layer data and the second intermediate enhancement data becomes smaller.

[0263] Optionally, in some embodiments, according to the second mapping relationship, converting the second intermediate enhancement data into the first enhancement layer data includes: the second intermediate enhancement layer data and the first enhancement layer data satisfy the following relationship: enhanceAfter[i]=TMB2(R enhance[i]_1 ),

[0264] Where enhanceAfter[i] represents the value of the i-th pixel of the first enhancement layer data, R enhance[i]_1 represents the first reference value of the i-th pixel determined according to the value of the i-th pixel of the second intermediate enhancement layer data, and TMB2() represents the second mapping relationship.

[0265] Optionally, in some embodiments, according to the second mapping relationship, converting the second intermediate enhancement data into the first enhancement layer data includes: the second intermediate enhancement layer data and the first enhancement layer data satisfy the following relationship: enhanceAfter[i]=TMB2(R enhance[i]_2 )×R enhance[i]_3 ,

[0266] Wherein, enhanceAfter[i] represents the value of the i-th pixel of the first enhancement layer data, TMB2() represents the second mapping relationship, R enhance[i]_2 is the second reference value of the i-th pixel determined according to the value of the i-th pixel of the second intermediate enhancement layer data, R enhance [i]_3 is a third reference value of the i-th pixel determined according to the value of the i-th pixel of the second intermediate enhancement layer data.

[0267] FIG8 is a schematic flow chart of a method for processing a digital signal according to an embodiment of the present application.

[0268] 801. Obtain first base layer data, first enhancement layer data, and first metadata, where the first base layer data is base layer data of a first HDR object, the first enhancement layer data is enhancement layer data of the first HDR object, and the first metadata is metadata of the first HDR object.

[0269] 802, determining second basic layer data according to the first basic layer data;

[0270] 803, determining second enhancement layer data according to the first enhancement layer data;

[0271] 804. Determine a second HDR object according to the first metadata, the second base layer data, and the second enhancement layer data.

[0272] 805. Determine a third HDR object based on the second HDR object.

[0273] Optionally, in some embodiments, determining the second basic layer data based on the first basic layer data includes: determining that the second basic layer data is the same as the first basic layer data.

[0274] Optionally, in some embodiments, determining the second base layer data based on the first base layer data includes: transforming the first base layer data according to the first metadata to obtain the second base layer data.

[0275] Optionally, in some embodiments, the first basic layer data is transformed according to the first metadata to obtain the second basic layer data, including: transforming the first basic layer data according to at least one reference value carried in the first metadata to obtain the second basic layer data.

[0276] Optionally, in some embodiments, the at least one reference value includes a first reference value and a second reference value, and the first base layer data is transformed according to the at least one reference value carried in the first metadata to obtain the second base layer data, including: the first reference value, the second reference value, the first base layer data, and the second base layer data satisfy the following relationship: baseAfter[i]=base[i]×REF1+(A-base[i])×REF2,

[0277] Among them, baseAfter[i] represents the value of the i-th pixel in the second basic layer data, base[i] represents the value of the i-th pixel in the first basic layer data, REF1 represents the first reference value, REF2 represents the second reference value, and A represents the maximum value of multiple pixels included in the first basic layer data.

[0278] Optionally, in some embodiments, the at least one reference value includes a third reference value, and the first base layer data is transformed according to the at least one reference value carried in the first metadata to obtain the second base layer data, including: the third reference value, the first base layer data, and the second base layer data satisfying the following relationship: baseAfter[i]=base[i]+REF3,

[0279] Wherein, baseAfter[i] represents the value of the i-th pixel in the second basic layer data, base[i] represents the value of the i-th pixel in the first basic layer data, and REF3 represents the third reference value.

[0280] Optionally, in some embodiments, the at least one reference value includes a fourth reference value, and the first base layer data is transformed according to the at least one reference value carried in the first metadata to obtain the second base layer data, including: the fourth reference value, the first base layer data, and the second base layer data satisfying the following relationship: baseAfter[i]=REF4+A-base[i],

[0281] Among them, baseAfter[i] represents the value of the i-th pixel in the second basic layer data, base[i] represents the value of the i-th pixel in the first basic layer data, REF4 represents the fourth reference value, and A represents the maximum value of multiple pixels included in the first basic layer data.

[0282] Optionally, in some embodiments, the first basic layer data is transformed according to the first metadata to obtain the second basic layer data, including: according to at least one mapping relationship carried in the first metadata, the first basic layer data is transformed to obtain the second basic layer data.

[0283] Optionally, in some embodiments, the at least one mapping relationship includes a first mapping relationship, and the first base layer data is transformed according to the at least one mapping relationship carried in the first metadata to obtain the second base layer data, including: the first base layer data and the second base layer data satisfy the following relationship: baseAfter[i]=TMB1(R base[i]_1 ),

[0284] Wherein, baseAfter[i] represents the value of the i-th pixel in the second basic layer data, R base[i]_1 represents the first reference value of the i-th pixel determined according to the value of the i-th pixel in the first basic layer data, and TMB1() represents the first mapping relationship.

[0285] Optionally, in some embodiments, the at least one mapping relationship includes a first mapping relationship, and the transforming the first base layer data according to the at least one mapping relationship carried in the first metadata to obtain the second base layer data includes:

[0286] The first basic layer data and the second basic layer data satisfy the following relationship: baseAfter[i]=TMB1(R base[i]_2 )×R base[i]_3 ,

[0287] Wherein, baseAfter[i] represents the value of the i-th pixel in the second basic layer data, TMB1() represents the first mapping relationship, R base[i]_2is the second reference value of the i-th pixel determined according to the value of the i-th pixel in the first basic layer data, R base[i]_3 is the third reference value of the i-th pixel determined according to the value of the i-th pixel in the first basic layer data.

[0288] Optionally, in some embodiments, the at least one mapping relationship includes a second mapping relationship and a third mapping relationship, and the transforming the first base layer data according to the at least one mapping relationship carried in the first metadata to obtain the second base layer data includes:

[0289] The first basic layer data and the second basic layer data satisfy the following relationship: baseAfter[i]=TMB3(TMB2(R base[i]_4 )),

[0290] Wherein, baseAfter[i] represents the value of the i-th pixel in the second basic layer data, R base[i]_4 represents the fourth reference value of the i-th pixel determined according to the value of the i-th pixel in the first basic layer data, TMB2() represents the second mapping relationship, and TMB3() represents the third mapping relationship.

[0291] Optionally, in some embodiments, the at least one mapping relationship includes a second mapping relationship and a third mapping relationship, and the transforming the first base layer data according to the at least one mapping relationship carried in the first metadata to obtain the second base layer data includes:

[0292] The first basic layer data and the second basic layer data satisfy the following relationship: baseAfter[i]=TMB3(R base[i]_6 )×TMB2(R base[i]_5 ),

[0293] Wherein, baseAfter[i] represents the value of the i-th pixel in the second basic layer data, R base[i]_5 represents the fifth reference value of the i-th pixel determined according to the value of the i-th pixel in the first basic layer data, R base[i]_6 represents the sixth reference value of the i-th pixel determined according to the value of the i-th pixel in the first basic layer data, TMB2() represents the second mapping relationship, and TMB3() represents the third mapping relationship.

[0294] Optionally, in some embodiments, the third mapping relationship is a global tone mapping function, or the third mapping relationship is a local tone mapping function.

[0295] Optionally, in some embodiments, the at least one mapping relationship includes at least one functional relationship.

[0296] Optionally, in some embodiments, the first basic layer data is transformed according to the first metadata to obtain the second basic layer data, including: transforming the first basic layer data according to at least one filter information carried in the first metadata to obtain the second basic layer data.

[0297] Optionally, in some embodiments, the at least one filter information includes first filter information, and the transforming the first base layer data according to the at least one filter information carried in the first metadata to obtain the second base layer data includes:

[0298] The first base layer data is filtered using a first filter to obtain first reference filtered data, wherein parameters of the first filter are indicated by the first filter information, or a type of the first filter and parameters of the first filter are indicated by the first filter information; N base layer filtered data are determined based on the first reference filtered data and the first base layer data, the second base layer data including the first reference filtered data and the N base layer filtered data, where N is a positive integer greater than or equal to 1, wherein,

[0299] The first reference filtered data and the first base layer data satisfy the following relationship:

[0300] baseAfter0[i]=F1[base[i]]; wherein baseAfter0[i] represents the value of pixel i in the first reference filtered data, base[i] represents the value of pixel i in the first basic layer data, and F1[] represents the first filter; the nth basic layer filtered data in the N basic layer filtered data and the first basic layer data satisfy the following relationship: baseAfter n [i]=base[i]-baseAfter n-1 [i],

[0301] where baseAfter n [i] represents the value of pixel i in the nth basic layer filter data, n=1,…,N.

[0302] Optionally, in some embodiments, transforming the first base layer data according to the first metadata to obtain the second base layer data includes:

[0303] The first base layer data is transformed according to the at least one filter information and the at least one mapping relationship carried in the first metadata to obtain the second base layer data.

[0304] Optionally, in some embodiments, the at least one filter information includes second filter information, the at least one mapping relationship includes a fourth mapping relationship and a fifth mapping relationship, and the first basic layer data is transformed according to the at least one filter information and the at least one mapping relationship carried in the first metadata to obtain the second basic layer data, including: filtering the first basic layer data using a second filter to obtain second reference filtered data, wherein the parameters of the second filter are indicated by the second filter information, or the type of the second filter and the parameters of the second filter are indicated by the second filter information; transforming the second reference filtered data according to the fourth mapping relationship and the fifth mapping relationship to obtain the second basic layer data.

[0305] Optionally, in some embodiments, transforming the second reference filtered data according to the fourth mapping relationship and the fifth mapping relationship to obtain the second base layer data includes:

[0306] The second reference filter and the second base layer data satisfy the following relationship: baseAfter[i]=TMB5(TMB4(baseMid1[i])),

[0307] Among them, baseAfter[i] represents the value of the i-th pixel in the second basic layer data, baseMid1[i] represents the value of pixel i in the second reference filtered data, TMB4() represents the fourth mapping relationship, and TMB5() represents the fifth mapping relationship.

[0308] Optionally, in some embodiments, transforming the second reference filtered data according to the fourth mapping relationship and the fifth mapping relationship to obtain the second base layer data includes:

[0309] The second reference filter and the second base layer data satisfy the following relationship: baseAfter[i]=TMB3(R base[i]_7 )×TMB2(baseMid1[i]),

[0310] Wherein, baseAfter[i] represents the value of the i-th pixel in the second basic layer data, baseMid1[i] represents the value of the pixel i in the second reference filtered data, TMB4() represents the fourth mapping relationship, TMB5() represents the fifth mapping relationship, R base[i]_7 Represents the seventh reference value of the i-th pixel determined according to the value of the i-th pixel in the first basic layer data.

[0311] Optionally, in some embodiments, determining the second layer data based on the first enhancement layer data includes: obtaining extended enhancement layer data based on the first enhancement layer data; or upsampling the first enhancement layer data to obtain extended enhancement layer data; and determining the second enhancement layer data based on the extended enhancement layer data.

[0312] Optionally, in some embodiments, the first enhancement layer data is upsampled to obtain extended enhancement layer data, including: obtaining interpolation method indication information obtained from the first metadata or preset information; and upsampling the first enhancement layer data using the interpolation method indicated by the interpolation method indication information to obtain the extended enhancement layer data.

[0313] Optionally, in some embodiments, upsampling the first enhancement layer data to obtain extended enhancement layer data includes: upsampling the first enhancement layer data using a preset interpolation method to obtain the extended enhancement layer data.

[0314] Optionally, in some embodiments, the first enhancement layer data is upsampled to obtain extended enhancement layer data, including: determining K reference pixels included in the first basic layer data, the K reference pixels are respectively located at K first positions, the first enhancement layer data has no pixels at the K first positions, and K is a positive integer greater than or equal to 1; determining K groups of adjacent pixels, wherein the K groups of adjacent pixels correspond one-to-one to the K reference pixels, the kth group of adjacent pixels in the K groups of adjacent pixels includes at least one pixel, the at least one pixel included in the kth group of adjacent pixels is adjacent to the kth reference pixel in the K reference pixels, each pixel in the at least one pixel included in the kth group of adjacent pixels has a corresponding enhancement pixel in the first enhancement layer data, the coordinates of each pixel are the same as the coordinates of the enhancement pixel of each pixel, k=1,…,K; determining the value of the kth reference enhancement pixel in the K reference enhancement pixels based on the value of at least one pixel included in the kth group of adjacent pixels, the K reference added pixels are located at the K first positions of the extended enhancement layer data.

[0315] Optionally, in some embodiments, determining the value of the kth reference enhanced pixel among the K reference enhanced pixels according to the value of at least one pixel included in the kth group of adjacent pixels includes:

[0316] Determine a first adjacent pixel, where a difference between a value of the k-th reference pixel and a value of the first adjacent pixel is smaller than a difference between a value of the k-th reference pixel and a value of any adjacent pixel in the k-th group of adjacent pixels other than the first adjacent pixel; and determine that a value of an enhanced pixel corresponding to the first adjacent pixel is the value of the lower k reference enhanced pixels.

[0317] Optionally, in some embodiments, determining the value of the kth reference enhanced pixel among the K reference enhanced pixels according to the value of at least one pixel included in the kth group of adjacent pixels includes:

[0318] Determine at least two second-adjacent pixels, where a difference between a value of the kth reference pixel and a value of any second-adjacent pixel among the at least two second-adjacent pixels is smaller than a difference between the value of the kth reference pixel and a value of any adjacent pixel in the kth group of adjacent pixels other than the at least two second-adjacent pixels; determine at least two enhanced pixels, where the at least two enhanced pixels have a one-to-one correspondence with the at least two second-adjacent pixels; and determine a value of the kth reference enhanced pixel based on the values ​​of the at least two enhanced pixels.

[0319] Optionally, in some embodiments, determining the second enhancement layer data according to the extended enhancement layer data includes:

[0320] It is determined that the second enhancement layer data is identical to the extended enhancement layer data.

[0321] Optionally, in some embodiments, determining the second enhancement layer data according to the extended enhancement layer data includes:

[0322] The extended enhancement layer data is transformed according to the first metadata to obtain the second enhancement layer data.

[0323] Optionally, in some embodiments, transforming the extended enhancement layer data according to the first metadata to obtain the second enhancement layer data includes:

[0324] The extended enhancement layer data is transformed according to at least one reference value carried in the first metadata to obtain the second enhancement layer data.

[0325] Optionally, in some embodiments, the at least one reference value includes a fifth reference value and a sixth reference value, and the transforming the extended enhancement layer data according to the at least one reference value carried in the first metadata to obtain the second enhancement layer data includes:

[0326] The fifth reference value, the sixth reference value, the extended enhancement layer data, and the second enhancement layer data satisfy the following relationship: enhanceAfter[i]=enhance[i]×REF5+(A-enhance[i])×REF6,

[0327] Wherein, enhanceAfter[i] represents the value of the i-th pixel in the second enhancement layer data, enhance[i] represents the value of the i-th pixel in the extended enhancement layer data, REF5 represents the fifth reference value, REF6 represents the sixth reference value, and A represents the maximum value of multiple pixels stored in the extended enhancement layer data.

[0328] Optionally, in some embodiments, the at least one reference value includes a seventh reference value, and the transforming the extended enhancement layer data according to the at least one reference value carried in the first metadata to obtain the second enhancement layer data includes:

[0329] The seventh reference value, the extended enhancement layer data, and the second enhancement layer data satisfy the following relationship: enhanceAfter[i]=enhance[i]+REF7,

[0330] Wherein, enhanceAfter[i] represents the value of the i-th pixel in the second enhancement layer data, enhance[i] represents the value of the i-th pixel in the extended enhancement layer data, and REF7 represents the seventh reference value.

[0331] Optionally, in some embodiments, the at least one reference value includes an eighth reference value, and the transforming the extended enhancement layer data according to the at least one reference value carried in the first metadata to obtain the second enhancement layer data includes:

[0332] The eighth reference value, the extended enhancement layer data, and the second enhancement layer data satisfy the following relationship: enhanceAfter[i]=REF8+A-enhance[i],

[0333] Wherein, enhanceAfter[i] represents the value of the i-th pixel in the second enhancement layer data, enhance[i] represents the value of the i-th pixel in the extended enhancement layer data, REF8 represents the eighth reference value, and A represents the maximum value of multiple pixels stored in the extended enhancement layer data.

[0334] Optionally, in some embodiments, transforming the extended enhancement layer data according to the first metadata to obtain the second enhancement layer data includes:

[0335] The extended enhancement layer data is transformed according to at least one mapping relationship carried in the first metadata to obtain the second enhancement layer data.

[0336] Optionally, in some embodiments, the at least one mapping relationship includes a sixth mapping relationship, and the transforming of the extended enhancement layer data according to the at least one mapping relationship carried in the first metadata to obtain the second enhancement layer data includes:

[0337] The extended enhancement layer data and the second enhancement layer data satisfy the following relationship: enhanceAfter[i]=TMB6(R enhance[i]_1 ),

[0338] Wherein, enhanceAfter[i] represents the value of the i-th pixel in the second enhancement layer data, R enhance[i]_1 represents the first reference value of the i-th pixel determined according to the value of the i-th pixel in the extended enhancement layer data, and TMB6() represents the sixth mapping relationship.

[0339] Optionally, in some embodiments, the at least one mapping relationship includes a sixth mapping relationship, and the transforming of the extended enhancement layer data according to the at least one mapping relationship carried in the first metadata to obtain the second enhancement layer data includes:

[0340] The extended enhancement layer data and the second enhancement layer data satisfy the following relationship: enhanceAfter[i]=TMB6(R enhance[i]_2 )×R enhance[i]_3 ,

[0341] Wherein, enhanceAfter[i] represents the value of the i-th pixel in the second enhancement layer data, TMB6() represents the sixth mapping relationship, R enhance[i]_2 is the second reference value of the i-th pixel determined according to the value of the i-th pixel in the extended enhancement layer data, R enhance[i]_3 is the third reference value of the i-th pixel determined according to the value of the i-th pixel in the extended enhancement layer data.

[0342] Optionally, in some embodiments, the at least one mapping relationship includes a seventh mapping relationship and an eighth mapping relationship, and the transforming the extended enhancement layer data according to the at least one mapping relationship carried in the first metadata to obtain the second enhancement layer data includes:

[0343] The extended enhancement layer data and the second enhancement layer data satisfy the following relationship: enhanceAfter[i]=TMB7(TMB8(R enhance[i]_4 )),

[0344] Wherein, enhanceAfter[i] represents the value of the i-th pixel in the second enhancement layer data, R enhance[i]_4 represents the fourth reference value of the i-th pixel determined according to the value of the i-th pixel in the extended enhancement layer data, TMB7() represents the seventh mapping relationship, and TMB8() represents the eighth mapping relationship.

[0345] Optionally, in some embodiments, the at least one mapping relationship includes a seventh mapping relationship and an eighth mapping relationship, and the transforming the extended enhancement layer data according to the at least one mapping relationship carried in the first metadata to obtain the second enhancement layer data includes:

[0346] The extended enhancement layer data and the second enhancement layer data satisfy the following relationship:

[0347] The extended enhancement layer data and the second enhancement layer data satisfy the following relationship: enhanceAfter[i]=TMB7(R enhance[i]_6 )×TMB8(R enhance[i]_5 ),

[0348] Wherein, enhanceAfter[i] represents the value of the i-th pixel in the second enhancement layer data, R enhance[i]_5 represents the fifth reference value of the i-th pixel determined according to the value of the i-th pixel in the extended enhancement layer data, R enhance[i]_6 represents the sixth reference value of the i-th pixel determined according to the value of the i-th pixel in the extended enhancement layer data, TMB7() represents the seventh mapping relationship, and TMB8() represents the eighth mapping relationship.

[0349] Optionally, in some embodiments, the seventh mapping relationship is a global tone mapping function, or the seventh mapping relationship is a local tone mapping function.

[0350] Optionally, in some embodiments, the at least one mapping relationship includes at least one functional relationship.

[0351] Optionally, in some embodiments, transforming the extended enhancement layer data according to the first metadata to obtain the second enhancement layer data includes: transforming the extended enhancement layer data according to at least one filter information carried in the first metadata to obtain the second enhancement layer data.

[0352] Optionally, in some embodiments, the at least one filter information includes third filter information, and the transforming the first enhancement layer data according to the at least one filter information carried in the first metadata to obtain the second enhancement layer data includes: filtering the first enhancement layer data using a third filter to obtain third reference filter data, wherein the parameters of the third filter are indicated by the third filter information, or the type of the third filter and the parameters of the third filter are indicated by the third filter information; determining N enhancement layer filter data according to the third reference filter data and the first enhancement layer data, the second enhancement layer data including the third reference filter data and the N enhancement layer filter data, where N is a positive integer greater than or equal to 1, wherein,

[0353] The third reference filtered data and the first enhancement layer data satisfy the following relationship:

[0354] enhanceAfter0[i]=F1[enhance[i]]; wherein enhanceAfter0[i] represents the value of pixel i in the third reference filtered data, enhance[i] represents the value of pixel i in the first enhanced layer data, and F1[] represents the third filter; the nth enhanced layer filtered data in the N enhanced layer filtered data and the first enhanced layer data satisfy the following relationship: enhanceAfter n [i]=enhance[i]-enhanceAfter n-1 [i],

[0355] where enhanceAfter n [i] represents the value of pixel i in the nth enhancement layer filter data, n=1,…,N.

[0356] Optionally, in some embodiments, the first enhancement layer data is transformed according to the first metadata to obtain the second enhancement layer data, including: according to at least one filter information and at least one mapping relationship carried in the first metadata, the first enhancement layer data is transformed to obtain the second enhancement layer data.

[0357] Optionally, in some embodiments, the at least one filter information includes fourth filter information, the at least one mapping relationship includes a ninth mapping relationship and a tenth mapping relationship, and the first enhancement layer data is transformed according to the at least one filter information and the at least one mapping relationship carried in the first metadata to obtain the second enhancement layer data, including: filtering the first enhancement layer data using a fourth filter to obtain fourth reference filtered data, wherein the parameters of the fourth filter are indicated by the fourth filter information, or the type of the fourth filter and the parameters of the fourth filter are indicated by the fourth filter information; transforming the fourth reference filtered data according to the ninth mapping relationship and the tenth mapping relationship to obtain the second enhancement layer data.

[0358] Optionally, in some embodiments, transforming the fourth reference filtered data according to the fourth mapping relationship and the fifth mapping relationship to obtain the second enhancement layer data includes:

[0359] The second reference filter and the second enhancement layer data satisfy the following relationship: enhanceAfter[i]=TMB 10 (TMB9(enhanceMid1[i])),

[0360] Wherein, enhanceAfter[i] represents the value of the i-th pixel in the second enhancement layer data, enhanceMid1[i] represents the value of the pixel i in the fourth reference filtered data, TMB9() represents the ninth mapping relationship, and TMB 10 () represents the tenth mapping relationship.

[0361] Optionally, in some embodiments, transforming the fourth reference filtered data according to the fourth mapping relationship and the fifth mapping relationship to obtain the second enhancement layer data includes:

[0362] The second reference filter and the second enhancement layer data satisfy the following relationship: enhanceAfter[i]=TMB9(R enhance[i]_7 )×TMB 10 (enhanceMid1[i]),

[0363] Wherein, enhanceAfter[i] represents the value of the i-th pixel in the second enhancement layer data, enhanceMid1[i] represents the value of the pixel i in the fourth reference filtered data, TMB9() represents the ninth mapping relationship, and TMB 10 () represents the tenth mapping relationship, R enhance[i]_7 represents the seventh reference value of the i-th pixel determined according to the value of the i-th pixel in the first enhancement layer data.

[0364] Optionally, in some embodiments, determining the second HDR object according to the first metadata or preset information, the second base layer data, and the second enhancement layer data includes:

[0365] The second HDR object is determined according to the conversion function obtained from the first metadata or preset information, the second base layer data, and the second enhancement layer data.

[0366] Optionally, in some embodiments, determining the second HDR object according to the conversion function obtained from the first metadata or preset information, the second base layer data, and the second enhancement layer data includes:

[0367] The second base layer data, the second enhancement layer data and the second HDR object satisfy the following relationship: recHDR[i]=baseAfter[i]×f(enhanceAfter[i]),

[0368] Among them, recHDR[i] represents the value of the i-th pixel in the second HDR object, baseAfter[i] represents the value of the i-th pixel in the second base layer data, enhanceAfter[i] represents the value of the i-th pixel in the second enhancement layer data, and f() represents the conversion function.

[0369] Optionally, in some embodiments, determining the second HDR object according to the conversion function obtained from the first metadata or preset information, the second base layer data, and the second enhancement layer data includes: the second base layer data, the second enhancement layer data, and the second HDR object satisfy the following relationship: recHDR[i]=baseAfter[i]+f(enhanceAfter[i]),

[0370] Among them, recHDR[i] represents the value of the i-th pixel in the second HDR object, baseAfter[i] represents the value of the i-th pixel in the second base layer data, enhanceAfter[i] represents the value of the i-th pixel in the second enhancement layer data, and f() represents the conversion function.

[0371] Optionally, in some embodiments, determining the second HDR object according to the conversion function obtained from the first metadata or preset information, the second base layer data, and the second enhancement layer data includes: determining the second HDR object according to the first conversion function and the second conversion function obtained from the first metadata or preset information, the second base layer data, and the second enhancement layer data, wherein the second base layer data includes first reference filter data and N base layer filter data, the second enhancement layer data includes third reference filter data and N enhancement layer filter data, and the first conversion function, the second conversion function, the second base layer data, the second enhancement layer data, and the second HDR object satisfy the following relationship:

[0372] Among them A j is the jth conversion parameter among the N+1 conversion parameters, g() represents the first conversion function, f() represents the second conversion function, baseAfter0[i] represents the value of pixel i in the first reference filtered data, baseAfter n [i] represents the value of pixel i in the nth base layer filter data, enhanceAfter0[i] represents the value of pixel i in the third reference filter data, enhanceAfter n [i] represents the value of pixel i in the nth enhancement layer filter data, n=1,…,N.

[0373] Optionally, in some embodiments, determining the second HDR object according to the conversion function obtained from the first metadata or preset information, the second base layer data, and the second enhancement layer data includes: determining the second HDR object according to a third conversion function obtained from the first metadata or preset information, the second base layer data, and the second enhancement layer data, wherein the second base layer data includes first reference filter data and base layer filter data, the second enhancement layer data includes third reference filter data and enhancement layer filter data, and the third conversion function, the second conversion function, the second base layer data, the second enhancement layer data, and the second HDR object satisfy the following relationship: recHDR[i]=A×baseAfter0[i]+A1×f1(enhanceAfter0[i])+B×baseAfter1[i]

[0374] Or recHDR[i]=A×baseAfter0[i]×f1(enhanceAfter0[i])+B×baseAfter1[i]

[0375] Or recHDR[i]=A×baseAfter0[i]×f1(enhanceAfter0[i])×f1(baseAfter1[i])

[0376] Wherein A, A1 and B are conversion parameters, f1() represents the third conversion function, baseAfter0[i] represents the value of pixel i in the first reference filtered data, baseAfter1[i] represents the value of pixel i in the base layer filtered data, enhanceAfter0[i] represents the value of pixel i in the third reference filtered data, and enhanceAfter1[i] represents the value of pixel i in the enhanced layer filtered data.

[0377] Optionally, in some embodiments, determining the second HDR object according to the conversion function obtained from the first metadata or preset information, the second base layer data, and the second enhancement layer data includes:

[0378] The second HDR object is determined according to the three conversion functions obtained from the first metadata or preset information, the second base layer data, and the second enhancement layer data, wherein the second base layer data includes first reference filter data and N base layer filter data, the second enhancement layer data includes third reference filter data and N enhancement layer filter data, N is equal to 2, and the three conversion functions, the second base layer data, and the second enhancement layer data satisfy the following relationship: recHDR[i]=A×baseAfter0[i]×f1(enhanceAfter0[i])+B×baseAfter1[i]×f2(enhanceAfter1[i])+C×baseAfter2[i]×f3(enhanceAfter2[i])

[0379] Where A, B and C are three conversion parameters, f1(), f2() and f3() represent the three conversion functions respectively, baseAfter0[i] represents the value of pixel i in the first reference filtered data, baseAfter n [i] represents the value of pixel i in the nth base layer filter data, enhanceAfter0[i] represents the value of pixel i in the third reference filter data, enhanceAfter n [i] represents the value of pixel i in the nth enhancement layer filter data, n=1,…,2.

[0380] Optionally, in some embodiments, determining the second HDR object according to the conversion function obtained from the first metadata or preset information, the second base layer data, and the second enhancement layer data includes: determining the second HDR object according to a fourth conversion function obtained from the first metadata or preset information, the second base layer data, and the second enhancement layer data, wherein the second base layer data includes first reference filter data and N base layer filter data, the second enhancement layer data includes third reference filter data and N enhancement layer filter data, N is equal to 2, and the fourth conversion function, the second base layer data, and the second enhancement layer data satisfy the following relationship: recHDR[i]=A×baseAfter0[i]+A1×f1(enhanceAfter0[i])+B×baseAfter1[i]+B1×f1(enhanceAfter1[i])+ C×baseAfter2[i]+C1×f1(enhanceAfter2[i])

[0381] Wherein A, A1, B, B1, C and C1 are six conversion parameters, f1() represents the fourth conversion function, baseAfter0[i] represents the value of pixel i in the first reference filtered data, baseAfter n[i] represents the value of pixel i in the nth base layer filter data, enhanceAfter0[i] represents the value of pixel i in the third reference filter data, enhanceAfter n [i] represents the value of pixel i in the nth enhancement layer filter data, n=1,…,2.

[0382] Optionally, in some embodiments, determining a third HDR object based on the second HDR object includes: determining that the third HDR object is the same as the second HDR object.

[0383] Optionally, in some embodiments, determining a third HDR object based on the second HDR object includes: performing the same transformation on the second HDR object according to the first metadata to obtain the third HDR object.

[0384] Optionally, in some embodiments, the first basic layer data is transformed according to the first metadata to obtain the second basic layer data, including: according to at least one reference value carried in the first metadata, the second HDR object is transformed in the same manner to obtain the third HDR object.

[0385] Optionally, in some embodiments, the at least one reference value includes a ninth reference value and a tenth reference value, and the second HDR object is transformed in the same manner according to the at least one reference value carried in the first metadata to obtain the third HDR object, including: the ninth reference value, the tenth reference value, the third HDR object and the second HDR object satisfying the following relationship: recHDRAfter[i]=recHDR[i]×REF9+(A-recHDR[i])×REF 10 ,

[0386] Wherein, recHDRAfter[i] represents the value of the i-th pixel in the third HDR object, recHDR[i] represents the value of the i-th pixel in the second HDR object, REF9 represents the ninth reference value, REF 10 represents the tenth reference value, and A represents the maximum value of multiple pixels included in the first basic layer data.

[0387] Optionally, in some embodiments, the at least one reference value includes an eleventh reference value, and the second HDR object is transformed in the same manner according to the at least one reference value carried in the first metadata to obtain the third HDR object, including: the eleventh reference value, the third HDR object and the second HDR object satisfying the following relationship: recHDRAfter[i]=recHDR[i]+REF 11 ,

[0388] Among them, recHDRAfter[i] represents the value of the i-th pixel in the third HDR object, recHDR[i] represents the value of the i-th pixel in the second HDR object, REF 11 represents the eleventh reference value.

[0389] Optionally, in some embodiments, the at least one reference value includes a twelfth reference value, and the second HDR object is transformed in the same manner according to the at least one reference value carried in the first metadata to obtain the third HDR object, including: the twelfth reference value, the third HDR object and the second HDR object satisfying the following relationship: recHDRAfter[i]=REF 12 +A-recHDR[i],

[0390] Among them, recHDRAfter[i] represents the value of the i-th pixel in the third HDR object, recHDR[i] represents the value of the i-th pixel in the second HDR object, REF 12 Indicates the twelfth reference value.

[0391] Figure 9A is a flow chart of a signal processing method provided in an embodiment of the present application, including: step 0901, obtaining a first base image, a first gain map and metadata; step 0902, processing the first base image according to the metadata to obtain a second base image; step 0903, obtaining a target image according to the second base image and the first gain map.

[0392] Step 0901, obtaining a first base image, a first gain map, and metadata, can be implemented by the following scheme:

[0393] A. Obtain the first basic image from the bitstream. This process involves decoding the received bitstream through any decoder (HEVC, JPEG, ProRes, HEIF) to obtain image data. The image data obtained in this process includes image data in any color space format such as RGB or YUV. It should be noted that the present invention does not limit the bitstream format. In terms of color space, it can be YUV or RGB. In terms of data bit width, it can be 8 bits, 10 bits, or 12 bits, etc. The decoder can be HEVC, JPEG, ProRes, HEIF, etc.

[0394] B. Get metadata from the bitstream. Metadata can be obtained from the SEI of HEVC or VVC, user-defined NAL units or reserved packet units, or JFIF-encapsulated APP extension information, or MP4-encapsulated data segments. Metadata can also be obtained from other file locations in the bitstream, such as the position after the EOI (end of image) of a complete JPEG file. It should be noted that metadata mainly includes data such as the source data format, region division information, region traversal order information, or image features and curve parameters, as well as one or more metadata information units. The metadata information unit contains data such as coordinate information, image features and curve parameters.

[0395] C. Obtain enhancement data (i.e., the first gain map). The enhancement data may be obtained from the SEI of HEVC or VVC, user-defined NAL units, or reserved packet units, or from APP extension information encapsulated in JFIF, or from data segments encapsulated in MP4. The first gain map may also be obtained from other file locations indicated in the metadata, such as the location after the EOI (end of image) of a complete JPEG file. The enhancement data may be: one pixel value corresponds to one enhancement data, or multiple pixel values ​​correspond to one enhancement data, where the enhancement data may contain multiple values, and the number of values ​​contained in the enhancement data corresponding to each pixel is the same.

[0396] Step 0902, based on the metadata, processes the first base image to obtain a second base image, including: Step A, if the resolution of the base image is different from the resolution of the enhanced data, or different from the preset target resolution, up-sampling / down-sampling the base image according to the metadata or preset information; Step B, processing the base layer Base according to the base layer remapping information contained in the metadata to obtain baseAfter. It should be noted that there are two orders for executing this step of the present invention: the first is to execute step A first, then step B, which is more convenient for system implementation; the second is to execute step B first, then step A, which makes the overall numerical processing process more reasonable. In the second implementation method, that is, when step B is executed first and then step A, step B processes the first base image to obtain baseAfter, and then up-sampling / down-sampling the baseAfter obtained in step B is performed in step A.

[0397] Step A, if the resolution of the base image is different from the resolution of the enhanced data, or different from the preset target resolution, upsampling / downsampling the base image according to metadata or preset information, includes:

[0398] 1) The metadata or preset information includes phase information PhaseAx and PhaseAy related to upsampling / downsampling of the basic image.

[0399] Phase information refers to position information of a current image pixel relative to an up / down sampled image pixel, or position information of an up / down sampled image relative to a current image pixel.

[0400] FIG16 is a schematic diagram of the positional relationship between the current image pixels and the sampled image pixels provided in an embodiment of the present application. As shown in FIG16 , the boxes in the figure represent the current image pixels, and the circles represent the up / down sampled image pixels. In case 1, the current image pixels are sampled at intervals to obtain downsampled image pixels, and the downsampled image pixels are located at the same position as the sampled image pixels. In case 2, the downsampled image pixels are obtained by interpolation based on multiple current image pixels, and the downsampled image pixels are usually located at different positions than the sampled image pixels. The pixel here can refer to either the brightness of the pixel or the chrominance of the pixel.

[0401] Because the pixels of the current image and the up / downsampled image may have different correspondences at different locations, the position difference information of the pixels of the current image and the up / downsampled image at a preset position can be used as a description of the phase information. The preset position can be a preset image position such as the upper left corner pixel, the upper right corner pixel, the lower left corner pixel, or the lower right corner pixel position of the image, or it can be transmitted in the form of image pixel coordinates. For example, if the upper left corner pixel is (y=0, x=0), then the pixel to its right is (y=0, x=1), where x and y are the horizontal and vertical coordinates of the pixel.

[0402] A preset manner may be adopted, for example, the position information of pixels at preset positions such as the upper left corner pixel of the current image and the up / down sampled image are the same.

[0403] 2) The metadata or preset information includes phase information PhaseBx and PhaseBy related to up / down sampling of channels other than the brightness channel.

[0404] One of the situations is: the image transmitted is in YUV color space, and the YUV format is not 4:4:4, but may be 4:2:2, 4:2:0 or 4:1:1. These formats all downsample the UV, and the position relationship between the UV chroma pixels and the Y is shown in the following figure:

[0405] Figure 17 is a schematic diagram of the phase information of luma and chroma pixels according to an embodiment of the present application. In Figure 17, the horizontal position difference is equivalent to the chroma pixel's horizontal coordinate minus the luma pixel's horizontal coordinate. The vertical position difference is equivalent to the chroma pixel's vertical coordinate minus the luma pixel's vertical coordinate.

[0406] A preset method may be used, for example, when adopting 4:2:2, 4:2:0 or 4:1:1, where the brightness is twice the chromaticity, the preset chromaticity position is located at the center of two or four brightness pixel positions.

[0407] 3) Metadata or preset information contains information about up / down sampling algorithms

[0408] Option 1: Pass the index and select a preferred interpolation method from a preset set. The preset set can be OpenGL's GL_NEAREST (nearest neighbor) and GL_LINEAR (nearest neighbor) texture interpolation methods; Vulkan's NEAREST (nearest neighbor) or LINEAR (nearest neighbor) texture interpolation methods; it can also pass multiple filter sets, or include multiple directional interpolation methods, or bicubic splines; or it can be all or a subset of the above methods.

[0409] Option 2: Pass the algorithm description and select a default texture interpolation method from OpenGL, Vulkan, or Matl. This can be GL_NEAREST (nearest neighbor) or GL_LINEAR (nearest neighbor) from OpenGL, or NEAREST (nearest neighbor) or LINEAR (nearest neighbor) from Vulkan. Alternatively, a preset filter (similar to the eight-tap filter in H.265's MC module or the eight-tap filter used for upsampling / downsampling between SHVC layers), or a combination of directional interpolation methods, such as bicubic splines.

[0410] Option 3: Use a preset algorithm, selecting a default method from OpenGL, Vulkan, or matel texture interpolation methods. This can be GL_NEAREST (nearest neighbor) or GL_LINEAR (nearest neighbor) for OpenGL texture interpolation, or NEAREST (nearest neighbor) or LINEAR (nearest neighbor) for Vulkan texture interpolation. Alternatively, use a preset filter (similar to the eight-tap filter in H.265's MC module or the eight-tap filter used for upsampling / downsampling between SHVC layers), or use a variety of directional interpolation methods, such as bicubic splines.

[0411] 4) Perform up / down sampling on the basic image base according to the up / down sampling related phase information, up / down sampling related algorithm information, and other related information to obtain the up / down sampled basic image base. An example solution is:

[0412] Brightness: Base[i]=filter1[PhaseAy][PhaseAx][0]*base[iN]+…+filter[PhaseAy][PhaseAx][2N+1]*base[i+N],

[0413] Chroma: Base[i] = filter2[PhaseBy][PhaseBx][0]*base[iM] + … + filter[PhaseBy][PhaseBx][2M+1]*base[i+M], where filter1 is a 2N+1 order luma upsampling filter and filter2 is a 2M+1 order chroma upsampling filter.

[0414] The present invention does not describe in detail how to perform up / down sampling. Up / down sampling can be performed according to a specified existing algorithm and related parameters. The present invention does not modify the up / down sampling algorithm.

[0415] It should be noted that if the resolution of the base image is different from the resolution of the enhanced data, or different from the preset target resolution, the up / down sampling of the base image can be performed without up / down sampling, only up / down sampling of the base image, or up / down sampling of the base image and the enhanced data at the same time. Step B, processing the base layer Base according to the base layer remapping information contained in the metadata to obtain baseAfter, includes:

[0416] Solution 1: baseAfter[i]=Base[i]

[0417] Solution 2: The metadata contains the upper limit THL and / or lower limit THH of the base layer data, and uses baseAfter[i]=base[i]*THH+(A-base[i])*THL, where A is the maximum value stored in base. When base is normalized to between 0 and 1.0, A is 1.0.

[0418] Solution 3: The metadata contains the lower limit THL of the basic layer data, and baseAfter[i]=base[i]+THL is used.

[0419] Solution 4: The metadata contains the upper THH of the base layer data, using baseAfter[i]=THH+A-base[i], where A is the maximum value stored in base. When base is normalized to between 0 and 1.0, A is 1.0.

[0420] Solution 5: The metadata contains the mapping relationship parameters, and TMB() is obtained based on the parameters, baseAfter[i] = TMB(base[i])

[0421] Solution 6: The metadata contains the parameters of the mapping relationship, and TMB() is obtained based on the parameters, baseAfter[i]=TMB()*base[i]

[0422] Solution 7: The metadata contains the mapping parameters, the upper limit THL, and / or lower limit THH of the base layer data. TMB() is obtained based on the parameters. baseAfter[i] = TMB(base[i]*THH+(A-base[i])*THL)

[0423] Solution 8: The metadata contains the mapping parameters, the upper limit THL, and / or lower limit THH of the base layer data. TMB() is obtained based on the parameters. baseAfter[i] = TMB() * (base[i] * THH + (A - base[i]) * THL)

[0424] Solution 9: The metadata contains the mapping parameters and the lower limit THL of the base layer data. According to the parameters, TMB() is obtained, baseAfter[i] = TMB(base[i] + THL)

[0425] Solution 10: The metadata contains the mapping parameters and the lower limit THL of the base layer data. According to the parameters, TMB() is obtained, and baseAfter[i] = TMB() * (base[i] + THL)

[0426] Solution 11: The data contains parameters of the mapping relationship. According to the parameters, TMB() and TMB1() are obtained. TMB() can be a global or local tone mapping function. TMB1 can be a linear, spline, or piecewise curve. baseAfter[i] = TMB(TMB1(base[i]))

[0427] Solution 12: The data contains parameters of the mapping relationship. According to the parameters, TMB() and TMB1() are obtained. TMB() can be a global or local tone mapping function. TMB1 can be a linear, spline, or piecewise curve. baseAfter[i] = TMB() * TMB1(base[i])

[0428] Solution 13: The metadata contains parameters of the mapping relationship. TMB(), TMB1(), and F[] are obtained based on the parameters. TMB() can be a global or local tone mapping function, TMB1 can be in the form of linear, spline, piecewise curve, etc., and F[] is a spatial filter parameter or other image smoothing function. F[] is used to obtain baseMid1[i] and baseAfter2[i]: baseMid1[i] = F[base[i]] or ∑F[n]*base[i+n], and baseAfter2[i] = base[i]-baseMid1[i]. There are various forms of filtering methods, such as bilateral filtering, interpolation filtering, etc., which are not limited by the present invention. Then, baseAfter[i] = TMB()*TMB1(baseMid1[i]) or baseAfter[i] = TMB(TMB1(baseMid1[i])).

[0429] Solution fourteen, the metadata contains parameters of the mapping relationship, and F[] is obtained according to the parameters, where F[] is the spatial filter parameter or other image smoothing processing function, and F[] is used to obtain baseAfter[i] and baseAfter2[i]: baseAfter[i] = F[base[i]] or ∑F[n]*base[i+n], and baseAfter2[i] = base[i]-baseMid1[i]. There are various forms of filtering methods, such as bilateral filtering, interpolation filtering, etc., which are not limited by the present invention.

[0430] Solution 15: The metadata includes parameters for filtering or other image processing. Based on these parameters, a specific function F() is obtained. F() acts on the basic layer to obtain multiple data after processing: baseAfter[i]=F(base[i])=[baseAfter1[i],baseAfter2[i],baseAfter3[i]……], and there are various forms of filtering methods, such as bilateral filtering, interpolation filtering, etc., which are not limited in the present invention (full modification).

[0431] The mapping relationship has various forms, such as Sigmoid, cubic spline, gamma, straight line, etc. or their inverse function forms, which are not limited in the present invention. The following curves can be used:

[0432] It should be noted that L and L' can be normalized optical signals or electrical signals, which is not limited in the present invention.

[0433] It should be noted that the present invention does not limit the format of metadata. Metadata can include histogram information and tone-mapping curve parameter information as in ST2094-40, or can include tone-mapping curve parameter information as in ST2094-10.

[0434] It should be noted that the present invention does not limit baseAfter[i] and base[i] to any domain, and can be a linear domain, PQ domain, log domain, etc. The present invention also does not limit the color space of baseAfter[i] and base[i], and can be a color space such as YUV, RGB, Lab, HSV, etc.

[0435] It should be noted that color gamut mapping can be added before or after processing to convert from the current color gamut to the target color gamut. The current color gamut and the target color gamut include but are not limited to BT.2020, BT.709, DCI-P3, sRGB, etc.

[0436] Step 0903, obtaining a target image based on the second base image and the first gain map, including: step A, obtaining a reconstructed high dynamic range image recHDR based on baseAfter (i.e., the second base image) and enhance (i.e., the first gain map); step B, obtaining a reconstructed high dynamic range image recHDRAfter (i.e., the target image) based on metadata. Optionally, step 0903 also includes step C, performing a transformation process on the above reconstructed high dynamic range image recHDRAfter. This transformation process can be an exponential of 2, log, or other processing, which is not limited in this application. In one embodiment, the target image is a standard dynamic range image, in which case SDR can be used instead of HDR in the following steps.

[0437] Step A, obtaining a reconstructed high dynamic range image recHDR according to baseAfter (i.e., the second base image) and enhance (i.e., the first gain map), can be implemented by the following scheme:

[0438] Solution 1: recHDR[i] = baseAfter[i]*f(enhance[i]), where f() is a conversion function in the numerical domain.

[0439] Solution 2: recHDR[i]=baseAfter[i]+f(enhance[i]), where f() is a conversion function in the numerical domain.

[0440] Where recHDR[i] can be any component of RGB or YUV components, and f(enhance[i]) is the gain of any component obtained by enhancing the data.

[0441] Solution 3: recHDR[i] = f(baseAfter[i] * g(enhance[i])), where baseAfter corresponds to the first OTF or EOTF, enhance corresponds to the second OTF or EOTF, g() represents the conversion from the second OTF or EOTF to the first OTF or EOTF, and f() represents the conversion of the synthesized image to the third OTF or EOTF corresponding to recHDR.

[0442] Solution 4: recHDR[i] = f(baseAfter[i] + g(enhance[i])), where baseAfter corresponds to the first OTF or EOTF, enhance corresponds to the second OTF or EOTF, g() represents the conversion from the second OTF or EOTF to the first OTF or EOTF, and f() represents the conversion of the synthesized image to the third OTF or EOTF corresponding to recHDR.

[0443] Solution 5: recHDR[i] = f(g(baseAfter[i])*enhance[i]), where baseAfter corresponds to the first photoelectric transfer function or electro-optical transfer function, enhance corresponds to the second photoelectric transfer function or electro-optical transfer function, g() represents the conversion from the first photoelectric transfer function or electro-optical transfer function to the second photoelectric transfer function or electro-optical transfer function. f() represents the conversion of the synthesized image to the third photoelectric transfer function or electro-optical transfer function corresponding to recHDR.

[0444] Solution 6: recHDR[i] = f(g(baseAfter[i]) + enhance[i]), where baseAfter corresponds to the first photoelectric transfer function or electro-optical transfer function, enhance corresponds to the second photoelectric transfer function or electro-optical transfer function, g() represents the conversion from the first photoelectric transfer function or electro-optical transfer function to the second photoelectric transfer function or electro-optical transfer function. f() represents the conversion of the synthesized image to the third photoelectric transfer function or electro-optical transfer function corresponding to recHDR.

[0445] Solution seven: recHDR[i] = f(g1(baseAfter[i])*g2(enhance[i])), where baseAfter corresponds to the first photoelectric transfer function or electro-optical transfer function, enhance corresponds to the second photoelectric transfer function or electro-optical transfer function, g1() represents the conversion from the first photoelectric transfer function or electro-optical transfer function to the fourth photoelectric transfer function or electro-optical transfer function, g2() represents the conversion from the second photoelectric transfer function or electro-optical transfer function to the fourth photoelectric transfer function or electro-optical transfer function, and f() represents the conversion of the synthesized image to the third photoelectric transfer function or electro-optical transfer function corresponding to recHDR.

[0446] Solution 8: recHDR[i]=f(g1(baseAfter[i])+g2(enhance[i])), where baseAfter corresponds to the first photoelectric transfer function or electro-optical transfer function, enhance corresponds to the second photoelectric transfer function or electro-optical transfer function, g1() represents the conversion from the first photoelectric transfer function or electro-optical transfer function to the fourth photoelectric transfer function or electro-optical transfer function, g2() represents the conversion from the second photoelectric transfer function or electro-optical transfer function to the fourth photoelectric transfer function or electro-optical transfer function, and f() represents the conversion of the synthesized image to the third photoelectric transfer function or electro-optical transfer function corresponding to recHDR.

[0447] The photoelectric transfer function or electro-optical transfer function in schemes 3 to 8 may be linear, log, HLG, PQ function and their inverse functions. The second photoelectric transfer function or electro-optical transfer function may be the same as the third photoelectric transfer function or electro-optical transfer function.

[0448] Step B, obtaining the reconstructed high dynamic range image recHDRAfter (i.e., the target image) according to the metadata, can be implemented by the following scheme:

[0449] Solution 1: recHDRAfter[i] = recHDR[i].

[0450] Solution 2: The metadata contains the upper limit THL and / or lower limit THH of the reconstructed high dynamic range image, and uses recHDRAfter[i]=recHDR[i]*THH+(A-recHDR[i])*THL, where A is the maximum value stored in recHDR, for example, when recHDR is normalized to between 0 and 1.0, A is 1.0.

[0451] Solution 3: The metadata contains the lower limit THL of the reconstructed high dynamic range image, and uses recHDRAfter[i]=recHDR[i]+THL.

[0452] Solution 4: The metadata contains the upper limit THH of the reconstructed high dynamic range image, and uses recHDRAfter[i]=THH+A-base[i], where A is the maximum value stored in base. When recHDR is normalized to between 0 and 1.0, A is 1.0.

[0453] Solution 5: The data contains mapping parameters, and TME() is obtained based on the parameters, recHDRAfter[i] = TMB(recHDR[i])

[0454] Solution 6: recHDR[i] = A*g(baseAfter1[i]*f1(enhance1[i]))+B*g(baseAfter2[i])..., where the f1() conversion function can be a conversion function for obtaining a specific numerical domain from metadata, or it can be an agreed-upon conversion function, which is f1(x) = x, or other forms, which are not limited by the patent; A, B..., etc. are preset constants or numerical values ​​transmitted by metadata; g() is a function of changing or inversely normalizing the numerical value sent in the metadata or preset, or recHDR[i] + THL, or y = x.

[0455] Solution 7: recHDR[i] = A*baseAfter1[i] + A1*f1(enhance1[i]) + baseAfter1[i] + B*baseAfter2[i], where the f1() conversion function can be a specific conversion function obtained from metadata, or an agreed conversion function, such as f1(x) = x, or other forms, which are not limited by the patent; A and B are constants, or values ​​transmitted by metadata; g() is a function of changing or inversely normalizing the value sent in the metadata or preset, or y = x.

[0456] Solution 8: Obtain the reconstructed high dynamic range image recHDR based on baseAfter1, baseAfter2, baseAfter3… and enhance1, enhance2, enhance3…: recHDR[i]=A*baseAfter1[i]*f1(enhance1[i])+B*baseAfter2[i]*f2(enhance2[i])+C*B*baseAfter3[i]*f3(enhance3[i])+…, where the f1(), f2(), f3() conversion functions can be specific conversion functions obtained from metadata, or can be agreed conversion functions, such as f1(x)=x, or other forms, which are not limited by the patent. Where A, B, C are constants, or values ​​transmitted by metadata,

[0457] Solution 9: recHDR[i]=A*baseAfter1[i]+A1*f1(enhance1[i])+baseAfter1[i]+B*baseAfter2[i]+B1*f1(enhance2[i])+C*baseAfter3[i]+C1*f1(enhance3[i])……wherein the conversion functions f1(), f2(), f3() can be specific conversion functions obtained from metadata, or they can be agreed conversion functions, such as f1(x)=x, or other forms, which are not limited by the patent. Wherein A, B, C, … are constants, or values ​​transmitted by metadata. C. Optionally, the above reconstructed image can be transformed. This transformation can be an exponential of 2, log, or other processing, which is not limited by the patent.

[0458] It should be noted that the present invention does not limit the form of f(x), that is, it does not limit baseAfter[i] and enhance[i] to any domain, which can be the linear domain, PQ domain, log domain, etc. The present invention also does not limit the color space of baseAfter[i] and enhance[i], which can be YUV, RGB, Lab, HSV, etc.

[0459] It should be noted that the present invention does not limit whether other image processing is performed on the reconstructed high dynamic range image recHDR after it is obtained and before it is displayed.

[0460] Optionally, in one implementation, step 0903, obtaining a target image based on the second base image and the first gain map, includes: step A1, transforming the enhanced data (i.e., the first gain map) according to the metadata to obtain the second gain map; step B1, obtaining a reconstructed high dynamic range image recHDR based on baseAfter (i.e., the second base image) and enhanceAfter (i.e., the second gain map); step C1, obtaining a reconstructed high dynamic range image recHDRAfter (i.e., the target image) according to the metadata. Optionally, step 0903 also includes step D1, transforming the above reconstructed high dynamic range image recHDRAfter. This transformation process can be an exponential of 2, log, or other processing, which is not limited in this application. In one embodiment, the target image is a standard dynamic range image. In this case, SDR can be used instead of HDR in the following steps.

[0461] Step A1, transforming the enhancement data (i.e., the first gain map) based on the metadata to obtain a second gain map, includes: Step A11, if the resolution of the enhancement data differs from the resolution of the base image or from a preset target resolution, upsampling / downsampling the enhancement data based on the metadata or preset information; Step A12, processing the enhancement layer enhance based on the enhancement layer remapping information contained in the metadata to obtain enhanceAfter. It should be noted that there are two possible orders for executing these steps of the present invention: the first is to execute step A11 first, then step A12, which is more convenient for system implementation; the second is to execute step A12 first, then step A11, which results in more reasonable overall numerical processing. In an example of the second implementation (i.e., executing step A12 first, then step A11), step A12 is first executed to process the first gain map to obtain enhanceAfter, and then step A11 is used to upsample / downsample the enhanceAfter obtained in step A12. In another example of the second implementation (i.e., executing step A12 first and then executing step A11), step A12 is first executed to perform numerical processing or mapping processing on the first gain map to obtain enhanceAfter, and then step A11 is performed on the enhanceAfter obtained in step A12. In another example of the second implementation (i.e., executing step A12 first and then executing step A11), step A12 is first executed to perform numerical processing or mapping processing on the first gain map to obtain enhanceAfter. This processing may be to process the first gain map according to information such as upper and lower limits of enhancement data, eigenvalues, and mapping functions contained in metadata to obtain enhanceAfter, and then step A11 is performed on the enhanceAfter obtained in step A12 by upsampling or downsampling.

[0462] In step A11 , the resolution of the enhanced data is different from the resolution of the basic image or different from the preset target resolution, and up / down sampling processing is performed on the enhanced data according to metadata or preset information.

[0463] Step A12 of processing the enhancement layer enhancement according to the enhancement layer remapping information included in the metadata to obtain enhanceAfter includes:

[0464] A possible step before processing is to normalize the enchence data. Usually, the range of 8-bit data is 0 to 255. By dividing all numbers by 255.0, they are converted to the range of 0 to 1.0. This application does not limit this.

[0465] Option 1: enchenceAfter[i]=enchence[i]

[0466] Solution 2: The metadata contains the upper limit THL and lower limit THH of the enhancement data, and uses enhancementAfter[i] = enhancement[i]*THH + (A-enchence[i])*THL, where A is the maximum value stored in enhancement. When enhancement is normalized to between 0 and 1.0, A is 1.0.

[0467] Solution 3: The metadata contains the upper limit THL and lower limit THH of the enhancement data, where the enhancement data can be non-linear domain data such as log, PQ, HLG or gamma. First, convert enhancement[i] to the linear domain, and then use enhancementAfter[i] = enhancement[i]*THH+(A-enchence[i])*THL, where A is the maximum value stored in enhancement. When enhancement is normalized to between 0 and 1.0, A is 1.0.

[0468] Solution 4: The data contains the lower limit THL of the basic layer data, and use enchenceAfter[i]=enchence[i]+THL.

[0469] Solution 5: The data contains the upper THH of the base layer data, and use enchenceAfter[i]=THH+A-enchence[i], where A is the maximum value stored in enchence. When enchence is normalized to between 0 and 1.0, A is 1.0.

[0470] Solution 6: The data contains mapping parameters, and TME() is obtained based on the parameters, enchenceAfter[i] = TMB(enchence[i])

[0471] Solution 7: The data contains mapping parameters, and TME() is obtained based on the parameters, enchenceAfter[i] = TMB() * enchence[i]

[0472] Solution 8: The data contains the parameters of the mapping relationship, the upper limit THL, and the lower limit THH of the enhanced data. According to the parameters, TME() is obtained, and enchenceAfter[i] = TMB(enchence[i]*THH+(A-enchence[i])*THL)

[0473] Solution 9: The data contains the parameters of the mapping relationship, the upper limit THL, and the lower limit THH of the enhanced data. According to the parameters, TME() is obtained, and enchenceAfter[i] = TMB()*enchence[i]*THH+(A-enchence[i])*THL

[0474] The mapping relationship has various forms, such as Sigmoid, cubic spline, gamma, straight line, etc. or their inverse function forms, which are not limited in the present invention. The following curves can be used:

[0475] Or its inverse curve. It should be noted that L and L' can be normalized optical signals or electrical signals, which is not limited by the present invention. It can also be a transformation in the form of a neural network: obtaining the type information of the neural network, such as transformer, convolutional neural network, etc.; constructing a neural network based on the network information in the enhanced data; and using the neural network to process the enhanced data. It should be noted that it can be global tone mapping or local tone mapping, which is not limited by the present invention.

[0476] It should be noted that the present invention does not limit enchenceAfter[i] and enchence[i] to any domain, which can be a linear domain, PQ domain, log domain, etc. The present invention also does not limit the color space of enchenceAfter[i] and enchence[i], which can be YUV, RGB, Lab, HSV, etc.

[0477] It should be noted that color gamut mapping can be added before or after processing to convert from the current color gamut to the target color gamut. The current color gamut and target color gamut include but are not limited to BT.2020, BT.709, DCI-P3, sRGB, etc.

[0478] It should be noted that there may be more than one enhancement layer information, including multiple enhance1, enhance2, enhance3, ....; to obtain enhanceAfter1, enhanceAfter2, enhanceAfter3..., it is also possible to adopt enhanceAfter1=enhance1; enhanceAfter2=enhance2; enhanceAfter3=enhance3.

[0479] Step B1, obtaining a reconstructed high dynamic range image recHDR according to baseAfter (i.e., the second base image) and enhanceAfter (i.e., the second gain map), can be implemented by the following scheme:

[0480] Solution 1: recHDR[i]=baseAfter[i]*f(enhanceAfter[i]), where f() is a conversion function in the numerical domain.

[0481] Solution 2: recHDR[i]=baseAfter[i]+f(enhanceAfter[i]), where f() is a conversion function in the numerical domain.

[0482] Where recHDR[i] can be any component of RGB or YUV components, and f(enhanceAfter[i]) is the gain of any component obtained by enhancing the data.

[0483] Solution 3: recHDR[i] = f(baseAfter[i] * g(enhanceAfter[i])), where baseAfter corresponds to the first OTF or EOTF, enhanceAfter corresponds to the second OTF or EOTF, g() represents the conversion from the second OTF or EOTF to the first OTF or EOTF, and f() represents the conversion of the synthesized image to the third OTF or EOTF corresponding to recHDR.

[0484] Solution 4: recHDR[i] = f(baseAfter[i] + g(enhanceAfter[i])), where baseAfter corresponds to the first photoelectric transfer function or electro-optical transfer function, enhanceAfter corresponds to the second photoelectric transfer function or electro-optical transfer function, g() represents the conversion from the second photoelectric transfer function or electro-optical transfer function to the first photoelectric transfer function or electro-optical transfer function. f() represents the conversion of the synthesized image to the third photoelectric transfer function or electro-optical transfer function corresponding to recHDR.

[0485] Solution 5: recHDR[i] = f(g(baseAfter[i])*enhanceAfter[i]), where baseAfter corresponds to the first photoelectric transfer function or electro-optical transfer function, enhanceAfter corresponds to the second photoelectric transfer function or electro-optical transfer function, g() represents the conversion from the first photoelectric transfer function or electro-optical transfer function to the second photoelectric transfer function or electro-optical transfer function. f() represents the conversion of the synthesized image to the third photoelectric transfer function or electro-optical transfer function corresponding to recHDR.

[0486] Solution 6: recHDR[i] = f(g(baseAfter[i]) + enhanceAfter[i]), where baseAfter corresponds to the first photoelectric transfer function or electro-optical transfer function, enhanceAfter corresponds to the second photoelectric transfer function or electro-optical transfer function, g() represents the conversion from the first photoelectric transfer function or electro-optical transfer function to the second photoelectric transfer function or electro-optical transfer function. f() represents the conversion of the synthesized image to the third photoelectric transfer function or electro-optical transfer function corresponding to recHDR.

[0487] Solution seven: recHDR[i] = f(g1(baseAfter[i])*g2(enhanceAfter[i])), where baseAfter corresponds to the first photoelectric transfer function or electro-optical transfer function, enhanceAfter corresponds to the second photoelectric transfer function or electro-optical transfer function, g1() represents the conversion from the first photoelectric transfer function or electro-optical transfer function to the fourth photoelectric transfer function or electro-optical transfer function, g2() represents the conversion from the second photoelectric transfer function or electro-optical transfer function to the fourth photoelectric transfer function or electro-optical transfer function, and f() represents the conversion of the synthesized image to the third photoelectric transfer function or electro-optical transfer function corresponding to recHDR.

[0488] Solution 8: recHDR[i]=f(g1(baseAfter[i])+g2(enhanceAfter[i])), where baseAfter corresponds to the first photoelectric transfer function or electro-optical transfer function, enhanceAfter corresponds to the second photoelectric transfer function or electro-optical transfer function, g1() represents the conversion from the first photoelectric transfer function or electro-optical transfer function to the fourth photoelectric transfer function or electro-optical transfer function, g2() represents the conversion from the second photoelectric transfer function or electro-optical transfer function to the fourth photoelectric transfer function or electro-optical transfer function, and f() represents the conversion of the synthesized image to the third photoelectric transfer function or electro-optical transfer function corresponding to recHDR.

[0489] The photoelectric transfer function or electro-optical transfer function in schemes 3 to 8 may be linear, log, HLG, PQ function and their inverse functions. The second photoelectric transfer function or electro-optical transfer function may be the same as the third photoelectric transfer function or electro-optical transfer function.

[0490] Step C1, obtaining a reconstructed high dynamic range image recHDRAfter (i.e., a target image) according to the metadata, can be implemented by the following scheme:

[0491] Solution 1: recHDRAfter[i] = recHDR[i].

[0492] Solution 2: The metadata contains the upper limit THL and / or lower limit THH of the reconstructed high dynamic range image, and uses recHDRAfter[i]=recHDR[i]*THH+(A-recHDR[i])*THL, where A is the maximum value stored in recHDR, for example, when recHDR is normalized to between 0 and 1.0, A is 1.0.

[0493] Solution 3: The metadata contains the lower limit THL of the reconstructed high dynamic range image, and uses recHDRAfter[i]=recHDR[i]+THL.

[0494] Solution 4: The metadata contains the upper limit THH of the reconstructed high dynamic range image, and uses recHDRAfter[i]=THH+A-base[i], where A is the maximum value stored in base. When recHDR is normalized to between 0 and 1.0, A is 1.0.

[0495] Solution 5: The data contains mapping parameters, and TME() is obtained based on the parameters, recHDRAfter[i] = TMB(recHDR[i])

[0496] Solution 6: recHDR[i]=A*g(baseAfter1[i]*f1(enhanceAfter1[i]))+B*g(baseAfter2[i])……, where the f1() conversion function can be a conversion function for obtaining a specific numerical domain from metadata, or it can be an agreed conversion function, which is f1(x)=x, or other forms, which are not limited by the patent; A, B……etc. are preset constants or numerical values ​​transmitted by metadata; g() is a function of changing or inverse normalizing the numerical value sent in the metadata or preset, or recHDR[i]+THL, or y=x.

[0497] Solution 7: recHDR[i] = A*baseAfter1[i] + A1*f1(enhanceAfter1[i]) + baseAfter1[i] + B*baseAfter2[i], where the f1() conversion function can be a specific conversion function obtained from metadata, or an agreed conversion function, such as f1(x) = x, or other forms, which are not limited by the patent; A and B are constants, or values ​​transmitted by metadata; g() is a function of changing or inversely normalizing the value sent in the metadata or preset, or y = x.

[0498] Solution 8: Obtain the reconstructed high dynamic range image recHDR based on baseAfter1, baseAfter2, baseAfter3… and enhanceAfter1, enhanceAfter2, enhanceAfter3…: recHDR[i]=A*baseAfter1[i]*f1(enhanceAfter1[i])+B*baseAfter2[i]*f2(enhanceAfter2[i])+C*B*baseAfter3[i]*f3(enhanceAfter3[i])+…, where f1(), f2(), f3() conversion functions can be specific conversion functions obtained from metadata, or can be agreed conversion functions, such as f1(x)=x, or other forms, which are not limited by the patent. Where A, B, C are constants, or values ​​transmitted by metadata,

[0499] Solution 9: recHDR[i]=A*baseAfter1[i]+A1*f1(enhanceAfter1[i])+baseAfter1[i]+B*baseAfter2[i]+ B1*f1(enhanceAfter2[i])+C*baseAfter3[i]+C1*f1(enhanceAfter3[i])……wherein the conversion functions f1(), f2(), f3() can be specific conversion functions obtained from metadata, or they can be agreed conversion functions, such as f1(x)=x, or other forms, which are not limited by the patent. Wherein A, B, C, … are constants, or values ​​transmitted by metadata. C. Optionally, the above reconstructed image can be transformed. This transformation can be an exponential of 2, log, or other processing, which is not limited by the patent.

[0500] It should be noted that the present invention does not limit the form of f(x), that is, it does not limit baseAfter[i] and enhanceAfter[i] to any domain, which can be linear domain, PQ domain, log domain, etc. The present invention also does not limit the color space of baseAfter[i] and enhanceAfter[i], which can be YUV, RGB, Lab, HSV, etc.

[0501] It should be noted that the present invention does not limit whether other image processing is performed on the reconstructed high dynamic range image recHDR after it is obtained and before it is displayed.

[0502] The solution provided by this application is compatible with image encoding and processing in various formats, and can achieve relatively good image quality on systems with different support capabilities. Up / down sampling can significantly reduce file size.

[0503] Figure 9 is a schematic diagram of a method for processing a digital signal according to an embodiment of the present application. The method shown in Figure 9 can be applied to a decoding end, such as the decoding module shown in Figure 6.

[0504] 901, obtain metadata, base layer data and enhancement layer data from the bitstream.

[0505] 902. Perform transformation processing on the basic layer data.

[0506] 903: Perform transformation processing on the enhancement layer data.

[0507] 904 , synthesize the transformed base layer data and the transformed enhancement layer data according to the metadata to obtain a reconstructed HDR object.

[0508] Figure 10 is a schematic flow chart of another method for processing digital signals according to an embodiment of the present application. The method shown in Figure 10 can be applied to a decoding end, such as the decoding module shown in Figure 6.

[0509] 1001, obtain metadata, base layer data and enhancement layer data from the bitstream.

[0510] 1002. Perform transformation processing on the base layer data according to the metadata.

[0511] 1003. Perform transformation processing on the enhancement layer data according to the metadata.

[0512] 1004 : Obtain a reconstructed HDR object based on the metadata-merged base layer data and the transformed enhancement layer data.

[0513] Figure 11 is a schematic flow chart of another method for processing digital signals according to an embodiment of the present application. The method shown in Figure 11 can be applied to a decoding end, such as the decoding module shown in Figure 6.

[0514] 1101, obtain metadata, base layer data and enhancement layer data from the bitstream.

[0515] 1102. Perform transformation processing on the base layer data according to the metadata.

[0516] 1103. Perform transformation processing on the enhancement layer data according to the metadata.

[0517] 1104 , merging the processed base layer data and the transformed enhancement layer data to obtain a reconstructed HDR object.

[0518] Figure 12 is a schematic flow chart of another method for processing digital signals according to an embodiment of the present application. The method shown in Figure 12 can be applied to a decoding end, such as the decoding module shown in Figure 6.

[0519] 1201, obtain base layer data and enhancement layer data from the code stream.

[0520] 1202. Perform transformation processing on the basic layer data.

[0521] 1203. Perform transformation processing on the enhancement layer data.

[0522] 1204 , merge the processed base layer data and the transformed enhancement layer data to obtain a reconstructed HDR object.

[0523] The following is a detailed introduction to the embodiments of the present application with reference to Figures 9 to 12.

[0524] As described above, in some embodiments, the decoding module needs to obtain metadata, base layer data, and enhanced data from the bitstream. In other words, the decoding module needs to obtain metadata from the bitstream, obtain base layer data from the bitstream, and obtain enhanced data from the bitstream.

[0525] The decoding module obtains base layer data from the bitstream.

[0526] The embodiments of the present application do not limit the format of the HDR object. The HDR object can be any compression standard image or video. For example, in some embodiments, the HDR object can be an image or video obtained based on high efficiency video coding (HEVC), joint photographic experts group (JPEG), Apple ProRes (ProRes), or high efficiency image file format (HEIF). Accordingly, the embodiments of the present application do not limit the type of decoder or decoding module. For example, the decoding module can be a HEVC decoding module, a JPEG decoding module, a ProRes decoding module, or a HEIF decoding module.

[0527] The embodiments of the present application do not limit the color space format of the HDR object. For example, red green blue (RGB), luminance chronminance (YUV), lightness AB (LAB), or hue saturation lightness (HSL) can be used.

[0528] The embodiments of the present application do not affect the bit width of the HDR object. For example, the bit width of the HDR object can be 10 bits, 12 bits, or 12 bits, etc.

[0529] The decoding module obtains metadata from the bitstream.

[0530] The embodiments of the present application do not limit the location of metadata. The location of the metadata in the code stream can be the same as the location of metadata in the code stream in existing protocols or applications, and the method of obtaining metadata can also be the same as that of existing protocols or applications. For example, in some embodiments, the decoding module can obtain metadata from the supplemental enhancement information (SEI) of the HEVC or universal video coding (VVC) file, the user-defined network abstraction layer (NAL) or reserved unit, the application (APP) extension information encapsulated in the JPEG file interchange format (JPEG file interchange format, JFIF), or the data segment encapsulated in the Motion Picture Experts Group (MPEG)-4 (MPEG-4, MP4). In other embodiments, some metadata can also be obtained from some specified locations, such as the location after the end of image (EOI) of the complete JPEG file. These specified locations can be indicated by the information carried in the location originally used to carry metadata.

[0531] Metadata mainly includes data such as the source data format, region division information, region traversal order information, or image features and curve parameters, as well as one or more metadata information units. Metadata information units include data such as coordinate information, image features and curve parameters.

[0532] The decoding module obtains enhancement layer data from the bitstream.

[0533] In some embodiments, the position of the enhancement layer data in the code stream is the same as the position of the metadata in the code stream. Or, in other words, the enhancement layer data can be carried by metadata. For example, in some embodiments, the decoding module can obtain the enhancement layer data from the SEI of the HEVC or VVC file, the user-defined NAL or reserved unit, the APP extension information of the JFIF package, or the data segment of the MP4 package. In other embodiments, the enhancement layer data can also be obtained from some specified locations, such as the position after the end of image (EOI) of the complete JPEG file.

[0534] One pixel value corresponds to one enhancement data, or multiple pixel values ​​correspond to one enhancement data, wherein the enhancement data may contain multiple values, and the number of values ​​contained in the enhancement data corresponding to each pixel is the same.

[0535] In other words, in some embodiments, one pixel value in the base layer data may correspond to one enhancement data in the enhancement layer data. In other embodiments, multiple pixel values ​​in the base layer image may correspond to one enhancement data in the enhancement layer data.

[0536] In some embodiments, one enhanced data may include one value. In other embodiments, one enhanced data may include multiple values. The enhanced data corresponding to each pixel includes the same number of values.

[0537] The base layer data is transformed. The base layer data in step 902 and the enhancement layer data in step 903 are the base layer data and enhancement layer data of the same HDR object. Similarly, the base layer data in step 1002 and the enhancement layer data in step 1003 are the base layer data and enhancement layer data of the same HDR object. The base layer data in step 1102 and the enhancement layer data in step 1103 are the base layer data and enhancement layer data of the same HDR object. The base layer data in step 1202 and the enhancement layer data in step 1203 are the base layer data and enhancement layer data of the same HDR object. For ease of description, the base layer data in steps 902, 1002, 1102, and 1202 may be referred to as first base layer data, and the enhancement layer data in steps 903, 1003, 1103, and 1203 may be referred to as first enhancement layer data. Accordingly, the first base layer data and the first enhancement layer data are the base layer data and enhancement layer data of the first HDR object. Accordingly, the metadata mentioned in Figures 9 to 11 may be referred to as first metadata.

[0538] Therefore, the transformation processing of the base layer data in Figures 9 to 12 can also be expressed as the transformation processing of the first base layer data. The transformation processing of the enhancement layer data in Figures 9 to 12 can also be expressed as the transformation processing of the first enhancement layer data. For ease of description, the transformed first base layer data can be referred to as the second base layer data. Correspondingly, the transformed first enhancement layer data can be referred to as the second enhancement layer data. Therefore, the merged base layer data and the transformed enhancement layer data in Figures 9 to 12 can be expressed as the merging of the second base layer data and the second enhancement layer data. Correspondingly, the reconstructed HDR object referred to in Figures 9 to 12 can be referred to as the second HDR object.

[0539] For ease of description, base[i] can be used to represent the value of the i-th pixel in the first base layer data, and baseAfter[i] can be used to represent the value of the i-th pixel in the second base layer data, where i is a positive integer greater than or equal to 1. As described above, the embodiments of the present application are not limited to color spaces. Therefore, the value of the i-th pixel can be a color value in the RGB color space, or a Y value, U value, or V value in the YUV color space.

[0540] Several first base layer data conversion schemes are provided below. The encoding module can use any of the following conversion schemes to convert the first base layer data into second base layer data.

[0541] Optionally, in some embodiments, the conversion scheme of the first base layer data is a default scheme. In other words, after acquiring the first base layer data, the encoding module can use the default conversion scheme to convert the first base layer data into the second base layer data.

[0542] Optionally, in other embodiments, the conversion scheme for the first base layer data is determined by an encoding module. The encoding module may select a conversion scheme from multiple conversion schemes based on some reference information to convert the first base layer data into the second base layer data. For example, the reference information may be some characteristic information of the first base layer data, such as data characteristics of the base layer data, such as the maximum value, minimum value, average value, and distribution of pixel values.

[0543] Optionally, in some other embodiments, the first metadata may include basic layer data conversion scheme indication information. The basic layer data conversion scheme indication information is used to indicate the basic layer data conversion scheme. The encoding module may convert the first basic layer data according to the conversion scheme indicated by the basic layer data conversion scheme indication information. For example, each conversion scheme has an index, and the basic layer conversion scheme indication information may carry the index of the conversion scheme. In this way, the encoding module may determine the conversion scheme to be used based on the index. For another example, the conversion scheme indication information may be implicit indication information. For example, if the first metadata contains some relevant information required for the conversion scheme, such as parameters, mapping relationships, filter information, etc., the encoding module may use the corresponding conversion scheme to perform a conversion operation on the first basic layer data based on the relevant information of the conversion scheme obtained.

[0544] Optionally, in other embodiments, the encoding module may first determine whether the first metadata includes the base layer data conversion scheme indication information. If the first metadata includes the base layer conversion indication information, the encoding module may convert the first base layer data according to the conversion method indicated by the base layer conversion indication information. If the first metadata does not include the base layer conversion indication information, the encoding module may use a default conversion scheme or determine a conversion scheme on its own.

[0545] In some embodiments, the conversion scheme may be one of the following schemes 1 to 15.

[0546] Solution 1: baseAfter[i]=Base[i].

[0547] Solution 2: The metadata contains the upper limit THL and / or lower limit THH of the base layer data, and uses baseAfter[i]=base[i]×THH+(A-base[i])×THL, where A is the maximum value stored in base, and when base is normalized to between 0 and 1.0, A is 1.0.

[0548] Solution 3: The metadata contains the lower limit THL of the basic layer data, and baseAfter[i]=base[i]+THL is used.

[0549] Solution 4: The metadata contains the upper limit THH of the base layer data, and uses baseAfter[i]=THH+A-base[i], where A is the maximum value stored in base. When base is normalized to between 0 and 1.0, A is 1.0.

[0550] Solution 5: The metadata contains parameters of the mapping relationship, and TMB() is obtained according to the parameters, baseAfter[i]=TMB(base[i]).

[0551] Solution 6: The metadata contains the parameters of the mapping relationship, and TMB() is obtained according to the parameters, baseAfter[i]=TMB(L i )×base[i], where L i It is a value determined according to base[i], for example, it can be a brightness value or other value determined according to the pixel value of the i-th pixel.

[0552] Solution 7: The metadata contains the mapping parameters, the upper limit THL, and / or lower limit THH of the base layer data. TMB() is obtained based on the parameters: baseAfter[i] = TMB(base[i] × THH + (A - base[i]) × THL)

[0553] Solution 8: The metadata contains the mapping relationship parameters, the upper limit THL, and / or lower limit THH of the base layer data, and obtains TMB() according to the parameters, baseAfter[i]=TMB(L i )×(base[i]×THH+(A-base[i])×THL), where L i It is a value determined according to base[i], for example, it can be a brightness value or other value determined according to the pixel value of the i-th pixel.

[0554] Solution 9: The metadata contains the mapping parameters and the lower limit THL of the base layer data. According to the parameters, TMB() is obtained, baseAfter[i] = TMB(base[i] + THL)

[0555] Solution 10: The metadata contains the mapping relationship parameters and the lower limit THL of the basic layer data. According to the parameters, TMB() is obtained, baseAfter[i]=TMB(L i )×(base[i]+THL), where L i It is a value determined according to base[i], for example, it can be a brightness value or other value determined according to the pixel value of the i-th pixel.

[0556] Solution 11: The data contains parameters of the mapping relationship. TMB() and TMB1() are obtained based on the parameters. TMB() can be a global or local tone mapping function. TMB1 can be in the form of linear, spline, or piecewise curve. baseAfter[i]=TMB(TMB1(base[i])).

[0557] Solution 12: The data contains parameters of the mapping relationship. According to the parameters, TMB() and TMB1() are obtained. TMB() can be a global or local tone mapping function. TMB1 can be a linear, spline, or piecewise curve. baseAfter[i] = TMB(L i )×TMB1(base[i]), where L i It is a value determined according to base[i], for example, it can be a brightness value or other value determined according to the pixel value of the i-th pixel.

[0558] Solution 13: The metadata contains parameters of the mapping relationship. According to the parameters, TMB(), TMB1() and F[] are obtained. TMB() can be a global or local tone mapping function, TMB1 can be in the form of linear, spline, segmented curve, etc., and F[] is a spatial filter parameter or other image smoothing function. F[] is used to obtain baseMid1[i] and baseAfter2[i]: baseMid1[i]=F[base[i]] or ∑F[n]×base[i+n], and baseAfter2[i]=base[i]-baseMid1[i]. There are various forms of filtering methods, such as bilateral filtering, interpolation filtering, etc., which are not limited in this application. Then baseAfter[i]=TMB(L i )×TMB1(baseMid1[i]) or baseAfter[i]=TMB(TMB1(baseMid1[i])), where L i It is a value determined according to base[i], for example, it can be a brightness value or other value determined according to the pixel value of the i-th pixel.

[0559] Solution 14: The metadata contains parameters of the mapping relationship, and F[] is obtained according to the parameters. F[] is the spatial filter parameter or other image smoothing processing function, and F[] is used to obtain baseAfter[i] and baseAfter2[i]: baseAfter[i] = F[base[i]] or ∑F[n]×base[i+n], and baseAfter2[i] = base[i]-baseMid1[i]. There are various forms of filtering methods, such as bilateral filtering, interpolation filtering, etc., which are not limited in this application.

[0560] Solution 15: The metadata contains parameters for filtering or other image processing. Based on these parameters, a specific function F() is obtained. F() acts on the basic layer to obtain multiple data after processing: baseAfter[i] = F(base[i]) = [baseAfter1[i], baseAfter2[i], baseAfter3[i]…], and there are various forms of filtering methods, such as bilateral filtering, interpolation filtering, etc., which are not limited in this application (full modification).

[0561] The mapping relationship (i.e., TMB or TMB1 in the above embodiment) can have various forms, such as Sigmoid, cubic spline, gamma, straight line, etc. or their inverse function forms, which are not limited in this application. The following curves can be used:

[0562] Or its inverse curve. It should be noted that L and L' can be normalized optical signals or electrical signals, which is not limited in this application. For the i-th element of the basic layer, the above formula can be transformed into:

[0563] In other words, TMB(L i )=F(L i ).

[0564] It should be noted that this application does not limit the format of metadata. Metadata can include histogram information and tone-mapping curve parameter information as in ST20114-40, or tone-mapping curve parameter information as in ST20114-12.

[0565] It should be noted that this application does not limit baseAfter[i] and base[i] to any domain, which can be a linear domain or a PQ domain or a log domain. This application also does not limit the color space of baseAfter[i] and base[i], which can be a YUV, RGB, Lab, HSV, etc. color space.

[0566] It should be noted that color gamut mapping can be added before or after processing to convert from the current color gamut to the target color gamut. The current color gamut and target color gamut include but are not limited to BT.2020, BT.9011, DCI-P3, sRGB, etc.

[0567] Solution 1: The second base layer data and the first base layer data can be the same. In other words, the second base layer data and the first base layer data satisfy the following relationship: baseAfter[i]=base[i] (Formula 6).

[0568] In some embodiments, the first metadata may carry an indication that instructs the decoding module to determine the second base layer data according to the relationship between the second base layer data and the first base layer data shown in Formula 6. In other embodiments, the relationship between the second base layer data and the first base layer data shown in Formula 6 may be a default relationship. Therefore, in this case, the decoding module can directly determine the second base layer data without obtaining information from the first metadata.

[0569] In other embodiments, the relationship between the second basic layer data and the first basic layer data can be determined by one or more mapping relationships. In some embodiments, the one or more mapping relationships can be in the form of a mapping table. In other words, the one or more mapping relationships can be one or more mapping tables. In other embodiments, the one or more mapping relationships can be functional relationships. In other words, the one or more mapping relationships can be one or more functions. In other embodiments, if there are multiple mapping relationships, then some of the mapping relationships in these mapping relationships can be functions, and another part of the mapping relationships can be mapping tables. It can be understood that the "multiple" referred to here (for example, multiple mapping relationships, multiple functions, or multiple mapping tables) can include two or more.

[0570] In some embodiments, the first basic layer data may be transformed using one function or multiple functions to obtain the second basic layer data.

[0571] Solution 2: The first metadata includes the upper limit and / or lower limit of the first base layer data. The first base layer data and the second base layer data satisfy the following relationship: baseAfter[i]=base[i]×THH+(A-base[i])×THL, (Formula 9)

[0572] Among them, base[i] represents the value of the i-th pixel in the first basic layer data, baseAfter[i] represents the value of the i-th pixel in the second basic layer data, THL represents the upper limit of the first basic layer data, THH represents the lower limit of the first basic layer data, A is the maximum value stored in the first basic layer data, and when the first basic layer data is normalized to between 0 and 1.0, A is the maximum value in the normalized interval, that is, 1.0.

[0573] It is understood that the above normalized interval of 0 to 1.0 is only an example of a normalized interval, and the normalized interval may also be other numerical ranges, for example, 0 to 2.0, 0.5 to 1, or 1.0 to 3.0.

[0574] It can be understood that the upper limit and / or lower limit of the first base layer data in formula 1.9 are just examples of two reference values ​​used to convert the first base layer data. In other words, the first metadata can carry two reference values, which can be called a first reference value and a second reference value. The first reference value, the second reference value, the first base layer data, and the second base layer data satisfy the following relationship: baseAfter[i] = base[i] × REF1 + (A-base[i]) × REF2, (Formula 10)

[0575] Where baseAfter[i] represents the value of the i-th pixel in the second base layer data, base[i] represents the value of the i-th pixel in the first base layer data, REF1 represents the first reference value, REF2 represents the second reference value, and A represents the maximum value of the multiple pixels included in the first base layer data. Similarly, the first base layer data in Formula 10 can also be normalized to a normalization interval. A is the maximum value in the normalization interval.

[0576] As described above, when the first reference value is equal to THH and the second reference value is equal to THL, Formula 10 is the same as Formula 9. The first reference value and / or the second reference value may also be other values. For example, in some embodiments, the first reference value may be equal to the average value of the first base layer data. For example, in other embodiments, the first reference value and / or the second reference value may be a preset value, a value determined by negotiation between the encoder and decoder, a value determined based on feature information of the first base layer data, or a value determined based on a coding scheme.

[0577] As mentioned above, Formula 9 is only an example of Solution 2, and Solution 2 can also be expressed as Formula 3.

[0578] Solution 3: The first metadata includes the lower limit of the first base layer data. The first base layer data and the second base layer data satisfy the following relationship: baseAfter[i]=base[i]+THL, (Formula 11)

[0579] Among them, base[i] represents the value of the i-th pixel in the first basic layer data, baseAfter[i] represents the value of the i-th pixel in the second basic layer data, and THH represents the lower limit of the first basic layer data.

[0580] Similarly, the lower limit of the first base layer data in Formula 11 indicates an example of a reference value. This reference value can be called a third reference value. The third reference value, the first base layer data, and the second base layer data satisfy the following relationship: baseAfter[i]=base[i]+REF3, (Formula 12)

[0581] Wherein, baseAfter[i] represents the value of the i-th pixel in the second basic layer data, base[i] represents the value of the i-th pixel in the first basic layer data, and REF3 represents the third reference value.

[0582] Similarly, the third reference value may be the lower limit of the first base layer data. In this case, Formula 11 may be equal to Formula 12. The third reference value may also be other values, such as a preset value, a value determined according to a coding scheme, a value determined by negotiation between the coding module and the decoding module, or a value determined according to characteristics of the first base layer data.

[0583] As described above, Formula 11 is only an example of Solution 3, and Solution 3 can also be expressed as Formula 12. Similarly, the first basic layer data in Formulas 11 and 12 can also be normalized to a normalization interval.

[0584] Solution 4: The first metadata includes the upper limit of the first base layer data. The first base layer data and the second base layer data satisfy the following relationship: baseAfter[i]=THH+A-base[i], (Formula 11)

[0585] Where base[i] represents the value of the i-th pixel in the first base layer data, baseAfter[i] represents the value of the i-th pixel in the second base layer data, THL represents the upper limit of the first base layer data, and A is the maximum value stored in the first base layer data. When the first base layer data is normalized to between 0 and 1.0, A is the maximum value in this normalization interval, i.e., 1.0. The normalization interval [0, 1.0] is only for reference; the upper and / or lower limits of the normalization interval can also be other values.

[0586] Similarly, the lower limit of the first base layer data in Formula 11 indicates an example of a reference value. This reference value can be called a fourth reference value. The fourth reference value, the first base layer data, and the second base layer data satisfy the following relationship: baseAfter[i]=REF4+A-base[i], (Formula 12)

[0587] Wherein, baseAfter[i] represents the value of the i-th pixel in the second basic layer data, base[i] represents the value of the i-th pixel in the first basic layer data, and REF4 represents the fourth reference value.

[0588] Similarly, the fourth reference value may be an upper limit of the first base layer data. In this case, Formula 11 may be equal to Formula 12. The fourth reference value may also be other values, such as a preset value, a value determined according to a coding scheme, a value determined by negotiation between the coding module and the decoding module, or a value determined according to characteristics of the first base layer data.

[0589] As described above, Formula 11 is only an example of Solution 4, and Solution 4 can also be expressed as Formula 12. Similarly, the first basic layer data in Formulas 11 and 12 can also be normalized to a normalization interval.

[0590] Solution 5: The first metadata includes a first mapping relationship. The first base layer data and the second base layer data satisfy the following relationship: baseAfter[i]=TMB1(R base[i]_1 ), (Formula 13)

[0591] Wherein, baseAfter[i] represents the value of the i-th pixel in the second basic layer data, R base[i]_1 represents the first reference value of the i-th pixel determined according to the value of the i-th pixel in the first basic layer data, and TMB1() represents the first mapping relationship.

[0592] In some embodiments, R base[i]_1 It can be other values ​​determined by base[i].

[0593] For example, in some embodiments, R base[i]_1 and base[i] satisfy the following relationship: R base[i]_1 =base[i]×REF1+(A-base[i])×REF2, (Formula 14)

[0594] Wherein, base[i] represents the value of the i-th pixel in the first basic layer data, REF1 represents the first reference value, REF2 represents the second reference value, and A represents the maximum value of multiple pixels included in the first basic layer data.

[0595] If REF1 is the lower limit of the first basic layer data and REF2 is the upper limit of the first basic layer data, then formula 14 can be expressed as: base[i]_1 =base[i]×THH+(A-base[i])×THL, (Formula 15)

[0596] Wherein, baseAfter[i] represents the value of the i-th pixel in the second basic layer data, THL represents the upper limit of the first basic layer data, THH represents the lower limit of the first basic layer data, A is the maximum value stored in the first basic layer data, and when the first basic layer data is normalized to between 0 and 1.0, A is the maximum value in the normalized interval, i.e., 1.0.

[0597] If Formula 15 is substituted into Formula 13, the following formula can be obtained: baseAfter[i]=TMB1(base[i]×THH+(A-base[i])×THL), (Formula 16).

[0598] If Formula 13 is substituted into Formula 13, the following formula can be obtained: baseAfter[i]=TMB1(base[i]×REF1+(A-base[i])×REF2), (Formula 19).

[0599] In other embodiments, R base[i]_1 Can be equal to base[i].

[0600] The main product forms of this application are video terminal devices such as mobile phones, TVs, tablets, and projectors.

[0601] The implementation method of this application is: on video terminal devices such as mobile phones, TVs, tablets, projectors, etc., it is mainly implemented in the form of hardware chips, and on live broadcast and video playback devices, it is mainly implemented in the form of software program codes.

[0602] Transform enhanced data based on metadata

[0603] A. Obtain the enhanced data corresponding to each pixel value, that is, upsample the enhanced data.

[0604] Solution 1: Pass an index or an index per region and select a preferred interpolation method from a preset set. The preset set can be OpenGL's GL_NEAREST (nearest neighbor) and GL_LINEAR (nearest neighbor) texture interpolation methods; Vulkan's NEAREST (nearest neighbor) or LINEAR (nearest neighbor) texture interpolation methods; multiple filter sets; multiple directional interpolation methods; bicubic splines; or a full set or subset of all the above methods.

[0605] Solution 2: Select a default texture interpolation method from OpenGL, Vulkan, or matel. This can be OPENGL's texture interpolation method GL_NEAREST (nearest neighbor) and GL_LINEAR (nearest neighbor); or Vulkan's texture interpolation method NEAREST (nearest neighbor) or LINEAR (nearest neighbor).

[0606] Option 3: A preset filter, or a combination of directional interpolation methods; bicubic splines and other methods are also possible.

[0607] Option 4: If the enhanced data and the basic layer image data are sampled in an integer multiple relationship, such as two, three or N basic layer image pixels in the width direction or the height direction correspond to one enhanced data, where one specific pixel among these two, three or N basic layer image pixels has the same or very close coordinates as the enhanced data in the image.

[0608] 1) Get the base image pixel value base[i] of the base layer image corresponding to the current pixel

[0609] 2) Obtain multiple specific pixels among the pixels surrounding the current pixel, which have corresponding enhancement data.

[0610] 3) Find the specific pixel whose base layer image value is closest to the base layer image value at the current position, and directly use the enhancement data corresponding to the specific pixel as the enhancement data of the current pixel;

[0611] Or two, the enhanced data of the current pixel is obtained by interpolating or fitting the curve according to the enhanced data corresponding to the two specific pixels.

[0612] The sampling of gainmap should be based on SDR. If there is no gainmap value at the current position, there are gainmap values ​​corresponding to multiple adjacent SDRs. The gainmap corresponding to the SDR value closest to the SDR brightness value at the current position is selected from these values.

[0613] B. Process the enhancement layer enhance according to the enhancement layer remapping information contained in the metadata and obtain enhanceAfter:

[0614] A possible step before processing is to normalize the enhance data. Usually, the range of 8-bit data is 0 to 255. By dividing all numbers by 255.0, they are converted to the range of 0 to 1.0. This process is not within the scope of this application.

[0615] Solution 1: enhanceAfter[i] = enhance[i]

[0616] Solution 2: The metadata contains the upper limit THL and lower limit THH of the enhancement data, and uses enhanceAfter[i] = enhance[i]*THH + (A-enhance[i])*THL, where A is the maximum value stored for enhance. When enhance is normalized to between 0 and 1.0, A is 1.0.

[0617] Solution 3: The metadata contains the upper limit THL and lower limit THH of the enhancement data, where the enhancement data can be non-linear domain data such as log, PQ, HLG or gamma. First, convert enhance[i] to the linear domain, and then use enhanceAfter[i] = enhance[i]*THH+(A-enhance[i])*THL, where A is the maximum value stored in enhance. When enhance is normalized to between 0 and 1.0, A is 1.0.

[0618] Solution 4: The data contains the lower limit THL of the basic layer data, and use enhanceAfter[i]=enhance[i]+THL.

[0619] Solution 5: The data contains the upper THH of the base layer data, and use enhanceAfter[i]=THH+A-enhance[i], where A is the maximum value stored in enhance. When enhance is normalized to between 0 and 1.0, A is 1.0.

[0620] Solution 6: The data contains mapping parameters, and TME() is obtained based on the parameters, enhanceAfter[i] = TMB(enhance[i])

[0621] Solution 6: The data contains mapping parameters, and TME() is obtained based on the parameters, enhanceAfter[i] = TMB() * enhance[i]

[0622] Solution 6: The data contains the mapping parameters, the upper limit THL, and the lower limit THH of the enhanced data. According to the parameters, TME() is obtained, and enhanceAfter[i] = TMB(enhance[i]*THH+(A-enhance[i])*THL)

[0623] Solution 6: The data contains the parameters of the mapping relationship, the upper limit THL, and the lower limit THH of the enhanced data. According to the parameters, TME() is obtained, enhanceAfter[i] = TMB()*enhance[i]*THH+(A-enhance[i])*THL

[0624] The mapping relationship can take various forms, including Sigmoid, cubic spline, gamma, straight line, piecewise curve, etc. or their inverse function forms, which are not limited in this application. The following curves can be used:

[0625] It should be noted that L and L' can be normalized optical signals or electrical signals, which is not limited in this application.

[0626] Alternatively, the transformation can be a neural network transformation: obtaining information about the type of neural network, such as a transformer or convolutional neural network; constructing a neural network based on the network information in the augmented data; and processing the augmented data using the neural network. It should be noted that this can be global tone mapping or local tone mapping, and this application does not limit this.

[0627] It should be noted that this application does not limit enhanceAfter[i] and enhance[i] to any domain, which can be linear domain, PQ domain, log domain, etc. This application also does not limit the color space of enhanceAfter[i] and enhance[i], which can be YUV, RGB, Lab, HSV, etc.

[0628] It should be noted that color gamut mapping can be added before or after processing to convert from the current color gamut to the target color gamut. The current color gamut and target color gamut include but are not limited to BT.2020, BT.709, DCI-P3, sRGB, etc.

[0629] It should be noted that there may be more than one enhancement layer information, including multiple enhance1, enhance2, enhance3, ....; to obtain enhanceAfter1, enhanceAfter2, enhanceAfter3..., it is also possible to adopt enhanceAfter1 = enhance1; enhanceAfter2 = enhance2; enhanceAfter3 = enhance3;

[0630] Merge the base image and enhanced data according to the metadata to obtain a reconstructed high dynamic range image

[0631] A. Get the reconstructed high dynamic range image recHDR based on baseAfter and enhanceAfter:

[0632] Solution 1: recHDR[i]=baseAfter[i]*f(enhanceAfter[i]), where f() is a conversion function in the numerical domain.

[0633] Solution 2: recHDR[i]=baseAfter[i]+f(enhanceAfter[i]), where f() is a conversion function in the numerical domain.

[0634] Where recHDR[i] can be any component of RGB or YUV components, and f(enhanceAfter[i]) is the gain of any component obtained by enhancing the data.

[0635] B. Get the reconstructed high dynamic range image recHDRAfter based on the metadata:

[0636] Solution 1: recHDRAfter[i] = recHDR[i].

[0637] Solution 2: The metadata contains the upper limit THL and / or lower limit THH of the reconstructed high dynamic range image, and uses recHDRAfter[i]=recHDR[i]*THH+(A-recHDR[i])*THL, where A is the maximum value stored in recHDR, for example, when recHDR is normalized to between 0 and 1.0, A is 1.0.

[0638] Solution 3: The metadata contains the lower limit THL of the reconstructed high dynamic range image, and uses recHDRAfter[i]=recHDR[i]+THL.

[0639] Solution 4: The metadata contains the upper limit THH of the reconstructed high dynamic range image, and uses recHDRAfter[i]=THH+A-base[i], where A is the maximum value stored in base. When recHDR is normalized to between 0 and 1.0, A is 1.0.

[0640] Solution 5: recHDR[i] = A*g(baseAfter1[i]*f1(enhanceAfter1[i]))+B*g(baseAfter2[i])..., where the f1() conversion function can be a conversion function for obtaining a specific numerical domain from metadata, or it can be an agreed-upon conversion function, which is f1(x) = x, or other forms, which are not limited by the patent; A, B..., etc. are preset constants or numerical values ​​transmitted by metadata; g() is a function of changing or inversely normalizing the numerical value sent in the metadata or preset, or recHDR[i] + THL, or y = x.

[0641] Solution 6: recHDR[i] = A*baseAfter1[i] + A1*f1(enhanceAfter1[i]) + baseAfter1[i] + B*baseAfter2[i], where the f1() conversion function can be a specific conversion function obtained from metadata, or an agreed conversion function, such as f1(x) = x, or other forms, which are not limited by the patent; A and B are constants, or values ​​transmitted by metadata; g() is a function of changing or inversely normalizing the value sent in the metadata or preset, or y = x.

[0642] Solution 7: Obtain a reconstructed high dynamic range image recHDR based on baseAfter1, baseAfter2, baseAfter3, etc. and enhanceAfter1, enhanceAfter2, enhanceAfter3, etc.: recHDR[i] = A*baseAfter1[i]*f1(enhanceAfter1[i])+B*baseAfter2[i]*f2(enhanceAfter2[i])+C*B*baseAfter3[i]*f3(enhanceAfter3[i])+…, where the conversion functions f1(), f2(), and f3() can be specific conversion functions obtained from metadata, or can be agreed conversion functions, such as f1(x) = x, or other forms, which are not limited by the patent. Where A, B, C, etc. are constants or values ​​transmitted by metadata. C. can optionally perform a transformation on the reconstructed image. This transformation can be an exponential of 2, log, or other processing, which is not limited by the patent.

[0643] Solution 8: recHDR[i]=A*baseAfter1[i]+A1*f1(enhanceAfter1[i])+baseAfter1[i]+B*baseAfter2[i]+B1*f1(enhanceAfter2[i])+C*baseAfter3[i]+C1*f1(enhanceAfter3[i])……wherein the conversion functions f1(), f2(), f3() can be specific conversion functions obtained from metadata, or they can be agreed conversion functions, such as f1(x)=x, or other forms, which are not limited by the patent. Wherein A, B, C, … are constants, or values ​​transmitted by metadata. C. Optionally, the above reconstructed image can be transformed. This transformation can be an exponential of 2, log, or other processing, which is not limited by the patent.

[0644] It should be noted that this application does not limit the form of f(x), that is, it does not limit baseAfter[i] and enhanceAfter[i] to any domain, which can be linear domain, PQ domain, log domain, etc. This application also does not limit the color space of baseAfter[i] and enhanceAfter[i], which can be YUV, RGB, Lab, HSV, etc.

[0645] It should be noted that the present application does not limit whether other image processing is performed on the reconstructed high dynamic range image recHDR after it is obtained and before it is displayed.

[0646] The technical solution of this application is compatible with image encoding and processing in multiple formats, and can achieve relatively good image quality on systems with different support capabilities. This application proposes an encoding and decoding method that is compatible with code streams in multiple formats, providing compatibility with systems with different capabilities.

[0647] The encoding scheme of the present invention mainly includes the following processes:

[0648] Acquiring high-dynamic image data

[0649] Obtain high-dynamic image data from the camera image acquisition channel:

[0650] This process can be achieved by collecting multiple lower dynamic range images with different exposure values ​​through a camera and synthesizing them to obtain image data with a higher dynamic range.

[0651] This process can make the camera's photosensitive element have different photoelectric characteristics at different positions, thereby obtaining high dynamic range data with more image grayscales.

[0652] Obtain high dynamic range image data from other sources:

[0653] This process can be to obtain high-dynamic images through professional acquisition equipment, and then transmit them to the processing equipment by sending or copying.

[0654] It should be noted that the present invention does not limit the method of obtaining high dynamic range image data.

[0655] It should be noted that the present invention does not limit the format of high-dynamic range image data. In terms of color space, it can be YUV or RGB. In terms of data bit width, it can be 8-bit, 10-bit, or 12-bit, etc. Photoelectric conversion characteristics can be PQ, gamma, log, HLG, etc.

[0656] It should be noted that the present invention may also include processing the numerical domain of high-dynamic image data to obtain the data range (minimum value, and / or maximum value) of the high-dynamic image data, mapping the minimum value of the intermediate basic layer image data to 0 or a preset value, and / or mapping the maximum value to 1.0 or a preset value, and / or mapping the intermediate value to a certain intermediate value based on the mapping relationship between the maximum value and / or the minimum value, thereby obtaining high-dynamic image data for subsequent calculations.

[0657] Get base layer image data

[0658] Get the base layer data from the camera image acquisition pipeline:

[0659] This process can be achieved by collecting different brightness and darkness features of image data from a camera that is different from high dynamic range image data, or basic layer image data in a format that adapts to the display characteristics of more existing devices.

[0660] Obtain base layer image data from other sources:

[0661] This process can be to obtain high-quality basic layer data through professional acquisition equipment, and then transfer it to the processing equipment by sending or copying.

[0662] Obtain base layer image data from high dynamic range data:

[0663] This process can be to obtain basic layer image data through tone mapping and neural network.

[0664] It should be noted that the present invention does not limit the method of obtaining the basic layer data.

[0665] It should be noted that the present invention does not limit the base layer image data format of the 2001 Institute. In terms of color space, it can be YUV or RGB. In terms of data bit width, it can be 8-bit, 10-bit, or 12-bit, etc. The photoelectric conversion characteristics can be PQ, gamma, log, HLG, etc.

[0666] Encode, decode and / or process the base layer image base[i] to obtain the second base layer image data baseAfter[i]

[0667] Get the intermediate base layer image data:

[0668] Option 1:

[0669] Using the base layer image directly as the intermediate base layer image data;

[0670] Option 2:

[0671] The base layer image is encoded to generate a basic bitstream. The encoding can be performed using codecs such as JPEG, HEIF, H.264, and HEVC, which are not limited in the present invention.

[0672] The basic code stream is decoded to obtain the intermediate basic layer image data.

[0673] Process the intermediate base layer image data to obtain the second base layer image data and obtain basic metadata:

[0674] Solution 1: Use the intermediate basic layer image data as the second basic layer image data.

[0675] Solution 2: Obtain the data range (minimum value, and / or maximum value) of the intermediate basic layer image data, map the minimum value of the intermediate basic layer image data to 0 or a preset value, and / or map the maximum value to 1.0 or a preset value, and / or then map the intermediate value to a certain intermediate value according to the mapping relationship between the maximum value and / or the minimum value, thereby obtaining the second basic layer image data.

[0676] Solution three: Obtain a local or global mapping relationship between the intermediate layer data and the high dynamic image data, and use the mapping relationship to map the intermediate basic layer image data to obtain the second basic layer image data.

[0677] Obtain the local or global mapping relationship between middle-layer data and highly dynamic data

[0678] Acquire multiple characteristic brightness values ​​of the high dynamic range image, and acquire basic layer image data corresponding to pixel positions of the characteristic brightness values.

[0679] For example, a histogram of a high-dynamic image is obtained, and the brightness of the peak position in the histogram is obtained as the brightness feature value.

[0680] For example, the average brightness value of a specific area (human face, green plants, blue sky) in a high-dynamic image is obtained as the feature brightness value.

[0681] Obtain mapping relationship based on feature brightness value and corresponding basic layer image data

[0682] The mapping relationship has various forms, such as Sigmoid, cubic spline, gamma, straight line, etc. or their inverse function forms, which are not limited in the present invention. The following curves can be used:

[0683] It should be noted that L and L' can be normalized optical signals or electrical signals, which is not limited in the present invention.

[0684] After mapping the intermediate base layer image data using the mapping relationship TMB(), the second base layer image data is obtained: baseAfter[i]=TMB(base[i]), or baseAfter[i]=TMB()*base[i]

[0685] It should be noted that the present invention does not limit baseAfter[i] and base[i] to any domain, and can be a linear domain, PQ domain, log domain, etc. The present invention also does not limit the color space of baseAfter[i] and base[i], and can be a color space such as YUV, RGB, Lab, HSV, etc.

[0686] It should be noted that color gamut mapping can be added before or after processing to convert from the current color gamut to the target color gamut. The current color gamut and the target color gamut include but are not limited to BT.2020, BT.709, DCI-P3, sRGB, etc.

[0687] Encode basic metadata into the code stream.

[0688] Get the enhancement data enhanceAfter and enhancement metadata, and encode the enhancement data and enhancement data into the code stream

[0689] Get intermediate enhancement data based on the high dynamic range image and the second base layer data:

[0690] Solution 1: Intermediate enhancement data = high dynamic range data / f(baseAfter[i]), where f() is a conversion function of the numerical domain, such as log, OETF or EOTF, piecewise curve, etc., and may also include processing of the second basic layer data (obtaining the data range (minimum value and / or maximum value) of the second basic layer data, mapping the minimum value of the second basic layer data to 0 or a preset value, and / or mapping the maximum value to 1.0 or a preset value, and / or mapping the intermediate value to a certain intermediate value based on the mapping relationship between the maximum value and / or the minimum value.)

[0691] .

[0692] Solution 2: Intermediate enhanced data = high dynamic range data - f(baseAfter[i]), where f() is a conversion function of the numerical domain, such as log, OETF or EOTF, segmented curve, etc., and may also include processing of the second basic layer data (obtaining the data range (minimum value and / or maximum value) of the second basic layer data, mapping the minimum value of the second basic layer data to 0 or a preset value, and / or mapping the maximum value to 1.0 or a preset value, and / or mapping the intermediate value to a certain intermediate value according to the mapping relationship between the maximum value and / or the minimum value.).

[0693] Downsample the intermediate enhanced data to obtain the second intermediate enhanced data enhance:

[0694] Solution 1: No downsampling, direct: Second intermediate enhanced data = intermediate enhanced data

[0695] Solution 2: Pass an index or pass an index for each area, and select a better interpolation method from a preset set. The preset set can be GL_NEAREST (nearest neighbor) and GL_LINEAR (nearest neighbor) of OPENGL texture interpolation methods; it can also be NEAREST (nearest neighbor) or LINEAR (nearest neighbor) of Vulkan texture interpolation methods; it can also be passing multiple sets of filters, or a method containing multiple directional interpolation methods, or bicubic splines; it can also be a full set or subset of all the above methods. Select the method that minimizes the difference between the second intermediate enhancement data after downsampling and upsampling using this method and the original intermediate enhancement data, and add this method to the enhancement metadata.

[0696] Option 2: Select a default texture interpolation method from OpenGL, Vulkan, or Matl. This can be GL_NEAREST (nearest neighbor) or GL_LINEAR (nearest neighbor) from OpenGL, or NEAREST (nearest neighbor) or LINEAR (nearest neighbor) from Vulkan. Select the method that minimizes the difference between the second intermediate enhancement data after downsampling and upsampling using this method and the original intermediate enhancement data, and add this method to the enhancement metadata.

[0697] Solution 3: Use a pre-set filter or a combination of directional interpolation methods, such as bicubic splines. Select the method that minimizes the difference between the second intermediate enhancement data after downsampling and upsampling using this method and the original intermediate enhancement data, and add this method to the enhancement metadata.

[0698] Process the second intermediate enhanced data enhance to obtain the final enhanced data

[0699] Solution 1: Obtain the data range (minimum value, sum, or maximum value) of the second intermediate enhanced data, map the minimum value of the second intermediate enhanced data to 0 or a preset value, and / or map the maximum value to 1.0 or a preset value, and then map the intermediate value to a certain intermediate value according to the mapping relationship between the maximum value and / or the minimum value, thereby obtaining the final enhanced data.

[0700] Solution 2: Get the histogram of the second intermediate enhanced data, use the histogram equalization method to obtain the mapping relationship TME(), and then use the mapping function enhanceAfter[i]=TMB(enhance[i]) or enhanceAfter[i]=TMB()*enhance[i].

[0701] The mapping relationship has various forms, including Sigmoid, cubic spline, gamma, straight line, piecewise curve, etc. or their inverse function forms, which are not limited in the present invention. The following curves can be used:

[0702] L^'=F(L)=a×((p×L^n) / ((p-1)×L^n+1))^m+b

[0703] It should be noted that L and L' can be normalized optical signals or electrical signals, which is not limited in the present invention.

[0704] It should be noted that the present invention does not limit the format of metadata. Metadata can include histogram information and tone-mapping curve parameter information as in ST2094-40, or can include tone-mapping curve parameter information as in ST2094-10.

[0705] It should be noted that the present invention does not limit enhanceAfter[i] and enhance[i] to any domain, which can be a linear domain, PQ domain, log domain, etc. The present invention also does not limit the color space of enhanceAfter[i] and enhance[i], which can be YUV, RGB, Lab, HSV, etc.

[0706] It should be noted that color gamut mapping can be added before or after processing to convert from the current color gamut to the target color gamut. The current color gamut and the target color gamut include but are not limited to BT.2020, BT.709, DCI-P3, sRGB, etc.

[0707] Encode final enhancement data and enhancement metadata

[0708] For encoding the enhanced data, codecs such as JPEG, HEIF, H.264, and HEVC can be used, and this invention does not limit this.

[0709] Enhancement metadata can be placed in non-coded pixel segments such as SEI and APPN.

[0710] This application is compatible with image encoding and processing in multiple formats, and can achieve relatively good image quality on systems with different support capabilities. This application proposes an encoding method that is compatible with code streams in multiple formats, providing compatibility with systems with different capabilities.

[0711] In the embodiments of the present application, the symbol * and the symbol × can be interchanged.

[0712] The technical solution of the present application can use a mapping function to process the base image during the transformation processing of the base image according to the metadata; can use a mapping function to process the enhanced data during the transformation processing of the enhanced data according to the metadata; can perform up / down sampling during the transformation processing of the enhanced data according to the metadata, use the interpolation algorithm in the texture sampling algorithm in OPENGL, Vulkan, or Metal for processing, or use an index to select a better algorithm from a set of interpolation algorithms.

[0713] The present application provides an electronic device that includes various modules for executing the above-mentioned encoding method. For example, it includes an encoding module, a processing module, a sending module, etc. The decoding module and the processing module can be implemented by a processor, and the sending module can be implemented by a transmitter.

[0714] The following describes a codec system used in the present application in conjunction with FIG1A . FIG1A is a schematic block diagram of a codec system used in an embodiment of the present application, such as a video codec system 10 (or simply codec system 10) that can utilize the techniques of the present application. The video encoder 20 (or simply encoder 20) and video decoder 30 (or simply decoder 30) in the video codec system 10 represent devices that can be used to perform various techniques according to the various examples described in this application.

[0715] As shown in FIG. 1A , a coding and decoding system 10 includes a source device 12 , which is configured to provide coded data 21 such as coded images to a destination device 14 for decoding the coded data.

[0716] The source device 12 includes an encoder 20 and, optionally, may further include an image source 16 , an image preprocessor 18 (or a preprocessing unit), and a communication interface or communication unit 22 .

[0717] The image source 16 may include or may be any type of image capture device for capturing real-world images, etc., and / or any type of image generation device, such as a computer graphics processor for generating computer-animated images, or any type of device for acquiring and / or providing real-world images, computer-generated images (e.g., screen content, virtual reality (VR) images, and / or any combination thereof (e.g., augmented reality (AR) images). The image source may be any type of memory or storage that stores any of the above images.

[0718] In order to distinguish the processing performed by the pre-processor 18 or the pre-processing unit 18 , the image or image data 17 may also be referred to as a raw image or raw image data 17 .

[0719] The preprocessor 18 is configured to receive (raw) image data 17 and preprocess the image data 17 to obtain a preprocessed image 19 or preprocessed image data 19. For example, the preprocessing performed by the preprocessor 18 may include cropping, color format conversion (e.g., from RGB to YCbCr), color grading, or denoising. It will be appreciated that the preprocessor 18 may be an optional component.

[0720] The video encoder 20 is configured to receive the pre-processed image data 19 and provide encoded image data 21 .

[0721] Communication interface 22 in source device 12 may be used to receive encoded image data 21 and transmit the encoded image data 21 (or any other processed version thereof) to another device, such as destination device 14, or any other device, via communication channel 13 for storage or direct reconstruction.

[0722] Destination device 14 includes a decoder 30 (eg, video decoder 30 ) and, in addition and optionally, may include a communication interface or communication unit 28 , a post-processor 32 (or post-processing unit 32 ), and a display device 34 .

[0723] The communication interface 28 in the destination device 14 is used to receive the encoded image data 21 (or any other processed version) directly from the source device 12 or from any other source device such as a storage device, for example, a storage device that is an encoded image data storage device, and provide the encoded image data 21 to the decoder 30.

[0724] The communication interface 22 and the communication interface 28 may be used to send or receive the encoded image data 21 or the encoded data via a direct communication link between the source device 12 and the destination device 14, such as a direct wired or wireless connection, or via any type of network, such as a wired network, a wireless network, or any combination thereof, any type of private network and public network, or any combination thereof.

[0725] For example, the communication interface 22 may be used to encapsulate the encoded image data 21 into a suitable format such as a message, and / or process the encoded image data using any type of transmission coding or processing for transmission over a communication link or network.

[0726] The communication interface 28 corresponds to the communication interface 22 , for example, and can be used to receive transmission data and process the transmission data using any type of corresponding transmission decoding or processing and / or decapsulation to obtain encoded image data 21 .

[0727] Both the communication interface 22 and the communication interface 28 can be configured as a unidirectional communication interface as indicated by the arrow pointing from the source device 12 to the corresponding communication channel 13 of the destination device 14 in Figure 1A, or a bidirectional communication interface, and can be used to send and receive messages, etc. to establish a connection, confirm and exchange any other information related to the communication link and / or data transmission such as encoded image data transmission, etc.

[0728] The decoder 30 is configured to receive the encoded image data 21 and provide decoded image data 31 or a decoded image 31 .

[0729] The post-processor 32 in the destination device 14 is configured to post-process the decoded image data 31 (also referred to as reconstructed image data), such as the decoded image 31, to obtain post-processed image data 33, such as the post-processed image 33. The post-processing performed by the post-processing unit 32 may include, for example, color format conversion (e.g., from YCbCr to RGB), color grading, cropping, or resampling, or any other processing for generating the decoded image data 31 for display on a display device 34, etc.

[0730] A display device 34 in the destination device 14 is configured to receive the post-processed image data 33 and display the image to a user or viewer. The display device 34 may be or include any type of display for displaying the reconstructed image, such as an integrated or external display screen or monitor. For example, the display screen may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro-LED display, a liquid crystal on silicon (LCoS) display, a digital light processor (DLP), or any other type of display screen.

[0731] Although FIG1A shows source device 12 and destination device 14 as separate devices, device embodiments may also include both source device 12 and destination device 14 or the functionality of both source device 12 and destination device 14, that is, both source device 12 or the corresponding functionality and destination device 14 or the corresponding functionality. In these embodiments, source device 12 or the corresponding functionality and destination device 14 or the corresponding functionality may be implemented using the same hardware and / or software or through separate hardware and / or software, or any combination thereof.

[0732] Based on the description, it will be apparent to those skilled in the art that the existence and (precise) division of different units or functions in the source device 12 and / or the destination device 14 shown in FIG. 1A may vary depending on actual devices and applications.

[0733] The content providing system of the content distribution service used in the present application is described below in conjunction with Figure 2A. Figure 2A is a block diagram of a content providing system for implementing the content distribution service used in an embodiment of the present application. The content providing system 2100 includes a capture device 2102, a terminal device 2106, and an (optional) display 2126. The capture device 2102 communicates with the terminal device 2106 via a communication link 2104. The communication link may include the above-mentioned communication channel 13. The communication link 2104 includes but is not limited to WiFi, Ethernet, wired, wireless (3G / 4G / 5G), USB, or any combination thereof.

[0734] Capture device 2102 generates data and can use the coding method shown in the above embodiment to encode the data. Alternatively, capture device 2102 can distribute data to a streaming media server (not shown), which encodes the data and transmits the encoded data to terminal device 2106. Capture device 2102 includes but is not limited to a camera, a smart phone or a tablet computer, a computer or a notebook computer, a video conferencing system, a PDA, a vehicle-mounted device, or any combination thereof. For example, capture device 2102 can include above-mentioned source device 12. When data includes video, the video encoder 20 in capture device 2102 can actually perform video encoding. When data includes audio (i.e., sound), the audio encoder 20 in capture device 2102 can actually perform audio encoding. In some actual scenarios, capture device 2102 distributes encoded video data and encoded audio data by multiplexing the encoded video data and the encoded audio data together. In other actual scenarios, for example, in a video conferencing system, the encoded audio data and the encoded video data are not multiplexed. Capture device 2102 distributes the encoded audio data and the encoded video data to terminal device 2106 respectively.

[0735] The terminal device 2106 in the content providing system 2100 receives and regenerates the encoded data. The terminal device 2106 can be a device with data reception and recovery capabilities, such as a smartphone or tablet computer 2108, a computer or laptop computer 2110, a network video recorder (NVR) / digital video recorder (DVR) 2112, a television 2114, a set-top box (STB) 2116, a video conferencing system 2118, a video surveillance system 2120, a personal digital assistant (PDA) 2122, an in-vehicle device 2124, or any combination thereof, or such devices capable of decoding the encoded data. For example, the terminal device 2106 can include the destination device 14 described above. When the encoded data includes video, the video decoder 30 in the terminal device prioritizes video decoding. When the encoded data includes audio, the audio decoder in the terminal device prioritizes audio decoding. The terminal device 2106 may be a video playback application, a streaming media playback application, a streaming media playback platform, a live broadcast platform, etc. running on the terminal device.

[0736] For end devices with displays, such as smartphones or tablets 2108, computers or laptops 2110, NVRs / DVRs 2112, televisions 2114, PDAs 2122, or in-vehicle devices 2124, the end device can send the decoded data to its display. For end devices without displays, such as STBs 2116, video conferencing systems 3118, or video surveillance systems 2120, the device can be connected to an external display 2126 to receive and display the decoded data.

[0737] When performing encoding or decoding, each device in this system may use the image encoding device or image decoding device shown in the above embodiments.

[0738] FIG2B is a schematic diagram of an example structure of the terminal device 2106 in FIG2A . After the terminal device 2106 receives the bitstream from the capture device 2102, the protocol processing unit 2202 analyzes the transmission protocol of the bitstream. The protocol includes, but is not limited to, Real Time Streaming Protocol (RTSP), Hyper Text Transfer Protocol (HTTP), HTTP Live Streaming Protocol (HLS), MPEG Dynamic Adaptive Streaming over HTTP (MPEG-DASH), Real-time Transport Protocol (RTP), Real Time Messaging Protocol (RTMP), or any combination thereof.

[0739] After processing the stream, protocol processing unit 2202 generates a stream file. This file is output to demultiplexing unit 2204. Demultiplexing unit 2204 can separate the multiplexed data into coded audio data and coded video data. As mentioned above, in other practical scenarios, such as video conferencing systems, coded audio data and coded video data are not multiplexed. In this case, the coded data is not transmitted to video decoder 3206 and audio decoder 2208 via demultiplexing unit 2204.

[0740] Through the demultiplexing process, a video elementary stream (ES), an audio ES, and optional subtitles are generated. The video decoder 2206, including the video decoder 30 described in the above embodiment, decodes the video ES by the decoding method shown in the above embodiment to generate video frames, and sends the data to the synchronization unit 2212. The audio decoder 2208 decodes the audio ES to generate audio frames, and sends the data to the synchronization unit 2212. Alternatively, the video frames can be stored in a buffer (not shown in the figure) before being sent to the synchronization unit 2212. Similarly, the audio frames can be stored in a buffer (not shown in the figure) before being sent to the synchronization unit 2212.

[0741] The synchronization unit 2212 synchronizes the video frames and audio frames and provides the video / audio to the video / audio display 3214. For example, the synchronization unit 2212 synchronizes the presentation of the video and audio information. The information can be encoded in the syntax using the timestamps associated with the representation of the coded audio and visual data and the timestamps associated with the transmission of the data stream.

[0742] If subtitles are included in the bitstream, the subtitle decoder 2210 decodes the subtitles, synchronizes them with the video frames and audio frames, and provides the video / audio / subtitles to the video / audio / subtitle display 2216.

[0743] The present invention is not limited to the above-mentioned system. The image encoding device or the image decoding device in the above-mentioned embodiments can be used in other systems such as automobiles.

[0744] The following describes a streaming media system applicable to an embodiment of the present application in conjunction with Figure 3A. Figure 3A is a schematic diagram of a workflow of a streaming media system applicable to an embodiment of the present application.

[0745] The streaming media system includes a content creation module that generates the required content data, such as video or audio. The streaming media system also includes a video encoding module that encodes the generated content through an encoder. The streaming media system also includes a video stream transmission module that transmits the encoded video in the form of a code stream. Optionally, the format of the video stream can be converted into a code stream format of a commonly used transmission protocol for OTT (over-the-top) devices, for example, the protocol includes but is not limited to Real Time Streaming Protocol (RTSP), Hyper Text Transfer Protocol (HTTP), HTTP Live streaming protocol (HLS), MPEG HTTP Dynamic Adaptive Streaming over HTTP (MPEG-DASH), Real-time Transport Protocol (RTP), Real Time Messaging Protocol (RTMP) or any combination thereof. Optionally, the video stream can be stored to store the original format of the video stream and / or the converted multiple code stream formats for easy use. Furthermore, the streaming system also includes a video stream encapsulation module for encapsulating the video stream to generate an encapsulated video stream. The encapsulated video stream can be referred to as a video streaming package. Exemplarily, the video streaming package can be generated based on a transcoded video stream or a stored video stream. Furthermore, the streaming system also includes a content distribution network (CDN) for distributing the video streaming package to multiple OTT devices, such as mobile phones, computers, tablets, and home projectors.

[0746] It should be noted that video encoding, video streaming transmission, video stream transcoding, video stream storage, video streaming package generation and content distribution network can all be implemented on cloud servers.

[0747] An exemplary streaming media system architecture of the present application is described below in conjunction with FIG3B . The streaming media system architecture includes: a client device, a content distribution network, and a cloud server.

[0748] The user on the client device sends a play or playback request to the cloud platform. Optionally, the content of the request can be the title of the movie or TV program to be played.

[0749] The cloud platform makes a decision and responds to the client, sending the client the address of the requested content on the CDN. Optionally, the content sent to the client can be a URL link (uniform resource locator). Specifically, the playback application service in the cloud platform checks user authorization and permissions, and then considers individual client characteristics and current network conditions to determine which specific files are needed to process the playback request. It should be noted that the content delivery network (CDN) regularly reports its operating status, learned routes, and available content (files) to the cache control service in the cloud platform.

[0750] The client then requests the CDN to play the content based on the address, and the CDN provides the content to the client, ultimately completing the client's request.

[0751] The following describes the system architecture applicable to the embodiment of the present application in conjunction with Figure 4A. Figure 4A is a schematic diagram of a possible system architecture applicable to the embodiment of the present application. The system architecture of the embodiment of the present application includes: front-end equipment, transmission links, and terminal display equipment.

[0752] Among them, the front-end equipment is used to capture or produce HDR / SDR content (for example, HDR / SDR video or images).

[0753] In a possible embodiment, the front-end device can also be used to extract corresponding metadata from the HDR content. The metadata may include global mapping information, local mapping information, and dynamic metadata and static metadata corresponding to the HDR content. The front-end device can send the HDR content and metadata to the terminal display device via a transmission link. Specifically, the HDR content and metadata can be transmitted in the form of one data packet, or respectively in two data packets, which is not specifically limited in the embodiment of the present application.

[0754] Optionally, the terminal display device can be used to receive metadata and HDR content, and extract the global mapping information, local mapping information, and the terminal display device information contained in the corresponding metadata according to the HDR content, obtain a mapping curve to perform global tone mapping and local tone mapping on the HDR content, convert it into display content adapted for the HDR display device or SDR device in the terminal display device, and display it. It should be understood that in different embodiments, the terminal display device may include a display device with a display capability of a lower dynamic range or a higher dynamic range than the HDR content generated by the front-end device, and this application is not limited thereto.

[0755] Optionally, the front-end device and the terminal display device in this application can be independent and different physical devices. For example, the front-end device can be a video capture device or a video production device, where the video capture device can be a video camera, a camera, an image rendering machine, etc. The terminal display device can be a device with video playback function, such as virtual reality (VR) glasses, a mobile phone, a tablet, a television, a projector, etc.

[0756] Optionally, the transmission link between the front-end device and the terminal display device can be a wireless connection or a wired connection, wherein the wireless connection can adopt technologies such as long term evolution (LTE), fifth generation (5G) mobile communications, and future mobile communications. Wireless connections can also include wireless fidelity (WiFi), Bluetooth, near field communication (NFC), and other technologies. Wired connections can include Ethernet connections, local area network connections, etc. There is no specific limitation on this.

[0757] The present application may also integrate the functions of the front-end device and the terminal display device into the same physical device, for example, a terminal device such as a mobile phone or tablet with a video capture function. The present application may also integrate some functions of the front-end device and some functions of the terminal display device into the same physical device. This is not specifically limited.

[0758] The following describes an end-to-end image processing system provided by an embodiment of the present application in conjunction with Figure 4B. This system can be applied to the system architecture shown in Figure 4A. Figure 4B is a schematic diagram of the structure of an image processing system provided by an embodiment of the present application. In Figure 3B, HDR / SDR content is exemplified by HDR video. The image processing system includes: an HDR preprocessing module, an HDR video encoding module, an HDR video decoding module, and a tone mapping module.

[0759] Among them, the HDR preprocessing module and the HDR video encoding module can be located in the front-end device shown in Figure 4A, and the HDR video decoding module and the tone mapping module can be located in the terminal display device shown in Figure 4A.

[0760] HDR pre-processing module: This module is used to extract dynamic metadata (e.g., maximum, minimum, average, and range of brightness) from HDR video, determine mapping curve parameters based on the dynamic metadata and the display capabilities of the target display device, write the mapping curve parameters into the dynamic metadata to obtain HDR metadata, and transmit the metadata. HDR video can be either captured or processed by a colorist; the display capabilities of the target display device are the brightness range that the target display device can display.

[0761] HDR video encoding module: used to encode HDR video and HDR metadata according to the video compression standard (for example, AVS or HEVC standard) (for example, embedding HDR metadata into the user-defined part of the bitstream) and output the corresponding bitstream (AVS or HEVC bitstream).

[0762] HDR video decoding module: used to decode the generated bitstream (AVS bitstream or HEVC bitstream) according to the standard corresponding to the bitstream format, and output the decoded HDR video and HDR metadata.

[0763] Tone mapping module: used to generate a mapping curve according to the parameters of the mapping curve in the decoded HDR metadata, and perform tone mapping on the decoded HDR video (i.e. HDR adaptation processing or SDR adaptation processing), and send the HDR adapted video after tone mapping to the HDR display terminal for display, or send the SDR adapted video to the SDR display terminal for display.

[0764] Exemplarily, the HDR pre-processing module may exist in a video acquisition device or a video production device.

[0765] Exemplarily, the HDR video encoding module may exist in a video acquisition device or a video production device.

[0766] Exemplarily, the HDR video decoding module may be present in a set-top box, a television display device, a mobile terminal display device, and a video conversion device for live broadcasting, online video applications, and the like.

[0767] For example, the tone mapping module may be present in a set-top box, a television display device, a mobile terminal display device, or a video conversion device for a live webcast or online video application. More specifically, the tone mapping module may be present in the form of a chip or software program in the set-top box, television display, or mobile terminal display, and may be present in the form of a software program in the video conversion device for a live webcast or online video application.

[0768] In a possible embodiment, when the tone mapping module and the HDR video decoding module are both present in a set-top box, the set-top box can complete the functions of receiving, decoding, and tone mapping the video stream. The set-top box sends the decoded video data to a display device through a high-definition multimedia interface (HDMI) for display, so that the user can enjoy the video content.

[0769] The present application also provides an image coding apparatus 1300. See Figure 13, which illustrates an example of an image coding apparatus provided in the present application. Apparatus 1300 is capable of implementing the functions of the front-end device described in Figure 3A. For example, the functions, units, or means may be implemented via software or hardware, or the corresponding software implementation may be executed by hardware.

[0770] For example, referring to FIG. 13 , the apparatus 1300 may include:

[0771] Coding module 1301 is used to write an image to be displayed into a code stream; it is also used to write reference information of the image to be displayed into the code stream; the reference information includes at least one of a reference display device size, a reference viewing distance, a reference viewing angle, a reference ambient light brightness, or a reference display device resolution; the reference information is used to perform tone mapping on the image to be displayed.

[0772] The sending module 1302 is configured to send the code stream.

[0773] In a possible embodiment, the encoding module is further used to: write reference tone mapping parameters of the image to be displayed into the code stream; wherein the reference tone mapping parameters are used to obtain initial tone mapping parameters; the reference information is used by the decoding end to adjust the initial tone mapping parameters to obtain tone mapping parameters, and the tone mapping parameters are used to tone map the image to be displayed.

[0774] In a possible embodiment, writing the reference information of the image to be displayed into the code stream includes: writing the reference information into the code stream; or writing an index of the reference information into the code stream.

[0775] In a possible embodiment, the encoding module is further used to: write one or more brightness characteristic values ​​of the image to be displayed into the code stream; wherein the one or more brightness characteristic values ​​are used by the decoding end to adjust the initial tone mapping parameters to obtain the tone mapping parameters.

[0776] It should be understood that all relevant contents of each step involved in the embodiment of the above-mentioned image coding method can be referred to the functional description of the corresponding functional module, and will not be repeated here.

[0777] The present application also provides an electronic device 1300A, see Figure 13A . The electronic device 1300 has the functions of the front-end device in Figure 3A . For example, the functions, units, or means may be implemented by software, hardware, or hardware executing the corresponding software implementation.

[0778] For example, referring to FIG. 13A , an electronic device 1300A may include:

[0779] a processing module, configured to obtain, based on a first image, a first base image, a first gain map, and metadata, wherein the first base image corresponds to a first dynamic range, the first image corresponds to a second dynamic range, and the second dynamic range is different from the first dynamic range;

[0780] An encoding module, configured to encode the first base image, the first gain map, and metadata according to the second aspect or any possible implementation of the second aspect to obtain a code stream

[0781] Optionally, the electronic device 1300A further includes a sending module for sending the code stream.

[0782] The present invention also provides an electronic device 1400 , as shown in FIG14 , which is an example diagram of an electronic device provided in the present invention, and is used to implement the method in the embodiment shown in FIG4 .

[0783] As shown in FIG14 , the electronic device 1400 may include a processor 1401 for executing programs or instructions stored in a memory 1402 . When the programs or instructions stored in the memory 1402 are executed, the processor is used to execute the method in the aforementioned embodiments of the present application.

[0784] Optionally, the electronic device 1400 may further include a communication interface 1403. FIG14 uses dashed lines to indicate that the communication interface 1403 is optional for the electronic device 1400.

[0785] Among them, the number of processors 1401, memories 1402 and communication interfaces 1403 does not constitute a limitation on the embodiments of the present application. During specific implementation, they can be arbitrarily configured according to business requirements.

[0786] Optionally, the memory 1402 is located outside the electronic device 1400 .

[0787] Optionally, the electronic device 1400 includes the memory 1402, which is connected to the at least one processor 1401 and stores instructions executable by the at least one processor 1401. FIG14 uses dashed lines to indicate that the memory 142 is optional for the electronic device 1400.

[0788] The processor 1401 and the memory 1402 may be coupled via an interface circuit or may be integrated together, which is not limited here.

[0789] The specific connection medium between the processor 1401, memory 1402, and communication interface 1403 is not limited in the embodiments of the present application. In Figure 14, the processor 1401, memory 1402, and communication interface 1403 are connected via a bus 1404. The bus is represented by a bold line in Figure 14. The connection between other components is only for schematic illustration and is not intended to be limiting. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one bold line is used in Figure 14, but this does not mean that there is only one bus or one type of bus.

[0790] An embodiment of the present application further provides an electronic device 1500 , see FIG15 , which is an example diagram of an electronic device provided in an embodiment of the present application, and is used to implement the image coding method in an embodiment of the present application.

[0791] As shown in FIG15 , the electronic device 1500 may include a processor 1501 for executing a program or instruction stored in a memory 1502 . When the program or instruction stored in the memory 1502 is executed, the processor 1501 is used to execute the image encoding method in the embodiment shown in FIG11 .

[0792] Optionally, the electronic device 1500 may further include a communication interface 1503. FIG15 uses a dotted line to indicate that the communication interface 1503 is optional for the electronic device 1500.

[0793] Among them, the number of processors 1501, memories 1502 and communication interfaces 1603 does not constitute a limitation on the embodiments of the present application. During specific implementation, they can be arbitrarily configured according to business requirements.

[0794] Optionally, the memory 1502 is located outside the electronic device 1500 .

[0795] Optionally, the electronic device 1500 includes the memory 1502, which is connected to the at least one processor 1501 and stores instructions executable by the at least one processor 1501. FIG15 uses dotted lines to indicate that the memory 1502 is optional for the electronic device 1500.

[0796] The processor 1501 and the memory 1502 may be coupled via an interface circuit or may be integrated together, which is not limited here.

[0797] The specific connection medium between the processor 1501, memory 1502, and communication interface 1503 is not limited in the embodiments of the present application. In Figure 15, the processor 1501, memory 1502, and communication interface 1503 are connected via a bus 1504. The bus is represented by a bold line in Figure 15. The connection between other components is only for schematic illustration and is not intended to be limiting. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one bold line is used in Figure 15, but this does not mean that there is only one bus or one type of bus.

[0798] It should be understood that the processors mentioned in the embodiments of the present application can be implemented by hardware or software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented by software, the processor can be a general-purpose processor that is implemented by reading software code stored in a memory.

[0799] Exemplarily, the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0800] It should be understood that the memory mentioned in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0801] It should be noted that when the processor is a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, the memory (storage module) can be integrated into the processor.

[0802] It should be noted that the memory described herein is intended to include, but not be limited to, these and any other suitable types of memory.

[0803] An embodiment of the present application also provides a computer-readable storage medium, including a program or instruction. When the program or instruction is run on a computer, the method in the embodiment shown in Figure 4 or the image encoding method in the embodiment shown in Figure 12 is executed.

[0804] The term "at least one" in the embodiments of the present application refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where any of a, b, c, ab, ac, bc, or abc can include a single a, a single b, a single c, and can also include multiple a, multiple b, and multiple c.

[0805] Furthermore, unless otherwise indicated, ordinal numbers such as "first" and "second" in the embodiments of this application are used to distinguish between multiple objects and are not used to define the order, timing, priority, or importance of multiple objects. For example, the first priority criterion and the second priority criterion are only used to distinguish different criteria and do not indicate differences in the content, priority, or importance of the two criteria.

[0806] In addition, the terms "including" and "having" in the embodiments, claims, and drawings of this application are not exclusive. For example, a process, method, system, product, or device that includes a series of steps or modules / units is not limited to the listed steps or modules and may also include steps or modules / units that are not listed.

[0807] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0808] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each flow and / or box in the flow chart and / or block diagram, as well as the combination of the flow chart and / or box in the flow chart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more flow charts and / or one or more boxes in the block diagram.

[0809] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0810] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0811] Obviously, those skilled in the art may make various modifications and variations to this application without departing from the spirit and scope of this application. Thus, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application is intended to include such modifications and variations.

[0812] The present invention also provides the following embodiments. It should be noted that the numbering of the following embodiments does not necessarily need to follow the numbering order of the previous embodiments:

[0813] Embodiment 1. A coding method, characterized in that the method comprises:

[0814] Acquire a first HDR image and first base layer data corresponding to the first HDR image;

[0815] encoding the first base layer data;

[0816] Determining first enhancement layer data according to the first base layer data and the first HDR image;

[0817] encoding the first enhancement layer data;

[0818] First metadata is encoded, where the first metadata includes metadata of the first base layer data and metadata of the first enhancement layer data.

[0819] Embodiment 2. The method according to embodiment 1, wherein determining the first enhancement layer data based on the first base layer data and the first HDR image comprises:

[0820] determining second basic layer data according to the first basic layer data;

[0821] determining the first enhancement layer data according to the second base layer data and the first HDR image,

[0822] According to the concept of rate-distortion optimization, the manner of determining conversion of the first enhancement layer data to the first enhancement data according to the second base layer data and the first HDR image is adjusted so that encoding of the first enhancement layer data meets a rate-distortion performance target.

[0823] Embodiment 3. The method according to embodiment 2, wherein determining the second base layer data based on the first base layer data comprises:

[0824] determining intermediate basic layer data according to the first basic layer data;

[0825] The second basic layer data is determined according to the intermediate basic layer data.

[0826] Embodiment 4. The method according to embodiment 3, wherein determining the intermediate base layer data based on the first base layer data comprises:

[0827] The intermediate base layer data is determined to be the first base layer data.

[0828] Embodiment 5. The method according to embodiment 3, wherein determining the intermediate base layer data based on the first base layer data comprises:

[0829] Encoding the first base layer data to obtain first base layer coding information;

[0830] The first base layer coding information result is decoded to obtain first base layer decoding information, and the intermediate base layer data is the first base layer decoding information.

[0831] Embodiment 6. The method according to any one of Embodiments 3 to 5, wherein determining the second base layer data based on the intermediate base layer data comprises:

[0832] The second basic layer data is determined to be the intermediate basic layer data.

[0833] Embodiment 7. The method according to any one of Embodiments 3 to 5, wherein determining the second base layer data based on the intermediate base layer data comprises:

[0834] At least one base layer characteristic value is mapped to at least one base layer reference value respectively to obtain the second base layer data, wherein the second base layer data includes the at least one base layer reference value and the values ​​of pixels in the intermediate base layer data except the at least one base layer characteristic value, and the at least one base layer characteristic value includes at least one of the following values: the maximum value, the minimum value, or the intermediate value of the pixels of the intermediate base layer data.

[0835] Embodiment 8. The method according to any one of Embodiments 3 to 5, wherein determining the second base layer data based on the intermediate base layer data comprises:

[0836] determining at least one characteristic brightness value based on the first HDR image;

[0837] Determining at least one pixel from the intermediate base layer data, the at least one pixel corresponding to the at least one characteristic brightness value, and a position of each pixel in the at least one pixel being the same as a position of the corresponding characteristic brightness value;

[0838] Determining a first mapping relationship by fitting function parameters according to a preset or selected mapping function and the at least one characteristic brightness value and the value of the at least one pixel;

[0839] The intermediate base layer data is converted into the second base layer data according to the first mapping relationship.

[0840] Embodiment 9. The method according to embodiment 8, wherein determining at least one characteristic brightness value based on the first HDR image comprises:

[0841] Determine a brightness value at a peak position in a histogram of the first HDR image as the characteristic brightness value.

[0842] Embodiment 10. The method according to embodiment 8, wherein determining at least one characteristic brightness value based on the first HDR image comprises:

[0843] determining at least one reference region in the first HDR image;

[0844] A reference brightness value of each reference area in the at least one reference area is determined to be the characteristic brightness value, where the reference brightness value of each reference area is an average brightness value or a maximum brightness value of each reference area.

[0845] Embodiment 11. The method according to embodiment 8, wherein converting the intermediate base layer data into the second base layer data according to the first mapping relationship comprises:

[0846] The intermediate base layer data and the second base layer data satisfy the following relationship: baseAfter[i]=TMB1(R base[i]_1 ),

[0847] Wherein, baseAfter[i] represents the value of the i-th pixel in the second basic layer data, R base[i]_1 represents the first reference value of the i-th pixel determined according to the value of the i-th pixel in the intermediate basic layer data, and TMB1() represents the first mapping relationship.

[0848] Embodiment 12. The method according to embodiment 8, wherein converting the intermediate base layer data into the second base layer data according to the first mapping relationship comprises:

[0849] The intermediate base layer data and the second base layer data satisfy the following relationship: baseAfter[i]=TMB1(R base[i]_2 )×R base[i]_3 ,

[0850] Wherein, baseAfter[i] represents the value of the i-th pixel in the second basic layer data, TMB1() represents the first mapping relationship, R base[i]_2 is the second reference value of the i-th pixel determined according to the value of the i-th pixel in the intermediate base layer data, R base[i]_3 is a third reference value of the i-th pixel determined according to the value of the i-th pixel in the intermediate base layer data.

[0851] Embodiment 13. The method according to any one of embodiments 2 to 12, wherein determining the first enhancement layer data from the second base layer data and the first HDR image comprises:

[0852] determining first intermediate enhancement data according to the second base layer data and the first HDR image;

[0853] determining second intermediate enhancement data according to the first intermediate enhancement data;

[0854] The first enhancement layer data is determined according to the second intermediate enhancement data.

[0855] Embodiment 14. The method according to embodiment 13, wherein determining first intermediate enhancement data according to the second base layer data and the first HDR image comprises:

[0856] Obtaining a target conversion result according to the second basic layer data and a target numerical conversion function;

[0857] The first intermediate enhancement layer data is determined according to the first HDR image and the target conversion result.

[0858] Embodiment 15. The method according to embodiment 14 is characterized in that the first intermediate enhancement layer data is the quotient of the data of the first HDR image and the target conversion result.

[0859] Embodiment 16. The method according to embodiment 14 is characterized in that the first intermediate enhancement layer data is the difference between the data of the first HDR image and the target conversion result.

[0860] Embodiment 17. The method of any one of embodiments 13 to 16, wherein determining second intermediate enhancement data based on the first intermediate enhancement data comprises:

[0861] The second intermediate enhanced data is determined to be the first intermediate enhanced data.

[0862] Embodiment 18. The method according to any one of embodiments 13 to 16, wherein determining the second intermediate enhancement data based on the first intermediate enhancement data comprises:

[0863] Determine the target interpolation method;

[0864] The first intermediate enhanced data is downsampled using the target interpolation method to obtain the second intermediate enhanced data.

[0865] Embodiment 19. The method according to embodiment 18, wherein determining the target interpolation mode comprises:

[0866] The target interpolation method is determined from multiple interpolation methods, wherein the difference between the recovered data obtained after upsampling the second intermediate enhancement data using the target interpolation method and the first intermediate enhancement data is smaller than the difference between the recovered data obtained after upsampling the reference intermediate enhancement data using the reference interpolation method and the first intermediate enhancement data, wherein the first reference intermediate enhancement data is the intermediate enhancement data obtained after downsampling the first intermediate enhancement data using the reference interpolation method, and the reference interpolation method is any interpolation method among the multiple interpolation methods except the target interpolation method.

[0867] Embodiment 20. The method according to embodiment 18 or 19 is characterized in that the first metadata includes interpolation mode indication information, and the interpolation mode indication information is used to indicate the target interpolation mode.

[0868] Embodiment 21. The method according to any one of embodiments 13 to 20, wherein determining the first enhancement layer data based on the second intermediate enhancement data comprises:

[0869] At least one enhancement layer characteristic value is mapped to at least one enhancement layer reference value respectively to obtain the first enhancement layer data, wherein the first enhancement layer data includes the at least one enhancement layer reference value and the values ​​of pixels in the second intermediate enhancement data except for the at least one enhancement layer characteristic value, and the at least one enhancement layer characteristic value includes at least one of the following values: the maximum value, the minimum value, or the intermediate value of the pixels of the intermediate basic layer data.

[0870] Embodiment 22. The method according to any one of embodiments 13 to 20, wherein determining the first enhancement layer data based on the second intermediate enhancement data comprises:

[0871] determining a histogram of the second intermediate enhanced data;

[0872] performing equalization processing on the histogram of the second intermediate enhanced data to obtain a balanced histogram of the second intermediate enhanced data;

[0873] determining a second mapping relationship according to the histogram of the second intermediate enhanced data and the equalized histogram of the second intermediate enhanced data;

[0874] converting the second intermediate enhancement data into the first enhancement layer data according to the second mapping relationship,

[0875] The second mapping relationship is iteratively optimized and adjusted to reduce the difference between the inversely transformed first enhanced data obtained by inversely transforming the first enhanced layer data and the second intermediate enhanced data.

[0876] Embodiment 23. The method according to embodiment 22, characterized in that converting the second intermediate enhancement data into the first enhancement layer data according to the second mapping relationship includes:

[0877] The second intermediate enhancement layer data and the first enhancement layer data satisfy the following relationship: enhanceAfter[i]=TMB2(R enhance[i]_1 ),

[0878] Where enhanceAfter[i] represents the value of the i-th pixel of the first enhancement layer data, R enhance[i]_1represents the first reference value of the i-th pixel determined according to the value of the i-th pixel of the second intermediate enhancement layer data, and TMB2() represents the second mapping relationship.

[0879] Embodiment 24. The method according to embodiment 22, wherein converting the second intermediate enhancement data into the first enhancement layer data according to the second mapping relationship comprises:

[0880] The second intermediate enhancement layer data and the first enhancement layer data satisfy the following relationship: enhanceAfter[i]=TMB2(R enhance[i]_2 )×R enhance[i]_3 ,

[0881] Wherein, enhanceAfter[i] represents the value of the i-th pixel of the first enhancement layer data, TMB2() represents the second mapping relationship, R enhance[i]_2 is the second reference value of the i-th pixel determined according to the value of the i-th pixel of the second intermediate enhancement layer data, R enhance[i]_3 is a third reference value of the i-th pixel determined according to the value of the i-th pixel of the second intermediate enhancement layer data.

[0882] Embodiment 1. A method for processing a digital signal, comprising:

[0883] Obtaining first base layer data, first enhancement layer data, and first metadata, wherein the first base layer data is base layer data of a first HDR object, the first enhancement layer data is enhancement layer data of the first HDR object, and the first metadata is metadata of the first HDR object;

[0884] determining second basic layer data according to the first basic layer data;

[0885] determining second enhancement layer data according to the first enhancement layer data;

[0886] determining a second HDR object based on the first metadata, the second base layer data, and the second enhancement layer data;

[0887] A third HDR object is determined based on the second HDR object.

[0888] Embodiment 2. The method according to embodiment 1 is characterized in that determining the second basic layer data based on the first basic layer data includes: determining that the second basic layer data is the same as the first basic layer data.

[0889] Embodiment 3. The method according to embodiment 1 is characterized in that determining the second basic layer data based on the first basic layer data includes: transforming the first basic layer data according to the first metadata to obtain the second basic layer data.

[0890] Embodiment 4. The method according to embodiment 3, wherein transforming the first base layer data according to the first metadata to obtain the second base layer data includes:

[0891] The first base layer data is transformed according to at least one reference value carried in the first metadata to obtain the second base layer data.

[0892] Embodiment 5. The method according to embodiment 4 is characterized in that the at least one reference value includes a first reference value and a second reference value, and the transforming the first base layer data according to the at least one reference value carried in the first metadata to obtain the second base layer data includes:

[0893] The first reference value, the second reference value, the first base layer data, and the second base layer data satisfy the following relationship: baseAfter[i]=base[i]×REF1+(A-base[i])×REF2,

[0894] Among them, baseAfter[i] represents the value of the i-th pixel in the second basic layer data, base[i] represents the value of the i-th pixel in the first basic layer data, REF1 represents the first reference value, REF2 represents the second reference value, and A represents the maximum value of multiple pixels included in the first basic layer data.

[0895] Embodiment 6. The method according to embodiment 4, wherein the at least one reference value includes a third reference value, and the transforming the first base layer data according to the at least one reference value carried in the first metadata to obtain the second base layer data includes:

[0896] The third reference value, the first base layer data and the second base layer data satisfy the following relationship: baseAfter[i]=base[i]+REF3,

[0897] Among them, baseAfter[i] represents the value of the i-th pixel in the second basic layer data, base[i] represents the value of the i-th pixel in the first basic layer data, and REF3 represents the third reference value.

[0898] Embodiment 7. The method according to embodiment 4, wherein the at least one reference value includes a fourth reference value, and the transforming the first base layer data according to the at least one reference value carried in the first metadata to obtain the second base layer data includes:

[0899] The fourth reference value, the first base layer data and the second base layer data satisfy the following relationship: baseAfter[i]=REF4+A-base[i],

[0900] Among them, baseAfter[i] represents the value of the i-th pixel in the second basic layer data, base[i] represents the value of the i-th pixel in the first basic layer data, REF4 represents the fourth reference value, and A represents the maximum value of multiple pixels included in the first basic layer data.

[0901] Embodiment 8. The method according to embodiment 3, wherein transforming the first base layer data according to the first metadata to obtain the second base layer data comprises:

[0902] The first base layer data is transformed according to at least one mapping relationship carried in the first metadata to obtain the second base layer data.

[0903] Embodiment 9. The method according to embodiment 8, wherein the at least one mapping relationship includes a first mapping relationship, and the transforming the first base layer data according to the at least one mapping relationship carried in the first metadata to obtain the second base layer data includes:

[0904] The first base layer data and the second base layer data satisfy the following relationship: baseAfter[i]=TMB1(R base[i]_1 ),

[0905] Wherein, baseAfter[i] represents the value of the i-th pixel in the second basic layer data, R base[i]_1 represents the first reference value of the i-th pixel determined according to the value of the i-th pixel in the first basic layer data, and TMB1() represents the first mapping relationship.

[0906] Embodiment 10. The method according to embodiment 8, wherein the at least one mapping relationship includes a first mapping relationship, and the transforming the first base layer data according to the at least one mapping relationship carried in the first metadata to obtain the second base layer data includes:

[0907] The first base layer data and the second base layer data satisfy the following relationship: baseAfter[i]=TMB1(R base[i]_2 )×R base[i]_3 ,

[0908] Wherein, baseAfter[i] represents the value of the i-th pixel in the second basic layer data, TMB1() represents the first mapping relationship, R base[i]_2 is the second reference value of the i-th pixel determined according to the value of the i-th pixel in the first basic layer data, R base[i]_3 is a third reference value of the i-th pixel determined according to the value of the i-th pixel in the first basic layer data.

[0909] Embodiment 11. The method according to embodiment 8, wherein the at least one mapping relationship includes a second mapping relationship and a third mapping relationship, and the transforming the first base layer data according to the at least one mapping relationship carried in the first metadata to obtain the second base layer data includes:

[0910] The first base layer data and the second base layer data satisfy the following relationship: baseAfter[i]=TMB3(TMB2(R base[i]_4 )),

[0911] Wherein, baseAfter[i] represents the value of the i-th pixel in the second basic layer data, R base[i]_4 represents the fourth reference value of the i-th pixel determined according to the value of the i-th pixel in the first basic layer data, TMB2() represents the second mapping relationship, and TMB3() represents the third mapping relationship.

[0912] Embodiment 12. The method according to embodiment 8, wherein the at least one mapping relationship includes a second mapping relationship and a third mapping relationship, and the transforming the first base layer data according to the at least one mapping relationship carried in the first metadata to obtain the second base layer data includes:

[0913] The first base layer data and the second base layer data satisfy the following relationship: baseAfter[i]=TMB3(R base[i]_6 )×TMB2(R base[i]_5 ),

[0914] Wherein, baseAfter[i] represents the value of the i-th pixel in the second basic layer data, R base[i]_5 represents the fifth reference value of the i-th pixel determined according to the value of the i-th pixel in the first basic layer data, R base[i]_6represents the sixth reference value of the i-th pixel determined according to the value of the i-th pixel in the first basic layer data, TMB2() represents the second mapping relationship, and TMB3() represents the third mapping relationship.

[0915] Embodiment 13. The method according to embodiment 11 or 12 is characterized in that the third mapping relationship is a global tone mapping function, or the third mapping relationship is a local tone mapping function.

[0916] Embodiment 14. The method according to any one of embodiments 8 to 13 is characterized in that the at least one mapping relationship includes at least one functional relationship.

[0917] Embodiment 15. The method according to embodiment 3, wherein transforming the first base layer data according to the first metadata to obtain the second base layer data comprises:

[0918] The first base layer data is transformed according to at least one filter information carried in the first metadata to obtain the second base layer data.

[0919] Embodiment 16. The method according to embodiment 15, wherein the at least one filter information includes first filter information, and the transforming the first base layer data according to the at least one filter information carried in the first metadata to obtain the second base layer data includes:

[0920] filtering the first base layer data using a first filter to obtain first reference filtered data, wherein parameters of the first filter are indicated by the first filter information, or a type of the first filter and parameters of the first filter are indicated by the first filter information;

[0921] Determine N base layer filter data according to the first reference filter data and the first base layer data, wherein the second base layer data includes the first reference filter data and the N base layer filter data, where N is a positive integer greater than or equal to 1,

[0922] The first reference filter data and the first base layer data satisfy the following relationship: baseAfter0[i]=F1[base[i]];

[0923] Wherein, baseAfter0[i] represents the value of pixel i in the first reference filtered data, base[i] represents the value of pixel i in the first base layer data, and F1[] represents the first filter;

[0924] The nth basic layer filter data in the N basic layer filter data and the first basic layer data satisfy the following relationship: baseAfter n [i]=base[i]-baseAfter n-1 [i],

[0925] where baseAfter n [i] represents the value of pixel i in the nth basic layer filter data, n=1,…,N.

[0926] Embodiment 17. The method according to embodiment 3, wherein transforming the first base layer data according to the first metadata to obtain the second base layer data comprises:

[0927] The first base layer data is transformed according to at least one filter information and at least one mapping relationship carried in the first metadata to obtain the second base layer data.

[0928] Embodiment 18. The method according to embodiment 17, characterized in that the at least one filter information includes second filter information, the at least one mapping relationship includes a fourth mapping relationship and a fifth mapping relationship, and the transforming the first base layer data according to the at least one filter information and the at least one mapping relationship carried in the first metadata to obtain the second base layer data includes:

[0929] filtering the first base layer data using a second filter to obtain second reference filtered data, wherein parameters of the second filter are indicated by the second filter information, or a type of the second filter and parameters of the second filter are indicated by the second filter information;

[0930] The second reference filtered data is transformed according to the fourth mapping relationship and the fifth mapping relationship to obtain the second basic layer data.

[0931] Embodiment 19. The method according to embodiment 18, wherein transforming the second reference filtered data according to the fourth mapping relationship and the fifth mapping relationship to obtain the second base layer data includes:

[0932] The second reference filter and the second base layer data satisfy the following relationship: baseAfter[i]=TMB5(TMB4(baseMid1[i])),

[0933] Among them, baseAfter[i] represents the value of the i-th pixel in the second basic layer data, baseMid1[i] represents the value of pixel i in the second reference filtered data, TMB4() represents the fourth mapping relationship, and TMB5() represents the fifth mapping relationship.

[0934] Embodiment 20. The method according to embodiment 18, wherein transforming the second reference filtered data according to the fourth mapping relationship and the fifth mapping relationship to obtain the second base layer data comprises:

[0935] The second reference filter and the second base layer data satisfy the following relationship: baseAfter[i]=TMB3(R base[i]_7 )×TMB2(baseMid1[i]),

[0936] Wherein, baseAfter[i] represents the value of the i-th pixel in the second basic layer data, baseMid1[i] represents the value of the pixel i in the second reference filtered data, TMB4() represents the fourth mapping relationship, TMB5() represents the fifth mapping relationship, R base[i]_7 Represents the seventh reference value of the i-th pixel determined according to the value of the i-th pixel in the first basic layer data.

[0937] Embodiment 21. The method according to any one of embodiments 1 to 20, wherein determining the second layer data based on the first enhancement layer data comprises:

[0938] Obtaining extended enhancement layer data according to the first enhancement layer data;

[0939] or

[0940] Upsampling the first enhancement layer data to obtain extended enhancement layer data;

[0941] The second enhancement layer data is determined according to the extended enhancement layer data.

[0942] Embodiment 22. The method according to embodiment 21, wherein upsampling the first enhancement layer data to obtain extended enhancement layer data comprises:

[0943] Obtaining interpolation mode indication information obtained from the first metadata or preset information;

[0944] The first enhancement layer data is up-sampled using the interpolation method indicated by the interpolation method indication information to obtain the extended enhancement layer data.

[0945] Embodiment 23. The method according to embodiment 21, wherein upsampling the first enhancement layer data to obtain extended enhancement layer data comprises:

[0946] The first enhancement layer data is up-sampled using a preset interpolation method to obtain the extended enhancement layer data.

[0947] Embodiment 24. The method according to embodiment 21, wherein upsampling the first enhancement layer data to obtain extended enhancement layer data comprises:

[0948] Determine K reference pixels included in the first base layer data, where the K reference pixels are located at K first positions, respectively, and the first enhancement layer data has no pixels at the K first positions, where K is a positive integer greater than or equal to 1;

[0949] Determine K groups of adjacent pixels, wherein the K groups of adjacent pixels correspond one-to-one to the K reference pixels, a kth group of adjacent pixels in the K groups of adjacent pixels includes at least one pixel, the at least one pixel included in the kth group of adjacent pixels is adjacent to the kth reference pixel in the K reference pixels, each pixel in the at least one pixel included in the kth group of adjacent pixels has a corresponding enhanced pixel in the first enhancement layer data, and the coordinates of each pixel are the same as the coordinates of the enhanced pixel of each pixel, and k=1, ..., K;

[0950] The value of a kth reference enhancement pixel among K reference enhancement pixels is determined according to the value of at least one pixel included in the kth group of adjacent pixels, and the K reference enhancement pixels are located at the K first positions of the extended enhancement layer data.

[0951] Embodiment 25. The method according to embodiment 24, wherein determining the value of the kth reference enhanced pixel among the K reference enhanced pixels based on the value of at least one pixel included in the kth group of adjacent pixels comprises:

[0952] Determine a first neighboring pixel, wherein a difference between a value of the kth reference pixel and a value of the first neighboring pixel is smaller than a difference between a value of the kth reference pixel and a value of any neighboring pixel in the kth group of neighboring pixels except the first neighboring pixel;

[0953] The value of the enhanced pixel corresponding to the first adjacent pixel is determined to be the values ​​of the lower k reference enhanced pixels.

[0954] Embodiment 26. The method according to embodiment 24, wherein determining the value of the kth reference enhanced pixel among the K reference enhanced pixels based on the value of at least one pixel included in the kth group of adjacent pixels comprises:

[0955] determining at least two second-neighboring pixels, wherein a difference between a value of the kth reference pixel and a value of any second-neighboring pixel of the at least two second-neighboring pixels is smaller than a difference between a value of the kth reference pixel and a value of any neighboring pixel in the kth group of neighboring pixels other than the at least two second-neighboring pixels;

[0956] Determining at least two enhanced pixels, where the at least two enhanced pixels correspond one-to-one to the at least two second adjacent pixels;

[0957] The value of the kth reference enhanced pixel is determined according to the values ​​of the at least two enhanced pixels.

[0958] Embodiment 27. The method according to any one of Embodiments 21 to 26, wherein determining the second enhancement layer data based on the extended enhancement layer data comprises:

[0959] It is determined that the second enhancement layer data is the same as the extended enhancement layer data.

[0960] Embodiment 28. The method according to any one of Embodiments 21 to 26, wherein determining the second enhancement layer data based on the extended enhancement layer data comprises:

[0961] The extended enhancement layer data is transformed according to the first metadata to obtain the second enhancement layer data.

[0962] Embodiment 29. The method according to embodiment 28, wherein transforming the extended enhancement layer data according to the first metadata to obtain the second enhancement layer data comprises:

[0963] The extended enhancement layer data is transformed according to at least one reference value carried in the first metadata to obtain the second enhancement layer data.

[0964] Embodiment 30. The method according to embodiment 29, wherein the at least one reference value includes a fifth reference value and a sixth reference value, and the transforming the extended enhancement layer data according to the at least one reference value carried in the first metadata to obtain the second enhancement layer data comprises:

[0965] The fifth reference value, the sixth reference value, the extended enhancement layer data, and the second enhancement layer data satisfy the following relationship: enhanceAfter[i]=enhance[i]×REF5+(A-enhance[i])×REF6,

[0966] Wherein, enhanceAfter[i] represents the value of the i-th pixel in the second enhancement layer data, enhance[i] represents the value of the i-th pixel in the extended enhancement layer data, REF5 represents the fifth reference value, REF6 represents the sixth reference value, and A represents the maximum value of multiple pixels stored in the extended enhancement layer data.

[0967] Embodiment 31. The method according to embodiment 29, wherein the at least one reference value includes a seventh reference value, and the transforming the extended enhancement layer data according to the at least one reference value carried in the first metadata to obtain the second enhancement layer data comprises:

[0968] The seventh reference value, the extended enhancement layer data, and the second enhancement layer data satisfy the following relationship: enhanceAfter[i]=enhance[i]+REF7,

[0969] Here, enhanceAfter[i] represents the value of the i-th pixel in the second enhancement layer data, enhance[i] represents the value of the i-th pixel in the extended enhancement layer data, and REF7 represents the seventh reference value.

[0970] Embodiment 32. The method according to embodiment 29, wherein the at least one reference value includes an eighth reference value, and the transforming the extended enhancement layer data according to the at least one reference value carried in the first metadata to obtain the second enhancement layer data includes:

[0971] The eighth reference value, the extended enhancement layer data, and the second enhancement layer data satisfy the following relationship: enhanceAfter[i]=REF8+A-enhance[i],

[0972] Wherein, enhanceAfter[i] represents the value of the i-th pixel in the second enhancement layer data, enhance[i] represents the value of the i-th pixel in the extended enhancement layer data, REF8 represents the eighth reference value, and A represents the maximum value of multiple pixels stored in the extended enhancement layer data.

[0973] Embodiment 33. The method according to embodiment 28, wherein transforming the extended enhancement layer data according to the first metadata to obtain the second enhancement layer data comprises:

[0974] The extended enhancement layer data is transformed according to at least one mapping relationship carried in the first metadata to obtain the second enhancement layer data.

[0975] Embodiment 34. The method according to embodiment 33, wherein the at least one mapping relationship includes a sixth mapping relationship, and the transforming the extended enhancement layer data according to the at least one mapping relationship carried in the first metadata to obtain the second enhancement layer data includes:

[0976] The extended enhancement layer data and the second enhancement layer data satisfy the following relationship: enhanceAfter[i]=TMB6(R enhance[i]_1 ),

[0977] Wherein, enhanceAfter[i] represents the value of the i-th pixel in the second enhancement layer data, R enhance[i]_1 represents the first reference value of the i-th pixel determined according to the value of the i-th pixel in the extended enhancement layer data, and TMB6() represents the sixth mapping relationship.

[0978] Embodiment 35. The method according to embodiment 33, wherein the at least one mapping relationship includes a sixth mapping relationship, and the transforming the extended enhancement layer data according to the at least one mapping relationship carried in the first metadata to obtain the second enhancement layer data includes:

[0979] The extended enhancement layer data and the second enhancement layer data satisfy the following relationship: enhanceAfter[i]=TMB6(R enhance[i]_2 )×R enhance[i]_3 ,

[0980] Wherein, enhanceAfter[i] represents the value of the i-th pixel in the second enhancement layer data, TMB6() represents the sixth mapping relationship, R enhance[i]_2 is the second reference value of the i-th pixel determined according to the value of the i-th pixel in the extended enhancement layer data, R enhance[i]_3 is a third reference value of the i-th pixel determined according to the value of the i-th pixel in the extended enhancement layer data.

[0981] Embodiment 36. The method according to embodiment 33, characterized in that the at least one mapping relationship includes a seventh mapping relationship and an eighth mapping relationship, and the transforming the extended enhancement layer data according to the at least one mapping relationship carried in the first metadata to obtain the second enhancement layer data includes:

[0982] The extended enhancement layer data and the second enhancement layer data satisfy the following relationship: enhanceAfter[i]=TMB7(TMB8(R enhance[i]_4 )),

[0983] Wherein, enhanceAfter[i] represents the value of the i-th pixel in the second enhancement layer data, R enhance[i]_4 represents the fourth reference value of the i-th pixel determined according to the value of the i-th pixel in the extended enhancement layer data, TMB7() represents the seventh mapping relationship, and TMB8() represents the eighth mapping relationship.

[0984] Embodiment 37. The method according to embodiment 33, characterized in that the at least one mapping relationship includes a seventh mapping relationship and an eighth mapping relationship, and the transforming the extended enhancement layer data according to the at least one mapping relationship carried in the first metadata to obtain the second enhancement layer data includes:

[0985] The extended enhancement layer data and the second enhancement layer data satisfy the following relationship:

[0986] The extended enhancement layer data and the second enhancement layer data satisfy the following relationship: enhanceAfter[i]=TMB7(R enhance[i]_6 )×TMB8(R enhance[i]_5 ),

[0987] Wherein, enhanceAfter[i] represents the value of the i-th pixel in the second enhancement layer data, R enhance[i]_5 represents the fifth reference value of the i-th pixel determined according to the value of the i-th pixel in the extended enhancement layer data, R enhance[i]_6 represents the sixth reference value of the i-th pixel determined according to the value of the i-th pixel in the extended enhancement layer data, TMB7() represents the seventh mapping relationship, and TMB8() represents the eighth mapping relationship.

[0988] Embodiment 38. The method according to embodiment 36 or 37 is characterized in that the seventh mapping relationship is a global tone mapping function, or the seventh mapping relationship is a local tone mapping function.

[0989] Example 39. The method according to any one of Examples 33 to 38 is characterized in that the at least one mapping relationship includes at least one functional relationship.

[0990] Embodiment 40. The method according to embodiment 28, wherein transforming the extended enhancement layer data according to the first metadata to obtain the second enhancement layer data comprises:

[0991] The extended enhancement layer data is transformed according to at least one filter information carried in the first metadata to obtain the second enhancement layer data.

[0992] Embodiment 41. The method according to embodiment 40, characterized in that the at least one filter information includes third filter info...

Claims

1. A signal processing method, characterized in that: include: Acquire a first base image, a first gain map, and metadata, wherein the first base image corresponds to a first dynamic range; Processing the first base image according to the metadata to obtain a second base image; A target image is obtained according to the second base image and the first gain map, wherein the target image corresponds to a second dynamic range, and the second dynamic range is different from the first dynamic range.

2. The method according to claim 1, characterized in that The step of obtaining a target image according to the second base image and the first gain map comprises: Processing the first gain map according to the metadata to obtain a second gain map; The target image is obtained according to the second base image and the second gain map.

3. The method according to claim 2, characterized in that The step of processing the first gain map according to the metadata comprises: The first gain map is upsampled or downsampled according to the metadata to obtain the second gain map.

4. The method according to claim 3, characterized in that The resolution of the second gain map is the same as the resolution of the first basic image.

5. The method according to claim 3, characterized in that: The second gain map resolution is a second preset resolution.

6. The method according to any one of claims 3 to 5, characterized in that: The metadata includes sampling phase information, and upsampling or downsampling the first gain map according to the metadata includes: upsampling or downsampling the first gain map according to the sampling phase information.

7. The method according to claim 6, characterized in that The sampling phase information is used to indicate the relative position relationship between pixels of the first gain map and the second gain map.

8. The method according to claim 6 or 7, characterized in that The sampling phase information includes pixel coordinates of a first preset position of the first gain map and pixel coordinates of a second preset position of the second gain map.

9. The method according to claim 6 or 7, characterized in that The sampling phase information includes an offset between pixel coordinates of the second gain map and pixel coordinates of the first gain map.

10. The method according to any one of claims 6 to 9, characterized in that: The sampling phase information includes brightness sampling phase information, and the upsampling or downsampling of the first gain map according to the sampling phase information includes: upsampling or downsampling of the brightness component of the first gain map according to the brightness sampling phase information; or The sampling phase information includes chroma sampling phase information, and upsampling or downsampling the first gain map according to the sampling phase information includes: upsampling or downsampling the chroma component of the first gain map according to the chroma sampling phase information; or The sampling phase information includes luminance sampling phase information and chrominance sampling phase information. Upsampling or downsampling the first gain map according to the sampling phase information includes: upsampling or downsampling the luminance component of the first gain map according to the luminance sampling phase information, and upsampling or downsampling the chrominance component of the first gain map according to the chrominance sampling phase information.

11. The method according to any one of claims 3 to 10, characterized in that: The metadata includes indication information of a target sampling algorithm, and upsampling or downsampling the first gain map according to the metadata includes: upsampling or downsampling the first gain map using the target sampling algorithm according to the indication information of the target sampling algorithm.

12. The method according to claim 11, characterized in that The indication information of the target sampling algorithm includes an index value used to indicate that the target sampling algorithm is determined from a plurality of preset sampling algorithms.

13. The method according to claim 12, characterized in that The indication information of the target sampling algorithm includes the name or other specific identifier of the target sampling algorithm.

14. The method according to any one of claims 11 to 13, characterized in that When the metadata includes sampling phase information and indication information of the target sampling algorithm, upsampling or downsampling the first gain map according to the metadata includes: upsampling or downsampling the first gain map using the target sampling algorithm according to the sampling phase information and the indication information of the target sampling algorithm.

15. A coding method, characterized in that: include: According to a first image, a first base image, a first gain map, and metadata are acquired, wherein the first base image corresponds to a first dynamic range, the first image corresponds to a second dynamic range, and the second dynamic range is different from the first dynamic range; encoding the first base image; encoding the first gain map; The metadata is encoded.

16. The method according to claim 15, characterized in that The metadata is used to upsample or downsample the first gain map to obtain the second gain map.

17. The method according to claim 16, characterized in that The metadata includes sampling phase information, where the sampling phase information is used to indicate a relative position relationship between pixels of the first gain map and the second gain map.

18. The method according to claim 17, characterized in that The sampling phase information includes pixel coordinates of a first preset position of the first gain map and pixel coordinates of a second preset position of the second gain map.

19. The method according to claim 17, characterized in that The sampling phase information includes an offset between pixel coordinates of the second gain map and pixel coordinates of the first gain map.

20. The method according to any one of claims 17 to 19, characterized in that The sampling phase information includes luminance sampling phase information and / or chrominance sampling phase information.

21. The method according to any one of claims 17 to 19, characterized in that: The sampling phase information includes one or more of red R channel sampling phase information, green G channel sampling phase information, and blue B channel sampling phase information.

22. The method according to any one of claims 15 to 21, characterized in that The metadata includes information indicating a target sampling algorithm.

23. The method according to claim 22, characterized in that The indication information of the target sampling algorithm includes an index value used to indicate the determination of the two target sampling algorithms from a plurality of preset sampling algorithms.

24. The method according to claim 22, characterized in that The indication information of the target sampling algorithm includes the name or other specific identifier of the target sampling algorithm.

25. An electronic device, characterized in that: include: A decoding module, configured to obtain a first base image, a first gain map, and metadata, wherein the first base image corresponds to a first dynamic range; a processing module, configured to process the first basic image according to the metadata to obtain a second basic image; The processing module is further used to obtain a target image according to the second basic image and the first gain map, wherein the target image corresponds to a second dynamic range, and the second dynamic range is different from the first dynamic range.

26. The electronic device according to claim 24, characterized in that: The processing module is also used to implement the method as described in any one of claims 2-14.

27. An electronic device, characterized in that: include: A processing module, configured to obtain a first base image, a first gain map, and metadata according to a first image, wherein the first base image corresponds to a first dynamic range, the first image corresponds to a second dynamic range, and the second dynamic range is different from the first dynamic range; A coding module for the first basic image; The encoding module is further used to encode the first gain map; The encoding module is also used to encode the metadata.

28. The electronic device according to claim 27, characterized in that: The processing module is also used to implement the method as described in any one of claims 15-25.

29. An electronic device, characterized in that: include: A memory and a processor coupled to each other, wherein the processor calls a program code stored in the memory to execute the method according to any one of claims 1 to 14 or claims 15 to 25.

30. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program runs on a computer or a processor, the method of any one of claims 1 to 14 or claims 15 to 25 is implemented.

31. A computer program product, characterized in that The method comprises a program or an instruction, which, when executed on a computer or a processor, causes the computer or the processor to execute the method according to any one of claims 1 to 14 or claims 15 to 25.

32. A code stream, characterized in that The code stream includes a first basic image, a first gain map and metadata, and the metadata includes information required for processing the first basic image and the first gain map to obtain a target image.