Image decoding method, image encoding method, encoder, decoder, and medium
By introducing a quality enhancement network into video coding and decoding technology, the quality of reconstructed blocks is enhanced, which solves the problem of limited quality improvement of reconstructed blocks, improves the intra-frame prediction accuracy of coded blocks, and enhances coding and decoding performance.
Patent Information
- Application Number
- PCT/CN2025/094028
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2025-05-09
- Publication Date
- 2026-11-12
AI Technical Summary
In existing video coding and decoding technologies, the quality improvement of reconstructed blocks is limited, which affects the accuracy of intra-frame prediction references for subsequent coding blocks.
A quality enhancement network is introduced into the hybrid coding framework to perform quality enhancement processing through at least one of the predicted block and residual block of the current block, as well as the reconstructed block of the current block, independently of the loop filtering process, thereby improving the quality of the reconstructed block.
This improves the quality of the reconstructed blocks, provides a more accurate reference for intra-frame prediction of subsequent coded blocks, and enhances encoding and decoding performance.
Smart Images

Figure CN2025094028_12112026_PF_FP_ABST
Abstract
Description
Image decoding, encoding methods, codecs and media Technical Field
[0001] This application relates to the field of video encoding and decoding technology, and in particular to an image decoding and encoding method, codec, and medium. Background Technology
[0002] During the image encoding and decoding process of video, after the reconstruction of the current block in the image is completed, the reconstruction result needs to be further filtered to improve the quality of the reconstructed image, thereby providing a better reference for subsequent encoding and decoding.
[0003] In related technologies, the in-loop filter process includes processing steps such as Luma Mapping with Chroma Scaling (LMCS), DeBlocking Filter (DBF), Sample Adaptive Offset (SAO), Adaptive Loop Filter (ALF), and Cross Component Adaptive Loop Filter (CCALF). Summary of the Invention
[0004] This application provides an image decoding and encoding method, a codec, and a medium. The technical solution is as follows:
[0005] On one hand, embodiments of this application provide an image decoding method, the method comprising:
[0006] Based on at least one of the predicted block and residual block of the current block, and the reconstructed block of the current block, the reconstructed block is quality-enhanced by a quality enhancement network.
[0007] On the other hand, embodiments of this application provide an image encoding method, the method comprising:
[0008] Based on at least one of the predicted block and residual block of the current block, and the reconstructed block of the current block, the reconstructed block is quality-enhanced by a quality enhancement network.
[0009] On the other hand, embodiments of this application provide a decoding device, the device comprising:
[0010] A decoding unit is configured to enhance the quality of the reconstructed block using a quality enhancement network, based on at least one of the predicted block and residual block of the current block, and the reconstructed block of the current block.
[0011] On the other hand, embodiments of this application provide an encoding device, the device comprising:
[0012] An encoding unit is configured to enhance the quality of a reconstructed block using a quality enhancement network, based on at least one of a predicted block and a residual block of the current block, and a reconstructed block of the current block.
[0013] On the other hand, embodiments of this application provide a decoder, the decoder including a memory and a processor, the memory being used to store a computer program running on the processor; the processor being used to execute the image decoding method as described above when running the computer program.
[0014] On the other hand, embodiments of this application provide an encoder, the encoder including a memory and a processor, the memory for storing a computer program running on the processor; the processor for executing the image encoding method as described above when running the computer program.
[0015] On the other hand, embodiments of this application provide a non-volatile computer-readable storage medium for storing a bitstream, the bitstream being generated by an image encoding method using an encoder, or the bitstream being decoded by an image decoding method using a decoder, wherein the image decoding method includes the image decoding method described above, or the image encoding method includes the image encoding method described above.
[0016] On the other hand, embodiments of this application provide a computer program product, the computer program product including computer instructions stored in a computer-readable storage medium, a processor obtaining the computer instructions from the computer-readable storage medium, and the processor executing the computer instructions to implement the image decoding method or the image encoding method as described above.
[0017] In this embodiment, quantization distortion during the encoding and decoding process is fully considered. Quality enhancement is added to the hybrid coding framework. Based on at least one of the prediction block and residual block of the current block, as well as the reconstruction block of the current block, the reconstruction block is enhanced by the quality enhancement network. This can improve the reconstruction quality of the current block and provide a more accurate reference for intra-frame prediction of subsequent coding blocks. Attached Figure Description
[0018] Figure 1 is a schematic diagram illustrating the encoding process at the encoding end in an exemplary embodiment of this application;
[0019] Figure 2 is a schematic diagram illustrating the decoding process at the decoding end in an exemplary embodiment of this application;
[0020] Figure 3 is a schematic diagram illustrating the ALF and CCALF processes in an exemplary embodiment of this application;
[0021] Figure 4 is a flowchart illustrating an exemplary embodiment of the image decoding method of this application;
[0022] Figure 5 is a schematic diagram illustrating the decoding process at the decoding end in another exemplary embodiment of this application;
[0023] Figure 6 is a schematic diagram of the structure of a quality enhancement network shown in an exemplary embodiment of this application;
[0024] Figure 7 is a flowchart illustrating an image decoding method in another exemplary embodiment of this application;
[0025] Figure 8 is a schematic diagram of the structure of a first self-attention subnetwork shown in an exemplary embodiment of this application;
[0026] Figure 9 is a schematic diagram of the structure of the first feedforward subnetwork shown in an exemplary embodiment of this application;
[0027] Figure 10 is a schematic diagram of the structure of the first feature extraction layer shown in an exemplary embodiment of this application;
[0028] Figure 11 is a schematic diagram of the structure of the first cross-attention subnetwork shown in an exemplary embodiment of this application;
[0029] Figure 12 is a schematic diagram of the structure of the second feature extraction layer shown in an exemplary embodiment of this application;
[0030] Figure 13 is a schematic diagram illustrating lightweight processing of the teacher network in an exemplary embodiment of this application;
[0031] Figure 14 is a schematic diagram of the structure of a teacher network shown in an exemplary embodiment of this application;
[0032] Figure 15 is a flowchart illustrating an exemplary embodiment of the image encoding method of this application;
[0033] Figure 16 is a flowchart illustrating the setting of the enable flag bit in an exemplary embodiment of this application;
[0034] Figure 17 is a flowchart illustrating the quality enhancement of the luminance reconstruction block during the encoding and decoding process, as shown in an exemplary embodiment of this application;
[0035] Figure 18 shows a structural block diagram of a decoding apparatus provided in an exemplary embodiment of this application;
[0036] Figure 19 shows a structural block diagram of an encoding device provided in an exemplary embodiment of this application;
[0037] Figure 20 shows a structural block diagram of a decoder provided in an exemplary embodiment of this application;
[0038] Figure 21 shows a structural block diagram of an encoder provided in an exemplary embodiment of this application. Detailed Implementation
[0039] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.
[0040] In this article, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.
[0041] Currently, most common video codec standards employ a block-based hybrid coding framework. Images in a video are divided into squares of equal size (e.g., 256×256, 128×128, 64×64) called Largest Coding Units (LCUs) or Coding Tree Units (CTUs). These LCUs or CTUs can be further subdivided into rectangular Coding Units (CUs) according to rules. Coding units may also be further divided into Prediction Units (PUs), Transform Units (TUs), etc.
[0042] The hybrid coding framework includes modules such as prediction, transform, quantization, entropy coding, and in-loop filtering. The prediction module includes intra-prediction and inter-prediction. Inter-prediction includes motion estimation and motion compensation. Because there is a strong correlation between adjacent samples within the same frame, intra-prediction can be used in video coding and decoding techniques to eliminate spatial redundancy between adjacent samples. Because there is a strong similarity between adjacent frames in a video, inter-prediction can be used in video coding and decoding techniques to eliminate temporal redundancy between adjacent frames, thereby improving coding efficiency.
[0043] At the encoding end, as shown in Figure 1, the encoder first divides the image into multiple coding blocks. Then, it uses intra-frame or inter-frame prediction (including motion estimation and motion compensation) algorithms to generate a prediction block for the current block. The encoder then subtracts the prediction block from the original block to obtain a residual block. This residual block is then transformed and quantized to obtain quantization coefficients. Finally, entropy coding is used to encode these quantization coefficients into the bitstream. Furthermore, the encoder reconstructs the image based on the inverse quantization / inverse transform results and the prediction block to obtain a reconstructed block. Loop filtering is then applied to the reconstructed block to compensate for distortion information and provide a better reference for subsequent encoding.
[0044] At the decoding end, as shown in Figure 2, the decoder uses intra-frame or inter-frame prediction algorithms to predict the prediction block of the current block. Simultaneously, it performs entropy decoding on the bitstream to obtain quantization coefficients, and then performs inverse quantization and inverse transform on these coefficients to obtain residual blocks. The prediction block and residual blocks are then added together to obtain the reconstructed block. For the reconstructed block, the decoder further performs loop filtering, and finally generates the decoded image based on the loop-filtered reconstructed block.
[0045] In related technologies, the reconstructed block is directly subjected to loop filtering. As shown in Figure 1, the loop filtering process includes LMCS, DBF, SAO, ALF, and CCALF.
[0046] Among them, LMCS is used for luminance mapping based on an adaptive piecewise linear model (applied at the sample level) to improve video coding efficiency, and luminance-based chrominance residual scaling (applied at the chrominance block level) to compensate for the impact of luminance signal mapping on chrominance signals.
[0047] DBF is used to reduce block artifacts in video coding. It works by applying it to the boundaries of coding blocks and reducing discontinuities between blocks through smoothing.
[0048] SAO is used to apply adaptive offsets to reduce coding artifacts (especially for high-frequency details).
[0049] ALF and CCALF are based on the Wiener filtering principle. They establish the Wiener-Hough equation based on the original image and the reconstructed image, and solve for the filter coefficients with the minimum mean square error, so as to reduce decoding error and improve coding performance.
[0050] The positions of ALF and CCALF in the loop filtering are shown in Figure 3. ALF comprises Luminance ALF (ALF Luma) and Chroma ALF (ALF Chroma). Luminance ALF is designed to minimize the mean square error between the reconstructed luminance image and the original image using luminance information; its input is the output of SAO Luma. Chroma ALF is designed to minimize the mean square error between the reconstructed chrominance image and the original image using chrominance information; its input is the outputs of SAO Cb and SAO Cr.
[0051] CCALF is designed to minimize the mean square error between the chroma reconstructed image and the original image by utilizing luminance information, and its output is the same as that of SAO Luma. It's important to note that CCALF filters the chroma reconstructed information output by SAO; therefore, the filtered result of CCALF (…) or It needs to be superimposed with the chromaticity ALF filtering result to obtain the final chromaticity filtering result (including the two chromaticity components Cb and Cr).
[0052] In this embodiment of the application, in order to further improve the encoding and decoding performance, a block-based quality enhancement is introduced. The quality enhancement network is used to perform quality enhancement processing on the reconstructed blocks, which not only helps to improve the quality of the reconstructed blocks, but also provides a more accurate reference for intra-frame prediction of subsequent coded blocks.
[0053] Please refer to Figure 4, which illustrates a flowchart of an image decoding method provided in an exemplary embodiment of this application. The method is applied to a decoder and may include the following steps:
[0054] Step 401: Based on at least one of the predicted block and residual block of the current block, and the reconstructed block of the current block, the quality of the reconstructed block is enhanced by a quality enhancement network.
[0055] In a block-based hybrid coding framework, a frame of video is divided into square CTU blocks of equal size. The current block refers to the coding block currently being processed by the decoder, i.e., the CTU block.
[0056] During the decoding process, the decoder uses intra-frame or inter-frame prediction algorithms to predict the prediction block of the current block. On the other hand, it performs entropy decoding on the bitstream to obtain quantization coefficients, and performs inverse quantization and inverse transformation on the quantization coefficients to obtain residual blocks. The prediction block and residual blocks are then added together to obtain the reconstructed block.
[0057] In this embodiment, to improve the quality of the reconstructed blocks, a quality enhancement network is used to enhance the quality of the reconstructed blocks. Optionally, the input to the quality enhancement network includes at least the reconstructed block of the current block, and at least one of the predicted block and the residual block can be used as auxiliary input information to help the quality enhancement network fully learn the image features of the current block. That is, the input to the quality enhancement network can be the reconstructed block and the predicted block, the reconstructed block and the residual block, or the reconstructed block, the predicted block, and the residual block. The output of the quality enhancement network is the reconstructed block after quality enhancement.
[0058] In some embodiments, the process of enhancing the quality of a reconstructed block using a quality enhancement network, based on at least one of the predicted block and residual block of the current block, and the reconstructed block of the current block, is independent of the loop filtering process and can be performed before loop filtering. That is, the decoder first enhances the quality of the reconstructed block, and then performs loop filtering on the enhanced reconstructed block.
[0059] As illustrated in Figure 5, the quality enhancement proposed in this application is performed before in-loop filtering. During decoding, the decoder uses intra-frame or inter-frame prediction algorithms to predict the prediction block of the current block. Simultaneously, it performs entropy decoding on the bitstream to obtain quantization coefficients, and then performs inverse quantization and inverse transform on the quantization coefficients to obtain residual blocks. The prediction block and residual blocks are then added together to obtain the reconstructed block. Before performing in-loop filtering on the reconstructed block, the quality enhancement network first enhances the quality of the reconstructed block, and then in-loop filtering is performed on the quality-enhanced reconstructed block.
[0060] In summary, the embodiments of this application fully consider the quantization distortion in the encoding and decoding process, add quality enhancement to the hybrid coding framework, and enhance the quality of the reconstructed block through the quality enhancement network based on at least one of the prediction block and residual block of the current block, as well as the reconstructed block of the current block. This can improve the reconstruction quality of the current block and provide a more accurate reference for intra-frame prediction of subsequent coding blocks.
[0061] In some embodiments, in order to improve the feature learning effect of the quality enhancement network on the reconstructed information, in addition to the predicted block and residual block of the current block, the neighboring reconstructed blocks of the reconstructed block can also be used as network inputs, so that the quality enhancement network can learn the correlation features between the reconstructed block and the neighboring reconstructed blocks, thereby improving the quality enhancement quality of the reconstructed block.
[0062] The decoder enhances the quality of the reconstructed block using a quality enhancement network, based on at least one of the predicted block, residual block, and neighboring reconstructed blocks of the current block, as well as the reconstructed block of the current block. The input to the quality enhancement network can be the reconstructed block and the predicted block, the reconstructed block and the residual block, the reconstructed block and neighboring reconstructed blocks, the reconstructed block, the predicted block and the residual block, the reconstructed block, the residual block and neighboring reconstructed blocks, or the reconstructed block, the predicted block, the residual block and neighboring reconstructed blocks.
[0063] Optionally, the adjacent reconstructed blocks of the reconstructed block can be blocks located within the γ-shaped region of the reconstructed block. The γ-shaped region refers to the left and upper regions of the reconstructed block in the current image.
[0064] During the decoding process, determine whether to perform quality enhancement on the reconstructed blocks.
[0065] Due to considerations of encoding / decoding costs and quality, quality enhancement of reconstructed blocks is not necessary in all scenarios. Therefore, before the decoder enhances the quality of reconstructed blocks through a quality enhancement network, it first needs to determine whether the reconstructed blocks of the current block require quality enhancement.
[0066] In one possible implementation, the decoder parses the bitstream, determines the enable flag, and, if the value of the enable flag indicates that the reconstructed block needs quality enhancement, performs quality enhancement on the reconstructed block based on at least one of the predicted block and residual block of the current block, and the reconstructed block of the current block, through a quality enhancement network.
[0067] Optionally, the value of the enable flag is used to indicate whether the reconstructed block of the current block needs to be enhanced in quality, and the enable flag is encoded into the bitstream by the encoder.
[0068] Optionally, when the enable flag is true (or 1), the decoder determines to perform quality enhancement on the reconstructed block through the quality enhancement network; when the enable flag is false (or 0), the decoder determines not to perform quality enhancement on the reconstructed block through the quality enhancement network.
[0069] Optionally, the value of the enable flag indicates whether the luminance reconstruction block of the current block needs quality enhancement. Here, the luminance reconstruction block refers to the reconstruction block composed of the luminance component Y of the reconstruction block. Optionally, when the enable flag is true (or 1), the decoder performs quality enhancement on the luminance reconstruction block; when the enable flag is false (or 0), the decoder does not perform quality enhancement on the luminance reconstruction block.
[0070] Regarding the quality enhancement process of the luminance reconstruction block, in one possible implementation, when the value of the enable flag indicates that the luminance reconstruction block needs quality enhancement, the decoder can enhance the luminance reconstruction block through a luminance quality enhancement network based on at least one of the luminance prediction block and luminance residual block of the current block, as well as the luminance reconstruction block of the current block.
[0071] Optionally, the value of the enable flag indicates whether the chroma reconstruction block of the current block needs quality enhancement. Here, the chroma reconstruction block refers to the reconstruction block composed of the chroma components Cb and Cr of the reconstruction block. Optionally, when the enable flag is true (or 1), the decoder performs quality enhancement on the chroma reconstruction block; when the enable flag is false (or 0), the decoder does not perform quality enhancement on the chroma reconstruction block.
[0072] Regarding the quality enhancement process of the chroma reconstruction block, in one possible implementation, when the value of the enable flag indicates that the chroma reconstruction block needs quality enhancement, the decoder can enhance the chroma reconstruction block through a chroma quality enhancement network based on at least one of the chroma prediction block and chroma residual block of the current block, as well as the chroma reconstruction block of the current block.
[0073] In another possible implementation, the luminance reconstruction block can be used as input to the chroma quality enhancement network to improve the network's learning of the reconstructed information. That is, when the value of the enable flag indicates that the chroma reconstruction block needs quality enhancement, the decoder enhances the chroma reconstruction block through the first chroma quality enhancement network based on at least one of the chroma prediction block, chroma residual block, and luminance reconstruction block of the current block, as well as the chroma reconstruction block of the current block.
[0074] In another possible implementation, the enhanced luminance reconstruction block can be used as input to the chroma quality enhancement network to further improve the network's learning of the reconstructed information. That is, when the value of the enable flag indicates that the chroma reconstruction block needs quality enhancement, the decoder enhances the chroma reconstruction block through a second chroma quality enhancement network based on at least one of the current block's chroma prediction block, chroma residual block, and enhanced luminance reconstruction block, as well as the current block's chroma reconstruction block.
[0075] Optionally, the enable flag can be used to characterize whether quality enhancement is performed on the reconstructed block in two dimensions, that is, whether the value of the enable flag indicates whether the luma and chroma reconstructed blocks of the current block each need quality enhancement. Optionally, the value of the enable flag can be a two-bit binary number. When the enable flag value is 11, it indicates that both the luma and chroma reconstructed blocks of the current block need quality enhancement, so the decoder enhances the luma reconstructed block through the luma quality enhancement network and enhances the chroma reconstructed block through the chroma quality enhancement network. When the enable flag value is 10, it indicates that the luma reconstructed block of the current block needs quality enhancement, but the chroma reconstructed block does not need quality enhancement, so the decoder enhances the luma reconstructed block through the luma quality enhancement network. When the enable flag value is 01, it indicates that the luma reconstructed block of the current block does not need quality enhancement, but the chroma reconstructed block needs quality enhancement, so the decoder enhances the chroma reconstructed block through the chroma quality enhancement network. When the enable flag is 00, the decoder does not perform quality enhancement on either the luminance reconstruction block or the chrominance reconstruction block.
[0076] Optionally, when the enable flag value is used to indicate whether the reconstructed block of the current block needs quality enhancement, or whether the luminance reconstructed block of the current block needs quality enhancement, or whether the chroma reconstructed block of the current block needs quality enhancement, the data unit of the enable flag value is 1 bit. When the enable flag value is used to indicate whether both the luminance and chroma reconstructed blocks of the current block need quality enhancement, the data unit of the enable flag value is 2 bits.
[0077] Optionally, a luminance enable flag can be set for the luminance reconstruction block, and a chroma enable flag can be set for the chroma reconstruction block. The luminance enable flag is `luma_enhanceformer_enable_flag`, and its value indicates whether the luminance reconstruction block of the current block needs quality enhancement. The chroma enable flag is `chroma_enhanceformer_enable_flag`, and its value indicates whether the chroma reconstruction block of the current block needs quality enhancement.
[0078] In some embodiments, the decoder parses the bitstream to obtain the respective enable flags for different blocks in the same image. That is, different blocks in the same image correspond to their respective enable flags, and the values of the enable flags are used to indicate whether the reconstructed block of the current block needs quality enhancement. Optionally, different blocks in the same image may correspond to their respective luminance enable flags or their respective chrominance enable flags.
[0079] In other embodiments, the decoder parses the bitstream to obtain the enable flags for different slices within the same image; that is, different blocks within the same slice in the same image correspond to a unified enable flag. Optionally, different blocks within the same slice in the same image can correspond to a unified luminance enable flag or a unified chrominance enable flag.
[0080] In the case where different blocks within the same slice of the same image correspond to the same enable flag, the value of the enable flag is determined based on the encoding / decoding cost and encoding / decoding quality of each block in the slice.
[0081] In other embodiments, the decoder parses the bitstream to obtain the enable flag corresponding to the current image; that is, different blocks in the same image correspond to the same enable flag. Optionally, different blocks in the same image can correspond to a unified luminance enable flag or a unified chrominance enable flag.
[0082] In the case where different blocks in the same image correspond to the same enable flag, the value of the enable flag is determined based on the encoding and decoding cost and encoding and decoding quality of each block in the image.
[0083] In other embodiments, the decoder parses the bitstream to obtain the enable flag corresponding to the current Group of Pictures (GOP), meaning that different blocks of different images within the same GOP correspond to a unified enable flag. Optionally, different blocks of different images within the same GOP can correspond to a unified luminance enable flag or a unified chrominance enable flag.
[0084] In the case where different blocks of different images in the same image group correspond to the same enable flag, the value of the enable flag is determined based on the encoding and decoding cost and encoding and decoding quality of each block in the image group.
[0085] Optionally, the correspondence between different blocks and the luminance enable flag, and the correspondence between different blocks and the chroma enable flag, can be the same or different. For example, different blocks in the same image may each have their own luminance enable flag, and different blocks in the same image may each have their own chroma enable flag. Alternatively, different blocks in the same image may each have their own luminance enable flag, and different blocks in the same image may each have a unified chroma enable flag. Another example is that different blocks within the same slice of the same image may each have a unified luminance enable flag, and different blocks in the same image may each have a unified chroma enable flag.
[0086] The process of enhancing the quality of reconstructed blocks using a quality enhancement network.
[0087] In some embodiments, to achieve both quality enhancement of the reconstructed block based on its own image features and quality enhancement based on the correlation features between the reconstructed block, the predicted block, and / or the residual block, the quality enhancement network can employ a two-branch network architecture combining self-attention and cross-attention mechanisms. That is, the quality enhancement network includes a first self-attention branch network and a first cross-attention branch network.
[0088] As illustrated in Figure 6, the quality enhancement network uses residual connections to directly pass the reconstructed block 601 to the output of the branch network. After the enhancement value is output through the first self-attention branch network 611 and the first cross-attention branch network 612, the enhancement value is added to the reconstructed block 601 to obtain the quality-enhanced reconstructed block 604.
[0089] The process of enhancing the quality of reconstructed blocks using a quality enhancement network will be described below. This process, as shown in Figure 7, may include the following steps:
[0090] Step 701: Based on the reconstruction block, determine the first enhancement value of the reconstruction block through the first self-attention branch network.
[0091] Optionally, the first self-attention network determines the first enhancement value of the reconstructed block by learning the image features of the reconstructed block itself. That is, the input of the first self-attention branch network is the reconstructed block, and the output is the first enhancement value of the reconstructed block.
[0092] In some embodiments, the first self-attention branch network includes a first encoding module and a first decoding module. The decoder can first extract features from the reconstructed block through the first encoding module to obtain the first deep features of the reconstructed block, and then perform feature fusion on the first deep features through the first decoding module to obtain the first enhancement value of the reconstructed block. By extracting features first and then fusing features, it can be ensured that the first self-attention branch network can fully learn the image features of the reconstructed block, thereby improving the accuracy of the first enhancement value.
[0093] The first encoding module may include a first convolutional layer and a first feature extraction layer. After the reconstructed block is input into the first self-attention branch network, the decoder first performs convolution processing on the reconstructed block through the first convolutional layer to obtain the first shallow features of the reconstructed block. Then, based on the first shallow features, the first feature extraction layer performs feature extraction to obtain the first deep features.
[0094] Optionally, the first convolutional layer is a 3×3 convolutional layer, used to perform 3×3 convolution and non-linear mapping operations on the input reconstructed block. In one possible implementation, taking the input of the first self-attention branch network as a normalized brightness reconstructed block I1 as an example, the feature dimension of I1 is 1×128×128, i.e. The process of convolving I1 through the first convolutional layer can be represented as M0(I1)=σ(W1(I1)). Here, σ is the ReLU activation function, W1 is a 3×3 convolutional kernel, and M0 is the first shallow feature, with a feature dimension of 32×128×128.
[0095] Optionally, the first feature extraction layer may adopt a Bias Block module structure, and the first feature extraction layer may include a first self-attention subnetwork and a first feedforward subnetwork. For the first shallow features output by the first convolutional layer, in the first feature extraction layer, the decoder first extracts and fuses features based on the first shallow features through the first self-attention subnetwork to obtain the first weighted features, and then performs feature fusion and nonlinear mapping based on the first weighted features through the first feedforward subnetwork to obtain the first deep features.
[0096] Optionally, the network structure of the first self-attention sub-network can be based on a nonlinear enhancement attention module (NEA) network structure. Thus, the first self-attention sub-network can dynamically enhance certain attention scores or reduce other irrelevant attention scores by adjusting learnable weights and biases based on the nonlinear enhancement mechanism, so that the network can optimize a wider range of attention distributions and better adapt to complex data distributions and feature relationships.
[0097] Optionally, the first self-attention subnetwork can adopt a dual-head NEA-based network structure. This first self-attention subnetwork is used to achieve feature extraction and weighted fusion. Schematic, as shown in Figure 8, illustrates the structure of the first self-attention subnetwork provided in an exemplary embodiment of this application. The first self-attention subnetwork consists of three parts: a dual-head attention mechanism and a single 1×1 convolution operation. The dual-head attention mechanism splits the input features into two parts on each channel, calculates the weighted features of the two parts separately in the upper and lower branches, and then concatenates the two weighted features on each channel to obtain the summarized weighted features. Furthermore, the dual-head attention mechanism introduces an attention matrix transpose calculation operation, which can effectively reduce the number of parameters.
[0098] Using the input features of the first self-attention subnetwork as For example, the input features are divided into two equal paths along the channel dimension to obtain... right and The upper and lower branches are respectively subjected to three 1×1 and 3×3 convolution operations to obtain... and Then, the features after convolution are subjected to dimension merging and transpose operations to change the dimensions, resulting in... for After matrix multiplication and normalization using the Softmax function, the attention matrix can be obtained. Similarly, the attention matrix can be obtained. After that and Perform matrix calculations to realize matrix The weighted operation yields Similarly, we can obtain Finally, by concatenating and changing the dimensions of the weighted features along the channel dimension, the summarized weighted features can be obtained.
[0099] Optionally, generate weighted features. The process can be expressed by the formula as follows:
[0100] Here, Softmax(·) represents the Softmax activation function, · represents the matrix multiplication operation, α is a learnable scaling parameter used to adjust the size of the matrix product, and Reshape(·) represents the operation of changing the dimensions of the matrix.
[0101] Optionally, the network structure of the first feedforward sub-network can be based on a feedforward neural network block (FFB) network structure, used to perform further feature fusion and nonlinear mapping on the weighted features obtained through the first self-attention sub-network.
[0102] Optionally, the first feedforward subnetwork can also be divided into two branches to extract and weightedly fuse the input features. The input of the first feedforward subnetwork is the output of the first self-attention subnetwork. After the output of the first feedforward subnetwork, the first deep features of the reconstructed block can be obtained.
[0103] Schematic illustration, as shown in Figure 9, illustrates a structural diagram of a first feedforward sub-network provided in an exemplary embodiment of this application. Taking input features as... For example, after inputting the features into the first feedforward sub-network, multi-layer convolution operations are performed on it in two paths to achieve feature extraction of the input features, resulting in... And on The Sigmoid nonlinear monotonic activation function is used to process the data to obtain weights, which are then used to... By weighting, we obtain the deep features.
[0104] Schematic illustration, as shown in Figure 10, illustrates the structure of the first feature extraction layer provided in an exemplary embodiment of this application. As shown in Figure 10, after the first shallow features are input into the first feature extraction layer, they first enter the first self-attention subnetwork (NEA). After feature extraction processing by the first self-attention subnetwork, the first weighted features are obtained, which are then input into the first feedforward subnetwork (FFB). The first deep features are output through the first feedforward subnetwork. Furthermore, a layer normalization layer is set before both the first self-attention subnetwork and the first feedforward subnetwork. By adding a layer normalization operation before the feature input network, the network's adaptability to multi-scale features can be improved.
[0105] In one possible implementation, taking the input of the first self-attention branch network as a normalized brightness reconstruction block I1 as an example, after convolution processing by the first convolutional layer, the first shallow feature M0 can be obtained. Therefore, the feature extraction process through the first feature extraction layer can be represented as M1(I1) = Transformer(M0). Here, Transformer(·) represents the operation of the first self-attention subnetwork, M1 is the first deep feature, and the feature dimension of M1 is 32×128×128, i.e.
[0106] The first decoding module may include a first feature fusion layer and a second convolutional layer. After obtaining the first deep features through the first encoding module, the decoder needs to further fuse these features through the decoding module. First, based on the first deep features, the decoder performs feature fusion through the first feature fusion layer to obtain the first fused features. Then, the first fused features are convolved through the second convolutional layer to obtain the first augmentation value of the reconstructed block.
[0107] Similar to the structure of the first feature extraction layer, the first feature fusion layer can also adopt a Bias Block module structure, comprising a second self-attention subnetwork and a second feedforward subnetwork. For the first deep feature output by the first feature extraction layer, the decoder first extracts and fuses features based on the first deep feature through the second self-attention subnetwork to obtain the second weighted feature. Then, based on the second weighted feature, it performs feature fusion and nonlinear mapping through the second feedforward subnetwork to obtain the first fused feature.
[0108] Similar to the first self-attention subnetwork, the second self-attention subnetwork also adopts a dual-headed NEA-based network structure. For a description of the second self-attention subnetwork's structure, please refer to the section on the first self-attention subnetwork described above; it will not be repeated here.
[0109] Similar to the first feedforward subnetwork, the second feedforward subnetwork can also be based on an FFB (Free Forward Bus) network structure. For a description of the second feedforward subnetwork's structure, please refer to the section on the first feedforward subnetwork; it will not be repeated here.
[0110] Optionally, the second convolutional layer is a 3×3 convolutional layer, which is used to perform 3×3 convolution and non-linear mapping operations on the first fused feature to obtain the first enhancement value of the reconstructed block.
[0111] Optionally, the feature fusion process through the first feature fusion layer can be represented as M2(Ι1) = Transformer(M1). Here, M2 is the first fused feature, and the feature dimension of M2 is 32×128×128, i.e. The convolution process using the second convolutional layer can be represented as C1 = σ(W2(M2)), where C1 is the first enhancement value, and the feature dimension of C1 is 1×128×128, i.e.
[0112] Step 702: Based on at least one of the prediction block and the residual block, and the reconstruction block, determine a second enhancement value for the reconstruction block through a first cross-attention branch network.
[0113] Optionally, the first cross-attention branch network determines a second enhancement value for the reconstructed block by learning the association features between the reconstructed block and the predicted block, or between the reconstructed block and the residual block, or between the reconstructed block and both the predicted and residual blocks. That is, the input of the first cross-attention branch network includes at least the reconstructed block, and may also include at least one of the predicted and residual blocks; the output is the second enhancement value of the reconstructed block.
[0114] Since the prediction block contains detailed block edge contour information, and the residual block provides information about quantization, using the prediction block and / or residual block as auxiliary inputs enables the first cross-attention branch network to improve the accuracy of the output second enhancement value based on the correlation features between the reconstruction block and the prediction block, or the reconstruction block and the residual block, or the reconstruction block and the prediction block and the residual block.
[0115] In some embodiments, the first cross-attention branch network includes a second encoding module and a second decoding module. The decoder can first extract features from the reconstructed block using the second encoding module based on at least one of the predicted block and the residual block, as well as the reconstructed block, to obtain a second deep feature of the reconstructed block. Then, based on at least one of the predicted block and the residual block, the second decoding module performs feature fusion on the second deep feature to obtain a second augmented value for the reconstructed block. By first extracting features and then fusing them, the first cross-attention branch network can fully learn the correlation features between the reconstructed block and the predicted block, or between the reconstructed block and the residual block, or between the reconstructed block and both the predicted and residual blocks, thereby improving the accuracy of the second augmented value.
[0116] The second encoding module may include a third convolutional layer and a second feature extraction layer. After inputting at least one of the predicted block and the residual block, as well as the reconstructed block, into the first cross-attention branch network, the decoder first performs convolution processing on the reconstructed block through the third convolutional layer to obtain the second shallow features of the reconstructed block, and then performs convolution processing on at least one of the predicted block and the residual block through the third convolutional layer to obtain at least one of the third shallow features of the predicted block and the fourth shallow features of the residual block. Then, based on at least one of the third shallow features and the fourth shallow features, as well as the second shallow features, the second feature extraction layer performs feature extraction to obtain the second deep features.
[0117] Optionally, the third convolutional layer is a 3×3 convolutional layer. Taking the input of the first cross-attention branch network as a reconstructed block, a predicted block, and a residual block as an example, three convolutional layers are needed to perform 3×3 convolution and nonlinear mapping operations on the predicted block, the residual block, and the reconstructed block, respectively. In one possible implementation, taking the input of the first cross-attention branch network as a normalized luminance reconstructed block I1, a luminance predicted block P1, and a luminance residual block R1 as an example, the feature dimensions of I1, P1, and R1 are 1×128×128, that is, I1, P1, The process of convolving I1, P1, and R1 through three third convolutional layers can be represented as N0(I1) = σ(W3(I1)), E0(I1) = σ(W3(P1)), F0(I1) = σ(W3(R1)), where σ is the ReLU activation function, W3 is a 3×3 convolutional kernel, N0 is the second shallow feature, E0 is the third shallow feature, and F0 is the fourth shallow feature. The feature dimensions of N0, E0, and F0 are 32×128×128, i.e., N0, E0, ...
[0118] Optionally, the second feature extraction layer can adopt a Cross Block module structure, which may include a first cross-attention subnetwork and a third feedforward subnetwork. For at least one of the third shallow features and the fourth shallow features output by the third convolutional layer, and the second shallow feature, in the second feature extraction layer, the decoder performs feature extraction and fusion through the first cross-attention subnetwork based on at least one of the third shallow features and the fourth shallow features, and the second shallow feature, to obtain a third weighted feature. Then, based on the third weighted feature, feature fusion and nonlinear mapping are performed through the third feedforward subnetwork to obtain the second deep feature.
[0119] The network structure of the first cross-attention sub-network can adopt a network structure based on the cross-attention mechanism (Crossformer Cross Attention Module, CCA). Thus, the first cross-attention sub-network can fully acquire the image features of the current block by extracting and fusing information from multiple sources, thereby achieving attentional weighting of the reconstructed information.
[0120] The first cross-attention subnetwork can adopt a single-head CCA-based network structure. The first cross-attention subnetwork is used to extract and fuse features. Schematic, as shown in Figure 11, illustrates the structure of the first cross-attention subnetwork provided in an exemplary embodiment of this application. Taking the input of the first cross-attention branch network as a normalized luminance reconstruction block I1, luminance prediction block P1, and luminance residual block R1 as an example, the first cross-attention subnetwork first needs to construct a query matrix Q, a key matrix K, and a value matrix V. Specifically, the second shallow feature N0 corresponding to reconstruction block I1 is used to construct the value matrix V. This network branch includes a 1×1 convolutional layer, a 3×3 convolutional layer, and a reshape layer. The second shallow feature N0 corresponding to reconstruction block I1, the third shallow feature E0 corresponding to prediction block P1, and the fourth shallow feature F0 corresponding to residual block R1 are used to construct the query matrix Q and the key matrix K. Both network branches include a concat layer, a 1×1 convolutional layer, a 3×3 convolutional layer, and a reshape layer. After constructing the query matrix Q, key matrix K, and value matrix V, the transpose attention mechanism weighting matrix can be calculated to obtain the weighted features.
[0121] Similar to the first and second feedforward subnetworks, the third feedforward subnetwork can also be based on an FFB-based network structure. For a description of the third feedforward subnetwork's structure, please refer to the section on the first feedforward subnetwork; it will not be repeated here.
[0122] Schematic illustration, as shown in Figure 12, illustrates the structure of the second feature extraction layer provided in an exemplary embodiment of this application. As shown in Figure 12, after the second, third, and fourth shallow features are input into the second feature extraction layer, they first enter the first cross-attention subnetwork (CCA). After feature extraction processing by the first cross-attention subnetwork, a third weighted feature is obtained. This third weighted feature is then input into the third feedforward subnetwork (FFB), which outputs the second deep feature. Furthermore, a layer normalization layer is set before both the first cross-attention subnetwork and the third feedforward subnetwork. By adding a layer normalization operation before the feature input network, the network's adaptability to multi-scale features can be improved.
[0123] In one possible implementation, taking the input of the first cross-attention branch network as a normalized luminance reconstruction block I1, luminance prediction block P1, and luminance residual block R1, after convolution processing by the third convolutional layer, the second shallow feature N0 corresponding to the reconstruction block I1, the third shallow feature E0 corresponding to the prediction block P1, and the fourth shallow feature F0 corresponding to the residual block R1 can be obtained. Therefore, the feature extraction process through the second feature extraction layer can be represented as N1(I1,P1,R1) = Transformer(N0). Here, N1 is the second deep feature, and the feature dimension of N1 is 32×128×128, i.e.
[0124] The second decoding module may include a second feature fusion layer and a fourth convolutional layer. After obtaining the second deep features through the second encoding module, the decoder needs to further fuse these features through the decoding module. First, based on at least one of the third and fourth shallow features, and the second deep features, feature fusion is performed through the second feature fusion network to obtain the second fused features. Then, the second fused features are convolved through the fourth convolutional layer to obtain the second enhancement value of the reconstructed block.
[0125] Taking the reconstruction block, prediction block and residual block as the input of the quality enhancement network in Figure 6 as an example, as shown in Figure 6, after the prediction block 602 and residual block 603 are processed by the convolution of the third convolutional layer, the resulting third shallow feature and fourth shallow feature need to be input into the second feature extraction layer and the second feature fusion layer.
[0126] Similar to the second feature extraction layer structure, the second feature fusion layer can also adopt a Cross Block module structure. The second feature fusion layer includes a second cross-attention subnetwork and a fourth feedforward subnetwork. For the second deep feature output by the second feature extraction layer, the decoder first extracts and fuses features based on at least one of the third and fourth shallow features, along with the second deep feature, through the second cross-attention subnetwork to obtain a fourth weighted feature. Then, based on the fourth weighted feature, feature fusion and nonlinear mapping are performed through the fourth feedforward subnetwork to obtain the second fused feature.
[0127] Similar to the first cross-attention subnetwork, the second cross-attention subnetwork can also adopt a CCA-based network structure. For a description of the second cross-attention subnetwork's structure, please refer to the section on the first cross-attention subnetwork described above; it will not be repeated here.
[0128] Similar to the first, second, and third feedforward subnetworks, the fourth feedforward subnetwork can also have an FFB-based network structure. For a description of the fourth feedforward subnetwork's structure, please refer to the section on the first feedforward subnetwork; it will not be repeated here.
[0129] Optionally, the fourth convolutional layer is a 3×3 convolutional layer, which is used to perform 3×3 convolution and non-linear mapping operations on the second fused features to obtain the second enhancement value of the reconstructed block.
[0130] Optionally, the feature fusion process through the second feature fusion layer can be represented as N2(Ι1,P1,R1) = Transformer(N1). Here, N2 is the second fused feature, and the feature dimension of N2 is 32×128×128, i.e. The convolution process through the fourth convolutional layer can be represented as C2 = σ(W4(N2)), where C2 is the second enhancement value, and the feature dimension of C2 is 1×128×128, i.e.
[0131] Step 703: Based on the first enhancement value and the second enhancement value, perform quality enhancement on the reconstructed block.
[0132] After obtaining the first enhancement value through the first self-attention branch network and the second enhancement value through the first cross-attention branch network, the decoder can fuse the first enhancement value, the second enhancement value, and the reconstructed block through the quality enhancement network to obtain the quality-enhanced reconstructed block.
[0133] In one possible implementation, in the quality enhancement network, learnable parameters can be set for the first enhancement value and the second enhancement value as weight coefficients, thereby dynamically adjusting the enhancement ratio of the first enhancement value and the second enhancement value to the reconstructed block, and improving the adaptability of the quality enhancement network to different compression distortion modes.
[0134] As illustrated in Figure 6, the first enhancement value corresponds to the first weighting coefficient α, and the second enhancement value corresponds to the second weighting coefficient β. Taking the input of the quality enhancement network as the normalized luminance reconstruction block I1 as an example, the quality-enhanced reconstruction block 604 can be represented as O1=I1+αC1+βC2.
[0135] In the above embodiments, determining the first enhancement value through the first self-attention branch network in the quality enhancement network can realize the capture of global dependencies within the image using the self-attention mechanism, which is beneficial for the quality enhancement network to understand the overall features of the image more comprehensively. Determining the second enhancement value through the first cross-attention branch network in the quality enhancement network can realize the understanding of the relationship between the reconstruction block, the prediction block, and the residual block using the cross-attention mechanism, which is beneficial for improving the quality enhancement network's ability to fuse cross-domain information.
[0136] The Quality Enhancement Network is a lightweight network for the Teacher Network.
[0137] Considering that using Transformer networks during video encoding and decoding inevitably impacts video encoding and decoding efficiency, this embodiment employs a lightweight quality enhancement network to enhance the quality of reconstructed blocks. To ensure good quality enhancement performance from this lightweight network, it needs to be trained using a corresponding teacher network first.
[0138] In this embodiment of the application, the quality enhancement network is obtained based on a lightweight processing of the teacher network.
[0139] Optionally, the teacher network also adopts a two-branch network architecture that combines self-attention and cross-attention mechanisms, with the teacher network including a second self-attention branch network and a second cross-attention branch network.
[0140] Optionally, in the teacher network, to achieve feature extraction at different levels and scales, the network structure of the second self-attention branch network is a U-net structure. The second self-attention branch network includes a third encoding module and a third decoding module, wherein the third encoding module includes at least two third feature extraction layers and the third decoding module includes at least two third feature fusion layers; the network structure of the second cross-attention branch network is a U-net structure, wherein the second cross-attention branch network includes a fourth encoding module and a fourth decoding module, wherein the fourth encoding module includes at least two fourth feature extraction layers and the fourth decoding module includes at least two fourth feature fusion layers.
[0141] Furthermore, in order to obtain a lightweight quality enhancement network, the first feature extraction layer in the quality enhancement network can be mapped to the first third feature extraction layer among at least two third feature extraction layers, the first feature fusion layer in the quality enhancement network can be mapped to the last third feature fusion layer among at least two third feature fusion layers, the second feature extraction layer in the quality enhancement network can be mapped to the first fourth feature extraction layer among at least two fourth feature extraction layers, and the second feature fusion layer in the quality enhancement network can be mapped to the last second feature fusion layer among at least two fourth feature fusion layers.
[0142] Schematic illustration, as shown in Figure 13, illustrates a lightweight processing of a teacher network provided by an exemplary embodiment of this application. The teacher network's second self-attention branch network includes four third feature extraction layers and corresponding four third feature fusion layers, while the second cross-attention branch network includes two fourth feature extraction layers and corresponding two fourth feature fusion layers.
[0143] The third feature extraction layer and the third feature fusion layer both adopt a Bias Block module structure, and each Bias Block module includes an NEA network and an FFB network; the fourth feature extraction layer and the fourth feature fusion layer both adopt a Cross Block module structure, and each Cross Block module includes a CCA network and an FFB network.
[0144] As shown in Figure 13, the first bias block of the second self-attention branch network in the teacher network corresponds to the first bias block of the first self-attention branch network in the quality enhancement network, and the last bias block of the second self-attention branch network corresponds to the last bias block of the first self-attention branch network in the quality enhancement network; the first cross block of the second cross attention branch network corresponds to the first cross block of the first cross attention branch network in the quality enhancement network, and the last cross block of the second cross attention branch network corresponds to the last cross block of the first cross attention branch network in the quality enhancement network.
[0145] In some embodiments, to achieve feature extraction and fusion at different levels and scales, for the second self-attention branch network, the four third feature extraction layers can be divided into two stages for feature extraction, and the features are downsampled and the channel dimension is doubled at each stage; the four third feature fusion layers can also be divided into two stages for feature fusion, and the features are upsampled and the channel dimension is halved at each stage. For the second cross-attention branch network, the features can also be downsampled and the channel dimension doubled between the two fourth feature extraction layers, and the features can be upsampled and the channel dimension halved between the two fourth feature fusion layers.
[0146] Taking the second self-attention branch network as an example, this paper explains the process of feature extraction and fusion at different levels and scales.
[0147] As illustrated in FIG14, it shows a schematic diagram of the structure of a teacher network provided in an exemplary embodiment of this application.
[0148] For the feature encoding process, the third feature encoding module includes a 3×3 convolutional layer and four third feature extraction layers.
[0149] First, the normalized brightness reconstruction block I1 is input into the second self-attention branch network. The brightness reconstruction block I1 enters a single 3×3 convolutional layer. After 3×3 convolution and non-linear mapping operations, shallow feature extraction and channel number increase are achieved, thus obtaining shallow features. This process can be expressed by the formula: M0(I1)=σ(W1(I1)),
[0150] Where σ represents the ReLU activation function and W1 represents a 3×3 convolution kernel.
[0151] Secondly, the shallow features M0 are input into the multi-level feature extraction layer of the first stage, and after passing through the first Bias Block module, the features are obtained. Feature M1 is downsampled, reducing its width and height to half their original dimensions while doubling its channel dimension. The downsampled feature M1 is then input into the second Bias Block module to obtain the feature... Clearly, features M1 and M2 represent features at different levels and scales. This process can be expressed by the following formula:
[0152] Wherein, Transformer(·) represents the Transformer operation, W2 represents the 3×3 convolution kernel, Pixelunshuffle(·) represents the inverse sample rearrangement operation, and downsample(·) represents the downsampling operation. The downsampling operation is specifically composed of a 3×3 convolution operation that halves the number of channels and an inverse sample rearrangement operation (pixelunshuffle) with a sampling factor of 2.
[0153] Next, feature M2 is input into the second-stage multi-level feature extraction layer for further feature extraction at different levels. After passing through the third Bias Block module, the features are obtained. Similarly, feature M3 is downsampled, reducing its width and height to half their original dimensions while doubling its channel dimensions. The downsampled feature M3 is then input into the fourth Bias Block module to obtain the feature... This process can be expressed by the following formula:
[0154] Each Bias Block, the third feature extraction layer, consists of a cascaded self-attention sub-network and a feedforward sub-network. Within each Bias Block, the output of the self-attention sub-network serves as the input to the feedforward sub-network, and the output of the feedforward sub-network is the final output of the current Bias Block. The self-attention sub-network is used for feature extraction and weighted fusion, while the feedforward sub-network is used for feature fusion and non-linear mapping.
[0155] During the feature decoding process, the same number of Bias Block modules as those used in the feature extraction process are employed. After channel splicing of features at different levels and scales, multi-scale feature extraction and weighted fusion are performed to complete the in-depth mining of features and the fusion of multi-level features, so as to better restore the lost details and obtain the first enhancement value.
[0156] For the feature decoding process, the third feature decoding module includes four third feature fusion layers and one 3×3 convolutional layer. Among them, the two-stage multi-level feature fusion layers realize feature fusion at different levels and scales, while the single 3×3 convolutional layer realizes deep feature fusion and reduces the number of channels to obtain the enhanced value.
[0157] Optionally, the third feature decoding module can be understood as the inverse operation of the third feature encoding module. Feature M4 is further input into four cascaded Bias Block modules for feature extraction. Upsampling is used to perform scale matching of pairs of features, and the scale-matched features are then concatenated along the channel.
[0158] First, feature M4 is upsampled, doubling its width and height while halving its channel count. The upsampled M4 and feature M3 are then concatenated along the channel dimension. A 1×1 convolution is used to fuse the concatenated features and halve the channel count. The concatenated features with halved channels are then input into the first-stage multi-layer feature fusion layer. After passing through the first Bias Block module, the feature is obtained. Furthermore, after concatenating features M5 and M2, a 1×1 convolution is used to fuse the concatenated features and halve the number of channels. This result is then input into the second Bias Block module to obtain the feature... This process can be expressed by the following formula:
[0159] In this context, Transformer(·) represents the Transformer operation, W3 represents a 3×3 convolution kernel, Pixelunshuffle(·) represents the inverse sample rearrangement operation, and upsample(·) represents the upsampling operation. The upsampling operation is specifically composed of a 3×3 convolution operation that halves the number of channels and a sample rearrangement operation (pixelshuffle) with a sampling factor of 2.
[0160] Next, feature M6 is upsampled, doubling its width and height while halving its channel count. The upsampled feature M6 is then concatenated with feature M1 along the channel dimension. A 1×1 convolution is used to fuse the concatenated features and halve the channel count. The concatenated feature with halved channels is then input into the second-stage multi-level feature fusion layer. After passing through the third Bias Block module, the final feature is obtained. The feature M7 is then input into the fourth Bias Block module to obtain the feature. This process can be expressed by the following formula:
[0161] In this context, Transformer(·) represents the Transformer operation, W4 represents a 3×3 convolution kernel, Pixelunshuffle(·) represents the inverse sample rearrangement operation, and upsample(·) represents the upsampling operation. The upsampling operation is specifically composed of a 3×3 convolution operation that halves the number of channels and a sample rearrangement operation (pixelshuffle) with a sampling factor of 2.
[0162] Finally, the feature M8 is input into a single 3×3 convolutional layer. After 3×3 convolution and non-linear mapping operations, the final feature fusion and channel number reduction are achieved, thus obtaining the first enhancement value. This process can be expressed by the formula: C1=σ(W5(M8)),
[0163] Where W5 represents a 3×3 convolution kernel.
[0164] As can be seen from the above process, each Bias Block module corresponds to its own feature input size, which are respectively
[0165] Taking the second cross-attention branch network as an example, this paper explains the process of feature extraction and fusion at different levels and scales.
[0166] As illustrated in Figure 14, the input to the second cross-attention branch includes a normalized luminance reconstruction block I1, a luminance prediction block P1, and a luminance residual block R1. For the feature encoding process, the fourth feature encoding module includes three 3×3 convolutional layers and two fourth feature extraction layers.
[0167] First, the normalized luminance reconstruction block I1, luminance prediction block P1, and luminance residual block R1 are input into the second cross-attention branch. The luminance reconstruction block I1 enters a single 3×3 convolutional layer. After 3×3 convolution and nonlinear mapping operations, shallow feature extraction and channel number increase are achieved, thus obtaining shallow features. The brightness prediction block P1 enters a single 3×3 convolutional layer. After 3×3 convolution and non-linear mapping operations, shallow feature extraction and channel number increase are achieved, thus obtaining shallow features. The luminance residual block R1 enters a single 3×3 convolutional layer. After 3×3 convolution and nonlinear mapping operations, shallow feature extraction and channel number increase are achieved, thus obtaining shallow features. This process can be expressed by the following formula:
[0168] Where σ represents the ReLU activation function, and W1, W2, and W3 represent 3×3 convolution kernels.
[0169] Secondly, the shallow features N0, E0, and F0 are input into the multi-level feature extraction layer, and after passing through the first Cross Block module, the features are obtained. Feature N1 is downsampled, reducing its width and height to half their original dimensions while doubling its channel dimensions. The downsampled feature N1 is then input into the second Cross Block module to obtain the feature... Clearly, features N1 and N2 represent features at different levels and scales. This process can be expressed by the following formula:
[0170] In this context, Transformer(·) represents the Transformer operation, W4 represents a 3×3 convolution kernel, Pixelunshuffle(·) represents the inverse sample rearrangement operation, and downsample(·) represents the downsampling operation. The downsampling operation is specifically composed of a 3×3 convolution operation that halves the number of channels and an inverse sample rearrangement operation (pixelunshuffle) with a sampling factor of 2.
[0171] It should be noted that while performing downsampling on feature N1 to reduce its width and height to half of their original size and double its channel dimension, in order to ensure that the predicted features and residual features input into the second Cross Block module also conform to the feature input scale of the second Cross Block module, it is also necessary to perform downsampling on the shallow features E0 corresponding to the brightness prediction block and the shallow features F0 corresponding to the brightness residual block (not shown in Figure 14), so that the feature dimensions of the downsampled features E0 and F0 are both 64×64×64.
[0172] During the feature decoding process, the same number of Cross Block modules as those used in the feature extraction process are employed. After channel splicing of features at different levels and scales, multi-scale feature extraction and weighted fusion are performed to achieve in-depth feature mining and multi-level feature fusion, thereby better restoring lost detail information and obtaining the second enhancement value.
[0173] For the feature decoding process, the fourth feature decoding module includes two fourth feature fusion layers and a 3×3 convolutional layer.
[0174] Optionally, the fourth feature decoding module can be understood as the inverse operation of the fourth feature encoding module. Feature N2 is further input into two cascaded Cross Block modules for feature extraction. Upsampling is used to perform scale matching of pairs of features, and the scale-matched features are then concatenated along the channel.
[0175] First, feature N2 is upsampled, doubling its width and height while halving its channel count. The upsampled features N2 and N1 are then concatenated along the channel dimension. A 1×1 convolution is used to fuse the concatenated features and halve the channel count. The concatenated features with halved channels are then input into a multi-layer feature fusion layer. After passing through the first Cross Block module, the final feature is obtained. The feature N3 is then input into the second Cross Block module to obtain the feature. This process can be expressed by the following formula:
[0176] In this context, Transformer(·) represents the Transformer operation, W5 represents a 3×3 convolution kernel, Pixelunshuffle(·) represents the inverse sample rearrangement operation, and upsample(·) represents the upsampling operation. The upsampling operation is specifically composed of a 3×3 convolution operation that halves the number of channels and a sample rearrangement operation (pixelshuffle) with a sampling factor of 2.
[0177] It should be noted that for the two Cross Block modules in the feature decoding stage, the feature dimensions of the shallow feature E0 corresponding to the input luminance prediction block and the shallow feature F0 corresponding to the luminance residual block are both 32×128×128.
[0178] Finally, feature N4 is input into a single 3×3 convolutional layer. After 3×3 convolution and non-linear mapping operations, the final feature fusion and channel number reduction are achieved, thus obtaining the second enhancement value. This process can be expressed by the formula: C2=σ(W6(N4)),
[0179] Where W6 represents a 3×3 convolution kernel.
[0180] As can be seen from the above process, each Cross Block module corresponds to its own feature input size, which are respectively
[0181] The training process of the quality enhancement network.
[0182] To ensure the quality enhancement quality of the quality enhancement network, it needs to be trained before being embedded into the coding framework. Optionally, the training process of the quality enhancement network can be divided into training the teacher network first, and then training the quality enhancement network.
[0183] A two-stage training strategy can be adopted for training the teacher network. The first stage mainly trains the second self-attention branch network, using the self-attention mechanism to extract global contextual features. The second stage, based on fixing the second self-attention branch network, focuses on optimizing the second cross-attention branch network, making the network more focused on the recovery of local details.
[0184] For the first stage of teacher network training, the training set can consist of 800 training images from the DIV2K dataset. During training, VTM-11.0_NNVC-6.0 was used, with quantization parameters (QP) set to 22, 27, 32, 37, 42, in All Intra (AI) mode, and loop filters LMCS, DBF, SAO, and ALF disabled. This resulted in 4000 encoded images. These 4000 encoded images were then cropped into 128×128 encoded blocks. After removing blocks whose content remained unchanged before and after encoding and decoding, a total of 607652 encoded blocks were obtained. These 607652 encoded blocks were used as input to the teacher network, and the corresponding unencoded original blocks were used as ground truth, forming the final training set. For the test set, the first frame of all sequences from Class A1, Class A2, Class B, Class C, Class D, Class E, and Class F, as specified in the general test conditions defined by the video coding standard, can be used, totaling 26 images. Using VTM-11.0_NNVC-6.0 with QP set to 22, 27, 32, 37, 42, AI mode, and with loop filters LMCS, DBF, SAO, and ALF disabled, these 26 images are encoded, resulting in 130 encoded images. These 130 encoded images are then cropped into 128×128 encoded blocks. After removing blocks whose content remains unchanged before and after encoding and decoding, a total of 19714 encoded blocks are obtained. These 19714 encoded blocks are used as input to the teacher network, and the corresponding unencoded raw blocks are used as ground truth, forming the final test set.
[0185] For the second stage of teacher network training, the training set can be constructed using the first frame of the BVI-DVC dataset, totaling 100 images. Using VTM-11.0_NNVC-6.0 with QP settings of 22, 27, 32, 37, 42, AI mode, and with loop filters LMCS, DBF, SAO, and ALF disabled, the 100 images are encoded, resulting in 500 encoded images. Simultaneously, the encoded reconstructed prediction and residual images are extracted. The encoded reconstructed, predicted, and residual images are then segmented at corresponding locations into 128×128 blocks. The original images are also segmented at corresponding locations. Then, the encoded reconstructed blocks are compared with their corresponding original blocks. If the differences between the two blocks are minimal, the reconstructed block is discarded, resulting in 577,710 image pairs, forming the final training set.
[0186] Optionally, for the two-stage training of the teacher network, a loss function based on mean squared error (MSE) can be used. The initial learning rate can be set to 1×10⁻⁶. 5 There are a total of 30 epochs, and the learning rate decreases to its original value every 10 epochs. Adam was used as the optimizer and trained in batch mode with a batch size of 16 for each group.
[0187] After training the teacher network, the proposed quality enhancement network can be trained. The quality enhancement network can be trained using either a single-stage or two-stage training strategy. Taking a single-stage training strategy for the quality enhancement network as an example, the training set used for the second stage of training of the teacher network, namely the BVI-DVC dataset, can be used.
[0188] Optionally, for single-stage training of the quality enhancement network, the initial learning rate can be set to 1×10⁻⁶. 5 There are a total of 30 epochs, and the learning rate decreases to its original value every 10 epochs. Adam was used as the optimizer and trained in batch mode with a batch size of 16 for each group.
[0189] Optionally, training the quality enhancement network requires the use of distillation loss between the quality enhancement network and the teacher network, as well as mean squared error loss between the reconstructed block and the original block. In other words, the quality enhancement network is trained based on the distillation loss between the quality enhancement network and the teacher network, and the mean squared error loss between the reconstructed block and the original block.
[0190] The distillation loss includes a first distillation loss, a second distillation loss, a third distillation loss, and a fourth distillation loss. The first distillation loss is determined by combining the deep features of the first sample output from the first feature extraction layer in the quality enhancement network with the deep features of the third sample output from the first of at least two third feature extraction layers. The second distillation loss is determined by combining the fused features of the first sample output from the first feature fusion layer in the quality enhancement network with the fused features of the third sample output from the last of at least two third feature fusion layers. The third distillation loss is determined by combining the deep features of the second sample output from the second feature extraction layer in the quality enhancement network with the deep features of the fourth sample output from the first of at least two fourth feature extraction layers. The fourth distillation loss is determined by combining the fused features of the second sample output from the second feature fusion layer in the quality enhancement network with the fused features of the fourth sample output from the last of at least two fourth feature fusion layers.
[0191] As illustrated in Figure 13, the first distillation loss (Distillation Loss 1) is the feature loss between the first deep feature of the first sample output by the first Bias Block module of the first self-attention branch network in the quality enhancement network and the third deep feature of the third sample output by the first Bias Block module of the second self-attention branch network in the teacher network; the second distillation loss (Distillation Loss 2) is the feature loss between the first fused feature of the first sample output by the last Bias Block module of the first self-attention branch network in the quality enhancement network and the third fused feature of the third sample output by the last Bias Block module of the second self-attention branch network in the teacher network. The third distillation loss (Distillation Loss 3) is the feature loss between the second deep feature output by the first Cross Block module of the first cross attention branch network in the quality enhancement network and the fourth deep feature output by the first Cross Block module of the second cross attention branch network in the teacher network; the fourth distillation loss (Distillation Loss 4) is the feature loss between the second fused feature output by the last Cross Block module of the first cross attention branch network in the quality enhancement network and the fourth fused feature output by the last Cross Block module of the second cross attention branch network in the teacher network.
[0192] In the above embodiments, after completing the phased training of the teacher network, the quality enhancement network proposed in this application is obtained by lightweighting the teacher network. Simultaneously, by combining distillation loss and mean squared error loss, the quality enhancement network is trained, which can effectively improve the quality enhancement quality. Furthermore, compared to directly embedding the teacher network into the coding framework, embedding the lightweight quality enhancement network into the coding framework can improve the reconstruction quality of the reconstructed blocks while reducing network complexity, minimizing the impact on encoding / decoding efficiency, reducing the consumption of computational and storage resources, and shortening the encoding / decoding time.
[0193] Please refer to Figure 15, which illustrates a flowchart of an image encoding method provided in an exemplary embodiment of this application. The method is applied to an encoder and may include the following steps:
[0194] Step 1501: Based on at least one of the predicted block and residual block of the current block, and the reconstructed block of the current block, the reconstructed block is quality-enhanced using a quality enhancement network.
[0195] In a block-based hybrid coding framework, a frame of video is divided into square CTU blocks of the same size. The current block refers to the coding block currently being processed by the encoder, i.e., the CTU block.
[0196] In this embodiment, to improve the quality of the reconstructed blocks, a quality enhancement network is used to enhance the quality of the reconstructed blocks. Optionally, the input to the quality enhancement network includes at least the reconstructed block of the current block, and at least one of the predicted block and the residual block can be used as auxiliary input information to help the quality enhancement network fully learn the image features of the current block. That is, the input to the quality enhancement network can be the reconstructed block and the predicted block, the reconstructed block and the residual block, or the reconstructed block, the predicted block, and the residual block. The output of the quality enhancement network is the reconstructed block after quality enhancement.
[0197] In some embodiments, the process of enhancing the quality of a reconstructed block using a quality enhancement network, based on at least one of the predicted block and residual block of the current block, and the reconstructed block of the current block, is independent of the loop filtering process and can be performed before loop filtering. That is, the encoder first enhances the quality of the reconstructed block, and then performs loop filtering on the enhanced reconstructed block.
[0198] During the encoding process, the encoder first divides the image into multiple coding blocks. Then, it uses intra-frame or inter-frame prediction (including motion estimation and motion compensation) algorithms to generate a prediction block for the current block, and subtracts the prediction block from the original block to obtain a residual block. Furthermore, the encoder reconstructs the image based on the inverse quantization / inverse transform results and the prediction blocks to obtain a reconstructed block. Before performing loop filtering on the reconstructed block, the encoder first enhances the quality of the reconstructed block using a quality enhancement network, based on at least one of the prediction block and residual block, as well as the reconstructed block, to obtain a quality-enhanced reconstructed block. Loop filtering is then performed on the quality-enhanced reconstructed block.
[0199] In summary, the embodiments of this application fully consider the quantization distortion in the encoding and decoding process, add quality enhancement to the hybrid coding framework, and enhance the quality of the reconstructed block through the quality enhancement network based on at least one of the prediction block and residual block of the current block, as well as the reconstructed block of the current block. This can improve the reconstruction quality of the current block and provide a more accurate reference for intra-frame prediction of subsequent coding blocks.
[0200] In some embodiments, in order to improve the feature learning effect of the quality enhancement network on the reconstructed information, in addition to the predicted block and residual block of the current block, the neighboring reconstructed blocks of the reconstructed block can also be used as network inputs, so that the quality enhancement network can learn the correlation features between the reconstructed block and the neighboring reconstructed blocks, thereby improving the quality enhancement quality of the reconstructed block.
[0201] That is, the encoder enhances the quality of the reconstructed block through a quality enhancement network based on at least one of the predicted block, residual block and adjacent reconstructed blocks of the current block, as well as the reconstructed block of the current block.
[0202] Optionally, the adjacent reconstructed blocks of the reconstructed block can be blocks located within the γ-shaped region of the reconstructed block. The γ-shaped region refers to the left and upper regions of the reconstructed block in the current image.
[0203] Due to considerations of encoding / decoding costs and quality, quality enhancement of reconstructed blocks is not necessary in all scenarios. Therefore, before enhancing the quality of a reconstructed block, the encoder must first determine whether quality enhancement is required and instruct the decoder via an enable flag in the bitstream.
[0204] In one possible implementation, the encoder can determine whether to enhance the quality of the reconstructed block based on the rate-distortion cost of using different encoding methods (enhancing the quality of the reconstructed block or not enhancing the quality of the reconstructed block).
[0205] In some embodiments, as shown in FIG16, the process of determining whether to enhance the quality of the reconstructed block during the encoding process may include the following steps.
[0206] Step 1601: Determine the first rate-distortion cost and the second rate-distortion cost. The first rate-distortion cost is the rate-distortion cost of not performing quality enhancement on the reconstructed block. The second rate-distortion cost is the rate-distortion cost of performing quality enhancement on the reconstructed block through a quality enhancement network based on at least one of the predicted block and the residual block, and the reconstructed block of the current block.
[0207] In some embodiments, when no quality enhancement is performed on the reconstructed block, the encoder determines a first rate-distortion cost based on the distortion value (which may be mean square error, absolute error, etc.) and rate of the reconstructed block using a rate-distortion cost function; after the reconstructed block is quality enhanced by a quality enhancement network based on at least one of the predicted block and the residual block, and the reconstructed block of the current block, the encoder determines a second rate-distortion cost based on the distortion value (which may be mean square error, absolute error, etc.) and rate of the quality-enhanced reconstructed block using a rate-distortion cost function.
[0208] Step 1602: Based on the comparison result of the first rate distortion cost and the second rate distortion cost, the value of the enable flag bit is encoded into the bitstream. The value of the enable flag bit is used to indicate whether the reconstructed block needs to be enhanced in quality.
[0209] Optionally, the value of the enable flag is used to characterize whether the reconstructed block of the current block needs quality enhancement. Optionally, the encoder compares the first rate-distortion cost and the second rate-distortion cost.
[0210] In one possible implementation, if the cost of the first rate distortion is greater than the cost of the second rate distortion, it indicates that the quality of the reconstructed block can be significantly improved by using a quality enhancement network compared to not performing quality enhancement on the reconstructed block. Therefore, the encoder determines that the reconstructed block needs quality enhancement and sets the value of the enable flag to true (or 1).
[0211] In another possible implementation, if the first rate distortion cost is less than or equal to the second rate distortion cost, it indicates that enhancing the quality of the reconstructed block through the quality enhancement network will not significantly improve the quality of the reconstructed block compared to not enhancing the quality of the reconstructed block. Therefore, the encoder determines that the reconstructed block does not need to be enhanced and sets the value of the enable flag to false (or 0).
[0212] Optionally, the value of the enable flag is used to indicate whether the luminance reconstruction block of the current block needs quality enhancement. Optionally, when the luminance reconstruction block needs quality enhancement, the encoder sets the value of the enable flag to 1; when the luminance reconstruction block does not need quality enhancement, the encoder sets the value of the enable flag to 0.
[0213] In one possible implementation, after the brightness reconstruction block is enhanced by a brightness quality enhancement network based on at least one of the brightness prediction block and brightness residual block of the current block, and the brightness reconstruction block of the current block, the encoder determines a third rate distortion cost based on the distortion value (which may be mean square error, absolute error, etc.) and rate of the enhanced brightness reconstruction block, and then encodes the value of the enable flag bit into the bitstream based on the comparison result of the first rate distortion cost and the third rate distortion cost.
[0214] When the cost of first rate distortion is greater than that of third rate distortion, it indicates that enhancing the quality of the luminance reconstruction block through the luminance quality enhancement network can significantly improve the quality of the luminance reconstruction block. Therefore, the encoder determines that the luminance reconstruction block needs to be enhanced and sets the value of the enable flag to true (or 1).
[0215] If the first rate distortion cost is less than or equal to the third rate distortion cost, it indicates that enhancing the quality of the luminance reconstruction block through the luminance quality enhancement network will not significantly improve the quality of the luminance reconstruction block. Therefore, the encoder determines that the luminance reconstruction block does not need to be enhanced and sets the value of the enable flag to false (or 0).
[0216] Optionally, a separate luminance enable flag can be set for the luminance reconstruction block. The luminance enable flag is named luma_enhanceformer_enable_flag, and its value is used to indicate whether the current block's luminance reconstruction block needs quality enhancement.
[0217] As illustrated in Figure 17, at the encoder end, after obtaining the luminance reconstruction block, the encoder enhances the luminance reconstruction block's quality through a luminance quality enhancement network based on at least one of the current block's luminance prediction block and luminance residual block, as well as the current block's reconstruction block. Then, based on the distortion value (which can be mean square error, absolute error, etc.) and rate of the enhanced luminance reconstruction block, a third rate-distortion cost is determined using a rate-distortion cost function. Illustratively, the formula for calculating the third rate-distortion cost can be expressed as: J = SSE(x,y) + λ CTU R exlraflag ,
[0218] Where x is the original block, y is the quality-enhanced luminance reconstruction block, and λ CTU R is the Lagrange factor at the CTU level. extraflag The bitrate spent on additional encoding enable flags.
[0219] Next, the encoder compares the third rate-distortion cost with the first rate-distortion cost. When the rate-distortion cost decreases, it indicates that enhancing the quality of the luminance reconstruction block through the luminance quality enhancement network significantly improves the quality of the luminance reconstruction block; therefore, the encoder sets the luminance enable flag to 1. When the rate-distortion cost increases, it indicates that enhancing the quality of the luminance reconstruction block through the luminance quality enhancement network does not significantly improve the quality of the luminance reconstruction block; therefore, the encoder sets the luminance enable flag to 0. Finally, the encoder encodes the luminance component residual, the optimal luminance prediction mode, and the luminance enable flag into the bitstream.
[0220] At the decoding end, the decoder parses the bitstream, determines the luminance enable flag, luminance component residuals, and the optimal luminance prediction mode. Based on the optimal luminance prediction mode, it uses intra-frame or inter-frame prediction algorithms to predict the luminance prediction block for the current block. Simultaneously, it performs entropy decoding on the bitstream to obtain quantization coefficients. Combining these coefficients with the luminance component residuals, it performs inverse quantization and inverse transform to obtain the luminance residual block, thus obtaining the luminance reconstruction block. When the luminance enable flag is 1, the luminance reconstruction block is enhanced using a luminance quality enhancement network, resulting in a quality-enhanced luminance reconstruction block. When the luminance enable flag is 0, the luminance reconstruction block is not enhanced using the luminance quality enhancement network. Finally, a loop filter is applied to either the quality-enhanced luminance reconstruction block or the luminance reconstruction block itself, and the decoded image is generated based on the loop-filtered reconstruction block.
[0221] Optionally, the enable flag value is used to indicate whether the chroma reconstruction block of the current block needs quality enhancement. Optionally, the encoder sets the enable flag value to 1 when the chroma reconstruction block needs quality enhancement; the encoder sets the enable flag value to 0 when the chroma reconstruction block does not need quality enhancement.
[0222] In one possible implementation, after the chroma reconstruction block is enhanced by a chroma quality enhancement network based on at least one of the chroma prediction block and chroma residual block of the current block, and the chroma reconstruction block of the current block, the encoder determines a fourth rate distortion cost based on the distortion value (which may be mean square error, absolute error, etc.) and rate of the enhanced chroma reconstruction block, and then encodes the value of the enable flag bit into the bitstream based on the comparison result of the first rate distortion cost and the fourth rate distortion cost.
[0223] When the cost of first rate distortion is greater than that of fourth rate distortion, it indicates that the quality of the chroma reconstruction block can be significantly improved by using a chroma quality enhancement network. Therefore, the encoder determines that the chroma reconstruction block needs quality enhancement and sets the value of the enable flag to true (or 1).
[0224] If the first rate distortion cost is less than or equal to the fourth rate distortion cost, it indicates that enhancing the chroma reconstruction block through the chroma quality enhancement network will not significantly improve the quality of the chroma reconstruction block. Therefore, the encoder determines that the chroma reconstruction block does not need to be enhanced and sets the value of the enable flag to false (or 0).
[0225] In another possible implementation, the chroma reconstruction block is enhanced by a first chroma quality enhancement network based on at least one of the chroma prediction block, chroma residual block, and luminance reconstruction block of the current block, and the chroma reconstruction block of the current block; and / or, the chroma reconstruction block is enhanced by a second chroma quality enhancement network based on at least one of the chroma prediction block, chroma residual block, and quality-enhanced luminance reconstruction block of the current block, and the chroma reconstruction block of the current block. The encoder determines a fifth rate distortion cost based on the distortion value (which may be mean square error, absolute error, etc.) and rate of the quality-enhanced chroma reconstruction block through a rate distortion cost function. Based on the comparison result of the first rate distortion cost and the fifth rate distortion cost, the value of the enable flag is encoded into the bitstream.
[0226] When the cost of first-rate distortion is greater than that of fifth-rate distortion, it indicates that the quality of the chroma reconstruction block can be significantly improved by using a chroma quality enhancement network. Therefore, the encoder determines that the chroma reconstruction block needs quality enhancement and sets the value of the enable flag to true (or 1).
[0227] If the first rate distortion cost is less than or equal to the fifth rate distortion cost, it indicates that enhancing the chroma reconstruction block through the chroma quality enhancement network will not significantly improve the quality of the chroma reconstruction block. Therefore, the encoder determines that the chroma reconstruction block does not need to be enhanced and sets the value of the enable flag to false (or 0).
[0228] Optionally, a separate chroma enable flag can be set for the chroma reconstruction block. The chroma enable flag is named chroma_enhanceformer_enable_flag, and its value is used to indicate whether the chroma reconstruction block of the current block needs quality enhancement.
[0229] Optionally, the enable flag can be used to indicate whether quality enhancement is performed on the reconstructed block in two dimensions, that is, whether the value of the enable flag indicates whether the luma and chroma reconstruction blocks of the current block need quality enhancement respectively. Optionally, the value of the enable flag can be a two-bit binary number. When both the luma and chroma reconstruction blocks need quality enhancement, the encoder sets the value of the enable flag to 11; when the luma reconstruction block needs quality enhancement but the chroma reconstruction block does not, the encoder sets the value of the enable flag to 10; when the luma reconstruction block does not need quality enhancement but the chroma reconstruction block does, the encoder sets the value of the enable flag to 01; when neither the luma nor the chroma reconstruction block needs quality enhancement, the encoder sets the value of the enable flag to 00.
[0230] Regarding the method of setting the enable flag, in one possible implementation, the encoder can set the enable flag corresponding to the current block based on the rate-distortion cost comparison result corresponding to the current block, that is, different blocks in the same image correspond to their own enable flags.
[0231] In another possible implementation, the encoder can accumulate rate-distortion costs corresponding to different blocks within the same slice to obtain a first rate-distortion cost accumulation value (obtained by accumulating the first rate-distortion costs of different blocks within the same slice) and a second rate-distortion cost accumulation value (obtained by accumulating the second rate-distortion costs of different blocks within the same slice). Based on the comparison result of the first rate-distortion cost accumulation value and the second rate-distortion cost accumulation value, the encoder sets the enable flag bit corresponding to the slice, that is, different blocks within the same slice in the same image correspond to the same enable flag bit.
[0232] In another possible implementation, the encoder can accumulate rate-distortion costs corresponding to different blocks in the same image to obtain a first rate-distortion cost accumulation value (obtained by accumulating the first rate-distortion costs of different blocks in the same image) and a second rate-distortion cost accumulation value (obtained by accumulating the second rate-distortion costs of different blocks in the same image). Based on the comparison result of the first rate-distortion cost accumulation value and the second rate-distortion cost accumulation value, the encoder sets the corresponding enable flag bit of the image, that is, different blocks in the same image correspond to the same enable flag bit.
[0233] In another possible implementation, the encoder can accumulate rate-distortion costs corresponding to different blocks in the same image group to obtain a first rate-distortion cost accumulation value (obtained by accumulating the first rate-distortion costs of different blocks in the same image group) and a second rate-distortion cost accumulation value (obtained by accumulating the second rate-distortion costs of different blocks in the same image group). Based on the comparison result of the first rate-distortion cost accumulation value and the second rate-distortion cost accumulation value, the encoder sets the enable flag bit corresponding to the image group, that is, different blocks of different images in the same image group correspond to a unified enable flag bit.
[0234] The process of enhancing the quality of reconstructed blocks using a quality enhancement network.
[0235] In some embodiments, to achieve both quality enhancement of the reconstructed block based on its own image features and quality enhancement based on the correlation features between the reconstructed block, the predicted block, and / or the residual block, the quality enhancement network can employ a two-branch network architecture combining self-attention and cross-attention mechanisms. That is, the quality enhancement network includes a first self-attention branch network and a first cross-attention branch network.
[0236] As illustrated in Figure 6, the quality enhancement network uses residual connections to directly pass the reconstructed block 601 to the output of the branch network. After the enhancement value is output through the first self-attention branch network 611 and the first cross-attention branch network 612, the enhancement value is added to the reconstructed block 601 to obtain the quality-enhanced reconstructed block 604.
[0237] The process of enhancing the quality of reconstructed blocks using a quality enhancement network will be described below. This process may include the following steps:
[0238] 1. Based on the reconstruction block, the first enhancement value of the reconstruction block is determined through the first self-attention branch network.
[0239] Optionally, the first self-attention network determines the first enhancement value of the reconstructed block by learning the image features of the reconstructed block itself. That is, the input of the first self-attention branch network is the reconstructed block, and the output is the first enhancement value of the reconstructed block.
[0240] In some embodiments, the first self-attention branch network includes a first encoding module and a first decoding module. The decoder can first extract features from the reconstructed block through the first encoding module to obtain the first deep features of the reconstructed block, and then perform feature fusion on the first deep features through the first decoding module to obtain the first enhancement value of the reconstructed block. By extracting features first and then fusing features, it can be ensured that the first self-attention branch network can fully learn the image features of the reconstructed block, thereby improving the accuracy of the first enhancement value.
[0241] The first encoding module may include a first convolutional layer and a first feature extraction layer. After the reconstructed block is input into the first self-attention branch network, the decoder first performs convolution processing on the reconstructed block through the first convolutional layer to obtain the first shallow features of the reconstructed block. Then, based on the first shallow features, the first feature extraction layer performs feature extraction to obtain the first deep features.
[0242] Optionally, the first convolutional layer is a 3×3 convolutional layer, used to perform 3×3 convolution and non-linear mapping operations on the input reconstructed block. In one possible implementation, taking the input of the first self-attention branch network as a normalized brightness reconstructed block I1 as an example, the feature dimension of I1 is 1×128×128, i.e. The process of convolving I1 through the first convolutional layer can be represented as M0(I1)=σ(W1(I1)). Here, σ is the ReLU activation function, W1 is a 3×3 convolutional kernel, and M0 is the first shallow feature, with a feature dimension of 32×128×128.
[0243] Optionally, the first feature extraction layer may adopt a Bias Block module structure, and the first feature extraction layer may include a first self-attention subnetwork and a first feedforward subnetwork. For the first shallow features output by the first convolutional layer, in the first feature extraction layer, the decoder first extracts and fuses features based on the first shallow features through the first self-attention subnetwork to obtain the first weighted features, and then performs feature fusion and nonlinear mapping based on the first weighted features through the first feedforward subnetwork to obtain the first deep features.
[0244] Optionally, the network structure of the first self-attention sub-network can be based on a nonlinear augmented attention mechanism (NEA). Thus, the first self-attention sub-network can dynamically enhance certain attention scores or reduce other irrelevant attention scores by adjusting learnable weights and biases based on the nonlinear augmentation mechanism, enabling the network to optimize a wider range of attention distributions and better adapt to complex data distributions and feature relationships.
[0245] Optionally, the first self-attention subnetwork can adopt a dual-head NEA-based network structure, which is used to achieve feature extraction and weighted fusion.
[0246] Optionally, the network structure of the first feedforward subnetwork can be a feedforward network (FFB) based network structure, used to further perform feature fusion and nonlinear mapping on the weighted features obtained through the first self-attention subnetwork.
[0247] Optionally, the first feedforward subnetwork can also be divided into two branches to extract and weightedly fuse the input features. The input of the first feedforward subnetwork is the output of the first self-attention subnetwork. After the output of the first feedforward subnetwork, the first deep features of the reconstructed block can be obtained.
[0248] In one possible implementation, taking the input of the first self-attention branch network as a normalized brightness reconstruction block I1 as an example, after convolution processing by the first convolutional layer, the first shallow feature M0 can be obtained. Therefore, the feature extraction process through the first feature extraction layer can be represented as M1(I1) = Transformer(M0). Here, M1 is the first deep feature, and the feature dimension of M1 is 32×128×128, i.e.
[0249] The first decoding module may include a first feature fusion layer and a second convolutional layer. After obtaining the first deep features through the first encoding module, the decoder needs to further fuse these features through the decoding module. First, based on the first deep features, the decoder performs feature fusion through the first feature fusion layer to obtain the first fused features. Then, the first fused features are convolved through the second convolutional layer to obtain the first augmentation value of the reconstructed block.
[0250] Similar to the structure of the first feature extraction layer, the first feature fusion layer can also adopt a Bias Block module structure, comprising a second self-attention subnetwork and a second feedforward subnetwork. For the first deep feature output by the first feature extraction layer, the decoder first extracts and fuses features based on the first deep feature through the second self-attention subnetwork to obtain the second weighted feature. Then, based on the second weighted feature, it performs feature fusion and nonlinear mapping through the second feedforward subnetwork to obtain the first fused feature.
[0251] Similar to the first self-attention subnetwork, the second self-attention subnetwork also adopts a dual-headed NEA-based network structure. For a description of the second self-attention subnetwork's structure, please refer to the section on the first self-attention subnetwork described above; it will not be repeated here.
[0252] Similar to the first feedforward subnetwork, the second feedforward subnetwork can also be based on an FFB (Free Forward Bus) network structure. For a description of the second feedforward subnetwork's structure, please refer to the section on the first feedforward subnetwork; it will not be repeated here.
[0253] Optionally, the second convolutional layer is a 3×3 convolutional layer, which is used to perform 3×3 convolution and non-linear mapping operations on the first fused feature to obtain the first enhancement value of the reconstructed block.
[0254] Optionally, the feature fusion process through the first feature fusion layer can be represented as M2(Ι1) = Transformer(M1). Here, M2 is the first fused feature, and the feature dimension of M2 is 32×128×128, i.e. The convolution process using the second convolutional layer can be represented as C1 = σ(W2(M2)), where C1 is the first enhancement value, and the feature dimension of C1 is 1×128×128, i.e.
[0255] 2. Based on at least one of the prediction block and the residual block, and the reconstruction block, a second enhancement value for the reconstruction block is determined through a first cross-attention branch network.
[0256] Optionally, the first cross-attention branch network determines a second enhancement value for the reconstructed block by learning the association features between the reconstructed block and the predicted block, or between the reconstructed block and the residual block, or between the reconstructed block and both the predicted and residual blocks. That is, the input of the first cross-attention branch network includes at least the reconstructed block, and may also include at least one of the predicted and residual blocks; the output is the second enhancement value of the reconstructed block.
[0257] Since the prediction block contains detailed block edge contour information, and the residual block provides information about quantization, using the prediction block and / or residual block as auxiliary inputs enables the first cross-attention branch network to improve the accuracy of the output second enhancement value based on the correlation features between the reconstruction block and the prediction block, or the reconstruction block and the residual block, or the reconstruction block and the prediction block and the residual block.
[0258] In some embodiments, the first cross-attention branch network includes a second encoding module and a second decoding module. The decoder can first extract features from the reconstructed block using the second encoding module based on at least one of the predicted block and the residual block, as well as the reconstructed block, to obtain a second deep feature of the reconstructed block. Then, based on at least one of the predicted block and the residual block, the second decoding module performs feature fusion on the second deep feature to obtain a second augmented value for the reconstructed block. By first extracting features and then fusing them, the first cross-attention branch network can fully learn the correlation features between the reconstructed block and the predicted block, or between the reconstructed block and the residual block, or between the reconstructed block and both the predicted and residual blocks, thereby improving the accuracy of the second augmented value.
[0259] The second encoding module may include a third convolutional layer and a second feature extraction layer. After inputting at least one of the predicted block and the residual block, as well as the reconstructed block, into the first cross-attention branch network, the decoder first performs convolution processing on the reconstructed block through the third convolutional layer to obtain the second shallow features of the reconstructed block, and then performs convolution processing on at least one of the predicted block and the residual block through the third convolutional layer to obtain at least one of the third shallow features of the predicted block and the fourth shallow features of the residual block. Then, based on at least one of the third shallow features and the fourth shallow features, as well as the second shallow features, the second feature extraction layer performs feature extraction to obtain the second deep features.
[0260] Optionally, the third convolutional layer is a 3×3 convolutional layer. Taking the input of the first cross-attention branch network as a reconstructed block, a predicted block, and a residual block as an example, three convolutional layers are needed to perform 3×3 convolution and nonlinear mapping operations on the predicted block, the residual block, and the reconstructed block, respectively. In one possible implementation, taking the input of the first cross-attention branch network as a normalized luminance reconstructed block I1, a luminance predicted block P1, and a luminance residual block R1 as an example, the feature dimensions of I1, P1, and R1 are 1×128×128, that is, I1, P1, The process of convolving I1, P1, and R1 through three third convolutional layers can be represented as N0(I1)=σ(W3(I1)) and E0(I1)=σ(W3(P1)). 12 F0(I1) = σ(W3(R1)), where σ is the ReLU activation function, W3 is a 3×3 convolution kernel, N0 is the second shallow feature, E0 is the third shallow feature, and F0 is the fourth shallow feature. The feature dimensions of N0, E0, and F0 are 32×128×128, i.e., N0, E0,
[0261] Optionally, the second feature extraction layer can adopt a Cross Block module structure, which may include a first cross-attention subnetwork and a third feedforward subnetwork. For at least one of the third shallow features and the fourth shallow features output by the third convolutional layer, and the second shallow feature, in the second feature extraction layer, the decoder performs feature extraction and fusion through the first cross-attention subnetwork based on at least one of the third shallow features and the fourth shallow features, and the second shallow feature, to obtain a third weighted feature. Then, based on the third weighted feature, feature fusion and nonlinear mapping are performed through the third feedforward subnetwork to obtain the second deep feature.
[0262] The network structure of the first cross-attention subnetwork can adopt a network structure based on the cross-attention mechanism (CCA). Thus, by extracting and fusing information from multiple sources, the first cross-attention subnetwork can fully acquire the image features of the current block, thereby achieving attention-weighted reconstructed information.
[0263] The first cross-attention subnetwork can adopt a single-head CCA-based network structure, which is used to extract and fuse features.
[0264] Similar to the first and second feedforward subnetworks, the third feedforward subnetwork can also be based on an FFB-based network structure. For a description of the third feedforward subnetwork's structure, please refer to the section on the first feedforward subnetwork; it will not be repeated here.
[0265] In one possible implementation, taking the input of the first cross-attention branch network as a normalized luminance reconstruction block I1, luminance prediction block P1, and luminance residual block R1, after convolution processing by the third convolutional layer, the second shallow feature N0 corresponding to the reconstruction block I1, the third shallow feature E0 corresponding to the prediction block P1, and the fourth shallow feature F0 corresponding to the residual block R1 can be obtained. Therefore, the feature extraction process through the second feature extraction layer can be represented as N1(I1,P1,R1) = Transformer(N0). Here, N1 is the second deep feature, and the feature dimension of N1 is 32×128×128, i.e.
[0266] The second decoding module may include a second feature fusion layer and a fourth convolutional layer. After obtaining the second deep features through the second encoding module, the decoder needs to further fuse these features through the decoding module. First, based on at least one of the third and fourth shallow features, and the second deep features, feature fusion is performed through the second feature fusion network to obtain the second fused features. Then, the second fused features are convolved through the fourth convolutional layer to obtain the second enhancement value of the reconstructed block.
[0267] Taking the reconstruction block, prediction block and residual block as the input of the quality enhancement network in Figure 6 as an example, as shown in Figure 6, after the prediction block 602 and residual block 603 are processed by the convolution of the third convolutional layer, the resulting third shallow feature and fourth shallow feature need to be input into the second feature extraction layer and the second feature fusion layer.
[0268] Similar to the second feature extraction layer structure, the second feature fusion layer can also adopt a Cross Block module structure. The second feature fusion layer includes a second cross-attention subnetwork and a fourth feedforward subnetwork. For the second deep feature output by the second feature extraction layer, the decoder first extracts and fuses features based on at least one of the third and fourth shallow features, along with the second deep feature, through the second cross-attention subnetwork to obtain a fourth weighted feature. Then, based on the fourth weighted feature, feature fusion and nonlinear mapping are performed through the fourth feedforward subnetwork to obtain the second fused feature.
[0269] Similar to the first cross-attention subnetwork, the second cross-attention subnetwork can also adopt a CCA-based network structure. For a description of the second cross-attention subnetwork's structure, please refer to the section on the first cross-attention subnetwork described above; it will not be repeated here.
[0270] Similar to the first, second, and third feedforward subnetworks, the fourth feedforward subnetwork can also have an FFB-based network structure. For a description of the fourth feedforward subnetwork's structure, please refer to the section on the first feedforward subnetwork; it will not be repeated here.
[0271] Optionally, the fourth convolutional layer is a 3×3 convolutional layer, which is used to perform 3×3 convolution and non-linear mapping operations on the second fused features to obtain the second enhancement value of the reconstructed block.
[0272] Optionally, the feature fusion process through the second feature fusion layer can be represented as N2(Ι1,P1,R1) = Transformer(N1). Here, N2 is the second fused feature, and the feature dimension of N2 is 32×128×128, i.e. The convolution process through the fourth convolutional layer can be represented as C2 = σ(W4(N2)), where C2 is the second enhancement value, and the feature dimension of C2 is 1×128×128, i.e.
[0273] 3. Based on the first enhancement value and the second enhancement value, the quality of the reconstructed block is enhanced.
[0274] After obtaining the first enhancement value through the first self-attention branch network and the second enhancement value through the first cross-attention branch network, the decoder can fuse the first enhancement value, the second enhancement value, and the reconstructed block through the quality enhancement network to obtain the quality-enhanced reconstructed block.
[0275] In one possible implementation, in the quality enhancement network, learnable parameters can be set for the first enhancement value and the second enhancement value as weight coefficients, thereby dynamically adjusting the enhancement ratio of the first enhancement value and the second enhancement value to the reconstructed block, and improving the adaptability of the quality enhancement network to different compression distortion modes.
[0276] As illustrated in Figure 6, the first enhancement value corresponds to the first weighting coefficient α, and the second enhancement value corresponds to the second weighting coefficient β. Taking the input of the quality enhancement network as the normalized luminance reconstruction block I1 as an example, the quality-enhanced reconstruction block 604 can be represented as O1=I1+αC1+βC2.
[0277] In the above embodiments, determining the first enhancement value through the first self-attention branch network in the quality enhancement network can realize the capture of global dependencies within the image using the self-attention mechanism, which is beneficial for the quality enhancement network to understand the overall features of the image more comprehensively. Determining the second enhancement value through the first cross-attention branch network in the quality enhancement network can realize the understanding of the relationship between the reconstruction block, the prediction block, and the residual block using the cross-attention mechanism, which is beneficial for improving the quality enhancement network's ability to fuse cross-domain information.
[0278] The Quality Enhancement Network is a lightweight network for the Teacher Network.
[0279] Considering that using Transformer networks during video encoding and decoding inevitably impacts video encoding and decoding efficiency, this embodiment employs a lightweight quality enhancement network to enhance the quality of reconstructed blocks. To ensure good quality enhancement performance from this lightweight network, it needs to be trained using a corresponding teacher network first.
[0280] In this embodiment of the application, the quality enhancement network is obtained based on a lightweight processing of the teacher network.
[0281] Optionally, the teacher network also adopts a two-branch network architecture that combines self-attention and cross-attention mechanisms, with the teacher network including a second self-attention branch network and a second cross-attention branch network.
[0282] Optionally, in the teacher network, to achieve feature extraction at different levels and scales, the network structure of the second self-attention branch network is a U-net structure. The second self-attention branch network includes a third encoding module and a third decoding module, wherein the third encoding module includes at least two third feature extraction layers and the third decoding module includes at least two third feature fusion layers; the network structure of the second cross-attention branch network is a U-net structure, wherein the second cross-attention branch network includes a fourth encoding module and a fourth decoding module, wherein the fourth encoding module includes at least two fourth feature extraction layers and the fourth decoding module includes at least two fourth feature fusion layers.
[0283] Furthermore, in order to obtain a lightweight quality enhancement network, the first feature extraction layer in the quality enhancement network can be mapped to the first third feature extraction layer among at least two third feature extraction layers, the first feature fusion layer in the quality enhancement network can be mapped to the last third feature fusion layer among at least two third feature fusion layers, the second feature extraction layer in the quality enhancement network can be mapped to the first fourth feature extraction layer among at least two fourth feature extraction layers, and the second feature fusion layer in the quality enhancement network can be mapped to the last second feature fusion layer among at least two fourth feature fusion layers.
[0284] The third feature extraction layer and the third feature fusion layer both adopt a Bias Block module structure, and each Bias Block module includes an NEA network and an FFB network; the fourth feature extraction layer and the fourth feature fusion layer both adopt a Cross Block module structure, and each Cross Block module includes a CCA network and an FFB network.
[0285] The training process of the quality enhancement network.
[0286] To ensure the quality enhancement quality of the quality enhancement network, it needs to be trained before being embedded into the coding framework. Optionally, the training process of the quality enhancement network can be divided into training the teacher network first, and then training the quality enhancement network.
[0287] A two-stage training strategy can be adopted for training the teacher network. The first stage mainly trains the second self-attention branch network, using the self-attention mechanism to extract global contextual features. The second stage, based on fixing the second self-attention branch network, focuses on optimizing the second cross-attention branch network, making the network more focused on the recovery of local details.
[0288] After training the teacher network, the proposed quality enhancement network can be trained. The quality enhancement network can be trained using either a single-stage or two-stage training strategy. Taking a single-stage training strategy for the quality enhancement network as an example, the training set used for the second stage of training of the teacher network, namely the BVI-DVC dataset, can be used.
[0289] Optionally, training the quality enhancement network requires the use of distillation loss between the quality enhancement network and the teacher network, as well as mean squared error loss between the reconstructed block and the original block. In other words, the quality enhancement network is trained based on the distillation loss between the quality enhancement network and the teacher network, and the mean squared error loss between the reconstructed block and the original block.
[0290] The distillation loss includes a first distillation loss, a second distillation loss, a third distillation loss, and a fourth distillation loss. The first distillation loss is determined by combining the deep features of the first sample output from the first feature extraction layer in the quality enhancement network with the deep features of the third sample output from the first of at least two third feature extraction layers. The second distillation loss is determined by combining the fused features of the first sample output from the first feature fusion layer in the quality enhancement network with the fused features of the third sample output from the last of at least two third feature fusion layers. The third distillation loss is determined by combining the deep features of the second sample output from the second feature extraction layer in the quality enhancement network with the deep features of the fourth sample output from the first of at least two fourth feature extraction layers. The fourth distillation loss is determined by combining the fused features of the second sample output from the second feature fusion layer in the quality enhancement network with the fused features of the fourth sample output from the last of at least two fourth feature fusion layers.
[0291] In the above embodiments, after completing the phased training of the teacher network, the quality enhancement network proposed in this application is obtained by lightweighting the teacher network. Simultaneously, by combining distillation loss and mean squared error loss, the quality enhancement network is trained, which can effectively improve the quality enhancement quality. Furthermore, compared to directly embedding the teacher network into the coding framework, embedding the lightweight quality enhancement network into the coding framework can improve the reconstruction quality of the reconstructed blocks while reducing network complexity, minimizing the impact on encoding / decoding efficiency, reducing the consumption of computational and storage resources, and shortening the encoding / decoding time.
[0292] Based on the above embodiments, after embedding the quality enhancement network proposed in this application into the coding framework, the coding performance results shown in Table 1 can be obtained through testing. Optionally, the test sequence is the Class A1, Class A2, Class B, Class C, Class D, Class E, and Class F sequence given in the general test conditions, and the coding performance test results in Table 1 are obtained under the configuration of QP set to 22, 27, 32, 37, 42 and AI mode. Optionally, the image encoding and decoding method proposed in this application can also run in Random Access (RA) and Low Delay (LD) modes. The LD mode can include LD B and LD P, representing that the video frames after the first frame are I-frames or B-frames or P-frames, respectively.
[0293] Table 1. Coding performance of the quality enhancement network under multi-code-rate conditions.
[0294] Where Class represents the video category, Sequence represents the specific test sequence, and Y, Cb, and Cr represent the performance of the luminance and chrominance components. The values in the table are BD-rate, a way to measure algorithm performance, indicating the changes in bitrate and Peak Signal-to-Noise Ratio (PSNR) / Structural Similarity Index (SSIM) of the new coding algorithm compared to the original algorithm. A generally negative value indicates a performance improvement, and a larger absolute value indicates a greater performance improvement. Table 1 shows that this coding algorithm reduces the coding bitrate of the Y component in each test sequence under the same quality (PSNR or SSIM) conditions, demonstrating the effectiveness of network lightweighting operations based on distillation learning. By adding quality enhancement to the hybrid coding framework, the reconstruction quality of the current block can be improved, providing a more accurate reference for intra-frame prediction of subsequent coding blocks.
[0295] Furthermore, this scheme employs a lightweight quality enhancement network to enhance the quality of the reconstructed blocks, which can improve the reconstruction quality of the blocks while reducing network complexity, minimizing the impact on encoding and decoding efficiency, reducing the consumption of computing and storage resources, and shortening the encoding and decoding time.
[0296] Please refer to Figure 18, which shows a structural block diagram of a decoding apparatus provided in an exemplary embodiment of this application. The decoding apparatus includes:
[0297] Decoding unit 1801 is configured to perform quality enhancement on the reconstructed block through a quality enhancement network based on at least one of the predicted block and residual block of the current block, and the reconstructed block of the current block.
[0298] Optionally, the decoding unit 1801 is used for:
[0299] Analyze the bitstream and determine the enable flag;
[0300] When the value of the enable flag indicates that the reconstructed block needs quality enhancement, the reconstructed block is quality enhanced through the quality enhancement network based on at least one of the predicted block and the residual block of the current block, and the reconstructed block of the current block.
[0301] Optionally, the value of the enable flag is used to indicate whether the reconstruction block of the current block needs quality enhancement, or to indicate whether the luminance reconstruction block of the current block needs quality enhancement, or to indicate whether the chroma reconstruction block of the current block needs quality enhancement, or to indicate whether both the luminance reconstruction block and the chroma reconstruction block of the current block need quality enhancement.
[0302] Optionally, the decoding unit 1801 is used for:
[0303] When the value of the enable flag indicates that the luminance reconstruction block needs quality enhancement, the luminance reconstruction block is quality enhanced through a luminance quality enhancement network based on at least one of the luminance prediction block and luminance residual block of the current block, and the luminance reconstruction block of the current block.
[0304] Optionally, the decoding unit 1801 is used for:
[0305] When the value of the enable flag indicates that the chroma reconstruction block needs quality enhancement, the chroma reconstruction block is quality enhanced through a chroma quality enhancement network based on at least one of the chroma prediction block and the chroma residual block of the current block, and the chroma reconstruction block of the current block.
[0306] Optionally, the decoding unit 1801 is used for:
[0307] When the value of the enable flag indicates that the chroma reconstruction block needs quality enhancement, the chroma reconstruction block is quality enhanced through a first chroma quality enhancement network based on at least one of the chroma prediction block, chroma residual block, and luminance reconstruction block of the current block, and the chroma reconstruction block of the current block; and / or,
[0308] When the value of the enable flag indicates that the chroma reconstruction block needs quality enhancement, the chroma reconstruction block is quality enhanced through a second chroma quality enhancement network based on at least one of the chroma prediction block, chroma residual block, and quality-enhanced luminance reconstruction block of the current block, as well as the chroma reconstruction block of the current block.
[0309] Optionally, different blocks in the same image may correspond to their respective enable flags, or different blocks within the same slice in the same image may correspond to a unified enable flag, or different blocks in the same image may correspond to a unified enable flag, or different blocks in different images within the same image group may correspond to a unified enable flag.
[0310] Optionally, the decoding unit 1801 is used for:
[0311] Based on at least one of the predicted block, the residual block, and the adjacent reconstructed blocks of the reconstructed block of the current block, and the reconstructed block of the current block, the reconstructed block is quality-enhanced by the quality enhancement network.
[0312] Optionally, the decoding unit 1801 is configured to perform quality enhancement on the reconstructed block via a quality enhancement network before loop filtering, based on at least one of the predicted block and residual block of the current block, and the reconstructed block of the current block.
[0313] Optionally, the current block is a CTU block.
[0314] Optionally, the decoding unit 1801 is used for:
[0315] Based on the reconstructed block, a first enhancement value of the reconstructed block is determined through the first self-attention branch network;
[0316] Based on at least one of the prediction block and the residual block, and the reconstruction block, a second enhancement value of the reconstruction block is determined through the first cross-attention branch network;
[0317] The quality of the reconstructed block is enhanced based on the first enhancement value and the second enhancement value.
[0318] Optionally, the first self-attention branch network includes a first encoding module and a first decoding module; the decoding unit 1801 is used for:
[0319] The first deep features of the reconstructed block are obtained by extracting features from the reconstructed block using the first encoding module.
[0320] The first deep feature is fused by the first decoding module to obtain the first enhancement value of the reconstructed block.
[0321] Optionally, the first encoding module includes a first convolutional layer and a first feature extraction layer, and the decoding unit 1801 is used for:
[0322] The reconstructed block is convolved by the first convolutional layer to obtain the first shallow layer features;
[0323] Based on the first shallow features, feature extraction is performed through the first feature extraction layer to obtain the first deep features.
[0324] Optionally, the first feature extraction layer includes a first self-attention subnetwork and a first feedforward subnetwork, and the decoding unit 1801 is used for:
[0325] Based on the first shallow features, feature extraction and fusion are performed through the first self-attention subnetwork to obtain the first weighted features;
[0326] Based on the first weighted feature, the first deep feature is obtained by feature fusion and nonlinear mapping through the first feedforward sub-network.
[0327] Optionally, the first decoding module includes a first feature fusion layer and a second convolutional layer, and the decoding unit 1801 is used for:
[0328] Based on the first deep feature, feature fusion is performed through the first feature fusion layer to obtain the first fused feature;
[0329] The first fusion feature is convolved by the second convolutional layer to obtain the first enhancement value of the reconstructed block.
[0330] Optionally, the first feature fusion layer includes a second self-attention subnetwork and a second feedforward subnetwork, and the decoding unit 1801 is used for:
[0331] Based on the first deep feature, the second weighted feature is obtained by feature extraction and fusion through the second self-attention subnetwork;
[0332] Based on the second weighted feature, the first fused feature is obtained by feature fusion and nonlinear mapping through the second feedforward sub-network.
[0333] Optionally, the first cross-attention branch network includes a second encoding module and a second decoding module; the decoding unit 1801 is used for:
[0334] Based on at least one of the prediction block and the residual block, and the reconstruction block, the second encoding module extracts features from the reconstruction block to obtain the second deep features of the reconstruction block;
[0335] Based on at least one of the prediction block and the residual block, the second deep feature is fused by the second decoding module to obtain the second enhancement value of the reconstruction block.
[0336] Optionally, the second encoding module includes a third convolutional layer and a second feature extraction layer, and the decoding unit 1801 is used for:
[0337] The reconstruction block is convolved by the third convolutional layer to obtain the second shallow feature of the reconstruction block. At least one of the prediction block and the residual block is convolved by the third convolutional layer to obtain at least one of the third shallow feature of the prediction block and the fourth shallow feature of the residual block.
[0338] Based on at least one of the third and fourth shallow features, and the second shallow feature, feature extraction is performed through the second feature extraction layer to obtain the second deep feature.
[0339] Optionally, the second feature extraction layer includes a first cross-attention subnetwork and a third feedforward subnetwork, and the decoding unit 1801 is used for:
[0340] Based on at least one of the third shallow features and the fourth shallow features, as well as the second shallow features, feature extraction and fusion are performed through the first cross-attention subnetwork to obtain the third weighted feature;
[0341] Based on the third weighted feature, the second deep feature is obtained by feature fusion and nonlinear mapping through the third feedforward sub-network.
[0342] Optionally, the second decoding module includes a second feature fusion layer and a fourth convolutional layer, and the decoding unit 1801 is used for:
[0343] Based on at least one of the third shallow features and the fourth shallow features, and the second deep features, feature fusion is performed through the second feature fusion network to obtain the second fused feature;
[0344] The second enhancement value of the reconstructed block is obtained by performing convolution processing on the second fused feature through the fourth convolutional layer.
[0345] Optionally, the second feature fusion layer includes a second cross-attention subnetwork and a fourth feedforward subnetwork, and the decoding unit 1801 is used for:
[0346] Based on at least one of the third and fourth shallow features, and the second deep feature, feature extraction and fusion are performed through the second cross-attention sub-network to obtain the fourth weighted feature;
[0347] Based on the fourth weighted feature, feature fusion and nonlinear mapping are performed through the fourth feedforward sub-network to obtain the second fused feature.
[0348] Optionally, the quality enhancement network is obtained based on a lightweight processing of the teacher network;
[0349] The teacher network includes a second self-attention branch network and a second cross-attention branch network; the network structure of the second self-attention branch network is a U-net structure, and the second self-attention branch network includes at least two third feature extraction layers and at least two third feature fusion layers; the network structure of the second cross-attention branch network is a U-net structure, and the second cross-attention branch network includes at least two fourth feature extraction layers and at least two fourth feature fusion layers.
[0350] The first feature extraction layer in the quality enhancement network corresponds to the first third feature extraction layer among the at least two third feature extraction layers; the first feature fusion layer in the quality enhancement network corresponds to the last third feature fusion layer among the at least two third feature fusion layers.
[0351] The second feature extraction layer in the quality enhancement network corresponds to the first fourth feature extraction layer among the at least two fourth feature extraction layers; the second feature fusion layer in the quality enhancement network corresponds to the last second feature fusion layer among the at least two fourth feature fusion layers.
[0352] Optionally, the quality enhancement network is trained based on the distillation loss between the quality enhancement network and the teacher network, and the mean squared error loss between the reconstructed block and the original block after quality enhancement.
[0353] Optionally, the distillation loss includes a first distillation loss, a second distillation loss, a third distillation loss, and a fourth distillation loss;
[0354] The first distillation loss is determined based on the first deep feature of the sample output by the first feature extraction layer in the quality enhancement network and the third deep feature of the sample output by the first third feature extraction layer among the at least two third feature extraction layers;
[0355] The second distillation loss is determined based on the first sample fusion feature output by the first feature fusion layer in the quality enhancement network and the third sample fusion feature output by the last of the at least two third feature fusion layers;
[0356] The third distillation loss is determined based on the second sample deep features output by the second feature extraction layer in the quality enhancement network and the fourth sample deep features output by the first of the at least two fourth feature extraction layers.
[0357] The fourth distillation loss is determined based on the second sample fusion feature output by the second feature fusion layer in the quality enhancement network and the fourth sample fusion feature output by the last of the at least two fourth feature fusion layers.
[0358] Please refer to Figure 19, which shows a structural block diagram of an encoding device provided in an exemplary embodiment of this application. The encoding device includes:
[0359] Encoding unit 1901 is configured to enhance the quality of the reconstructed block using a quality enhancement network based on at least one of the predicted block and residual block of the current block, and the reconstructed block of the current block.
[0360] Optionally, coding unit 1901 is used for:
[0361] A first rate-distortion cost and a second rate-distortion cost are determined. The first rate-distortion cost is the rate-distortion cost of not performing quality enhancement on the reconstructed block. The second rate-distortion cost is the rate-distortion cost of performing quality enhancement on the reconstructed block through a quality enhancement network based on at least one of the predicted block and the residual block, and the reconstructed block of the current block.
[0362] Based on the comparison result of the first rate-distortion cost and the second rate-distortion cost, the value of the enable flag is encoded into the bitstream, and the value of the enable flag is used to indicate whether the reconstructed block needs to be enhanced in quality.
[0363] Optionally, the value of the enable flag is used to indicate whether the reconstruction block of the current block needs quality enhancement, or to indicate whether the luminance reconstruction block of the current block needs quality enhancement, or to indicate whether the chroma reconstruction block of the current block needs quality enhancement, or to indicate whether both the luminance reconstruction block and the chroma reconstruction block of the current block need quality enhancement.
[0364] Optionally, coding unit 1901 is used for:
[0365] A third rate distortion cost is determined, which is the rate distortion cost of enhancing the brightness reconstruction block through a brightness quality enhancement network based on at least one of the brightness prediction block and brightness residual block of the current block, and the brightness reconstruction block of the current block.
[0366] Based on the comparison result of the first rate distortion cost and the third rate distortion cost, the value of the luminance enable flag is encoded into the bitstream.
[0367] Optionally, coding unit 1901 is used for:
[0368] A fourth rate distortion cost is determined, which is the rate distortion cost of enhancing the chroma reconstruction block through a chroma quality enhancement network based on at least one of the chroma prediction block and the chroma residual block of the current block, and the chroma reconstruction block of the current block.
[0369] Based on the comparison result of the first rate-distortion cost and the fourth rate-distortion cost, the value of the chroma enable flag is encoded into the bitstream.
[0370] Optionally, coding unit 1901 is used for:
[0371] A fifth rate distortion cost is determined, wherein the fifth rate distortion cost is a rate distortion cost based on at least one of the chroma prediction block, chroma residual block, and luminance reconstruction block of the current block, and the chroma reconstruction block of the current block, wherein the chroma reconstruction block is quality-enhanced through a first chroma quality enhancement network; and / or, a rate distortion cost based on at least one of the chroma prediction block, chroma residual block, and quality-enhanced luminance reconstruction block of the current block, and the chroma reconstruction block of the current block, wherein the chroma reconstruction block is quality-enhanced through a second chroma quality enhancement network.
[0372] Based on the comparison result of the first rate distortion cost and the fifth rate distortion cost, the value of the chroma enable flag is encoded into the bitstream.
[0373] Optionally, different blocks in the same image may correspond to their respective enable flags, or different blocks within the same slice in the same image may correspond to a unified enable flag, or different blocks in the same image may correspond to a unified enable flag, or different blocks in different images within the same image group may correspond to a unified enable flag.
[0374] Optionally, coding unit 1901 is used for:
[0375] Based on at least one of the predicted block, the residual block, and the adjacent reconstructed blocks of the reconstructed block of the current block, and the reconstructed block of the current block, the reconstructed block is quality-enhanced by the quality enhancement network.
[0376] Optionally, the encoding unit 1901 is configured to perform quality enhancement on the reconstructed block via a quality enhancement network based on at least one of the prediction block and residual block of the current block, and the reconstructed block of the current block, prior to loop filtering.
[0377] Optionally, the current block is a CTU block.
[0378] Optionally, the quality enhancement network includes a first self-attention branch network and a first cross-attention branch network, and the encoding unit 1901 is used for:
[0379] Based on the reconstructed block, a first enhancement value of the reconstructed block is determined through the first self-attention branch network;
[0380] Based on at least one of the prediction block and the residual block, and the reconstruction block, a second enhancement value of the reconstruction block is determined through the first cross-attention branch network;
[0381] The quality of the reconstructed block is enhanced based on the first enhancement value and the second enhancement value.
[0382] Optionally, the first self-attention branch network includes a first encoding module and a first decoding module; the encoding unit 1901 is used for:
[0383] The first deep features of the reconstructed block are obtained by extracting features from the reconstructed block using the first encoding module.
[0384] The first deep feature is fused by the first decoding module to obtain the first enhancement value of the reconstructed block.
[0385] Optionally, the first encoding module includes a first convolutional layer and a first feature extraction layer, and encoding unit 1901 is used for:
[0386] The reconstructed block is convolved by the first convolutional layer to obtain the first shallow layer features;
[0387] Based on the first shallow features, feature extraction is performed through the first feature extraction layer to obtain the first deep features.
[0388] Optionally, the first feature extraction layer includes a first self-attention subnetwork and a first feedforward subnetwork, with encoding unit 1901 used for:
[0389] Based on the first shallow features, feature extraction and fusion are performed through the first self-attention subnetwork to obtain the first weighted features;
[0390] Based on the first weighted feature, the first deep feature is obtained by feature fusion and nonlinear mapping through the first feedforward sub-network.
[0391] Optionally, the first decoding module includes a first feature fusion layer and a second convolutional layer, and the encoding unit 1901 is used for:
[0392] Based on the first deep feature, feature fusion is performed through the first feature fusion layer to obtain the first fused feature;
[0393] The first fusion feature is convolved by the second convolutional layer to obtain the first enhancement value of the reconstructed block.
[0394] Optionally, the first feature fusion layer includes a second self-attention subnetwork and a second feedforward subnetwork, with encoding unit 1901 used for:
[0395] Based on the first deep feature, the second weighted feature is obtained by feature extraction and fusion through the second self-attention subnetwork;
[0396] Based on the second weighted feature, the first fused feature is obtained by feature fusion and nonlinear mapping through the second feedforward sub-network.
[0397] Optionally, the first cross-attention branch network includes a second encoding module and a second decoding module; the encoding unit 1901 is used for:
[0398] Based on at least one of the prediction block and the residual block, and the reconstruction block, the second encoding module extracts features from the reconstruction block to obtain the second deep features of the reconstruction block;
[0399] Based on at least one of the prediction block and the residual block, the second deep feature is fused by the second decoding module to obtain the second enhancement value of the reconstruction block.
[0400] Optionally, the second encoding module includes a third convolutional layer and a second feature extraction layer, and encoding unit 1901 is used for:
[0401] The reconstruction block is convolved by the third convolutional layer to obtain the second shallow feature of the reconstruction block. At least one of the prediction block and the residual block is convolved by the third convolutional layer to obtain at least one of the third shallow feature of the prediction block and the fourth shallow feature of the residual block.
[0402] Based on at least one of the third and fourth shallow features, and the second shallow feature, feature extraction is performed through the second feature extraction layer to obtain the second deep feature.
[0403] Optionally, the second feature extraction layer includes a first cross-attention subnetwork and a third feedforward subnetwork, with encoding unit 1901 used for:
[0404] Based on at least one of the third shallow features and the fourth shallow features, as well as the second shallow features, feature extraction and fusion are performed through the first cross-attention subnetwork to obtain the third weighted feature;
[0405] Based on the third weighted feature, the second deep feature is obtained by feature fusion and nonlinear mapping through the third feedforward sub-network.
[0406] Optionally, the second decoding module includes a second feature fusion layer and a fourth convolutional layer, with encoding unit 1901 used for:
[0407] Based on at least one of the third shallow features and the fourth shallow features, and the second deep features, feature fusion is performed through the second feature fusion network to obtain the second fused feature;
[0408] The second enhancement value of the reconstructed block is obtained by performing convolution processing on the second fused feature through the fourth convolutional layer.
[0409] Optionally, the second feature fusion layer includes a second cross-attention subnetwork and a fourth feedforward subnetwork, with encoding unit 1901 used for:
[0410] Based on at least one of the third and fourth shallow features, and the second deep feature, feature extraction and fusion are performed through the second cross-attention sub-network to obtain the fourth weighted feature;
[0411] Based on the fourth weighted feature, feature fusion and nonlinear mapping are performed through the fourth feedforward sub-network to obtain the second fused feature.
[0412] Optionally, the quality enhancement network is obtained based on a lightweight processing of the teacher network;
[0413] The teacher network includes a second self-attention branch network and a second cross-attention branch network; the network structure of the second self-attention branch network is a U-net structure, and the second self-attention branch network includes at least two third feature extraction layers and at least two third feature fusion layers; the network structure of the second cross-attention branch network is a U-net structure, and the second cross-attention branch network includes at least two fourth feature extraction layers and at least two fourth feature fusion layers.
[0414] The first feature extraction layer in the quality enhancement network corresponds to the first third feature extraction layer among the at least two third feature extraction layers; the first feature fusion layer in the quality enhancement network corresponds to the last third feature fusion layer among the at least two third feature fusion layers.
[0415] The second feature extraction layer in the quality enhancement network corresponds to the first fourth feature extraction layer among the at least two fourth feature extraction layers; the second feature fusion layer in the quality enhancement network corresponds to the last second feature fusion layer among the at least two fourth feature fusion layers.
[0416] Optionally, the quality enhancement network is trained based on the distillation loss between the quality enhancement network and the teacher network, and the mean squared error loss between the reconstructed block and the original block after quality enhancement.
[0417] Optionally, the distillation loss includes a first distillation loss, a second distillation loss, a third distillation loss, and a fourth distillation loss;
[0418] The first distillation loss is determined based on the first deep feature of the sample output by the first feature extraction layer in the quality enhancement network and the third deep feature of the sample output by the first third feature extraction layer among the at least two third feature extraction layers;
[0419] The second distillation loss is determined based on the first sample fusion feature output by the first feature fusion layer in the quality enhancement network and the third sample fusion feature output by the last of the at least two third feature fusion layers;
[0420] The third distillation loss is determined based on the second sample deep features output by the second feature extraction layer in the quality enhancement network and the fourth sample deep features output by the first of the at least two fourth feature extraction layers.
[0421] The fourth distillation loss is determined based on the second sample fusion feature output by the second feature fusion layer in the quality enhancement network and the fourth sample fusion feature output by the last of the at least two fourth feature fusion layers.
[0422] It should be noted that the image decoding process of the above-mentioned decoding unit can refer to the above-mentioned image decoding method embodiment, and the image encoding process of the above-mentioned encoding unit can refer to the above-mentioned image encoding method embodiment. This embodiment will not be described in detail here.
[0423] Please refer to Figure 20, which shows a structural block diagram of a decoder provided in an exemplary embodiment of this application. The decoder may include one or more components such as a processor 2001 and a memory 2002. The components are coupled together via a bus system. It is understood that the bus system is used to implement communication between these components. In addition to a data bus, the bus system also includes a power bus, a control bus, and a status signal bus.
[0424] Memory 2002 is used to store computer programs that can run on processor 2001; processor 2001 is used to execute, when running the computer program:
[0425] Based on at least one of the predicted block and residual block of the current block, and the reconstructed block of the current block, the reconstructed block is quality-enhanced by a quality enhancement network.
[0426] It is understood that the memory 2002 in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory described in this application 2002 is intended to include, but is not limited to, these and any other suitable types of memory.
[0427] The processor 2001 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the aforementioned image decoding method can be completed by the integrated logic circuitry in the hardware of the processor 2001 or by instructions in software form. The processor 2001 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 2002. Processor 2001 reads information from memory 2002 and, in conjunction with its hardware, completes the steps of the above method. It is understood that the embodiments described in this application can be implemented using hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more ASICs, DSPs, digital signal processing devices (DSP Devices, DSPDs), programmable logic devices (PLDs), FPGAs, general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in this application, or combinations thereof. For software implementation, the technology described in this application can be implemented through modules (e.g., procedures, functions, etc.) that perform the functions described in this application. Software code can be stored in memory and executed by the processor. The memory can be implemented in the processor or external to the processor. Optionally, as another embodiment, processor 2001 is also configured to execute the image decoding method described in any of the foregoing embodiments when running the computer program.
[0428] Please refer to Figure 21, which shows a structural block diagram of an encoder provided in an exemplary embodiment of this application. The encoder may include one or more components such as a processor 2101 and a memory 2102. The components are coupled together via a bus system. It is understood that the bus system is used to enable communication between these components. In addition to a data bus, the bus system also includes a power bus, a control bus, and a status signal bus.
[0429] Memory 2102 is used to store computer programs that can run on processor 2101; processor 2101 is used to execute, when running the computer program:
[0430] Based on at least one of the predicted block and residual block of the current block, and the reconstructed block of the current block, the reconstructed block is quality-enhanced by a quality enhancement network.
[0431] It is understood that the memory 2102 in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 2102 described herein is intended to include, but is not limited to, these and any other suitable types of memory.
[0432] The processor 2101 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above image encoding method can be completed by the integrated logic circuitry in the hardware of the processor 2101 or by software instructions. The processor 2101 can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 2102. Processor 2101 reads information from memory 2102 and, in conjunction with its hardware, completes the steps of the above method. It is understood that the embodiments described in this application can be implemented using hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more ASICs, DSPs, digital signal processing devices (DSP Devices, DSPDs), programmable logic devices (PLDs), FPGAs, general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in this application, or combinations thereof. For software implementation, the technology described in this application can be implemented through modules (e.g., procedures, functions, etc.) that perform the functions described in this application. Software code can be stored in memory and executed by the processor. The memory can be implemented in the processor or external to the processor. Optionally, as another embodiment, processor 2101 is also configured to execute the image encoding method described in any of the foregoing embodiments when running the computer program.
[0433] This application also provides a non-volatile computer-readable storage medium for storing a bitstream, the bitstream being generated by an image encoding method using an encoder, or the bitstream being decoded by an image decoding method using a decoder, wherein the image encoding method is the image encoding method described in the above embodiments, and the image decoding method is the image decoding method described in the above embodiments.
[0434] This application also provides a computer program product, which includes computer instructions stored in a computer-readable storage medium. A processor retrieves the computer instructions from the computer-readable storage medium and executes the computer instructions to implement the image encoding method or the image decoding method as described in the above embodiments.
[0435] Those skilled in the art will recognize that the functions described in the embodiments of this application in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of a computer program from one place to another. Storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer.
[0436] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.
Claims
1. An image decoding method, applied to a decoder, characterized in that, The method includes: Based on at least one of the predicted block and residual block of the current block, and the reconstructed block of the current block, the reconstructed block is quality-enhanced by a quality enhancement network.
2. The method according to claim 1, characterized in that, The process of enhancing the quality of the reconstructed block using a quality enhancement network, based on at least one of the predicted block and residual block of the current block, and the reconstructed block of the current block, includes: Analyze the bitstream and determine the enable flag; When the value of the enable flag indicates that the reconstructed block needs quality enhancement, the reconstructed block is quality enhanced through the quality enhancement network based on at least one of the predicted block and the residual block of the current block, and the reconstructed block of the current block.
3. The method according to claim 2, characterized in that, The value of the enable flag is used to indicate whether the reconstruction block of the current block needs quality enhancement, or whether the luminance reconstruction block of the current block needs quality enhancement, or whether the chroma reconstruction block of the current block needs quality enhancement, or whether both the luminance reconstruction block and the chroma reconstruction block of the current block need quality enhancement.
4. The method according to claim 3, characterized in that, When the value of the enable flag indicates that the reconstructed block needs quality enhancement, the quality enhancement of the reconstructed block is performed through the quality enhancement network based on at least one of the predicted block and the residual block of the current block, and the reconstructed block of the current block, including: When the value of the enable flag indicates that the luminance reconstruction block needs quality enhancement, the luminance reconstruction block is quality enhanced through a luminance quality enhancement network based on at least one of the luminance prediction block and luminance residual block of the current block, and the luminance reconstruction block of the current block.
5. The method according to claim 3, characterized in that, When the value of the enable flag indicates that the reconstructed block needs quality enhancement, the quality enhancement of the reconstructed block is performed through the quality enhancement network based on at least one of the predicted block and the residual block of the current block, and the reconstructed block of the current block, including: When the value of the enable flag indicates that the chroma reconstruction block needs quality enhancement, the chroma reconstruction block is quality enhanced through a chroma quality enhancement network based on at least one of the chroma prediction block and the chroma residual block of the current block, and the chroma reconstruction block of the current block.
6. The method according to claim 3, characterized in that, When the value of the enable flag indicates that the reconstructed block needs quality enhancement, the quality enhancement of the reconstructed block is performed through the quality enhancement network based on at least one of the predicted block and the residual block of the current block, and the reconstructed block of the current block, including: When the value of the enable flag indicates that the chroma reconstruction block needs quality enhancement, the chroma reconstruction block is quality enhanced through a first chroma quality enhancement network based on at least one of the chroma prediction block, chroma residual block, and luminance reconstruction block of the current block, and the chroma reconstruction block of the current block; and / or, When the value of the chroma enable flag indicates that the chroma reconstruction block needs quality enhancement, the chroma reconstruction block is quality enhanced through a second chroma quality enhancement network based on at least one of the chroma prediction block, chroma residual block, and quality-enhanced luminance reconstruction block of the current block, as well as the chroma reconstruction block of the current block.
7. The method according to any one of claims 2 to 6, characterized in that, Different blocks in the same image correspond to their respective enable flags, or different blocks within the same slice in the same image correspond to a unified enable flag, or different blocks in the same image correspond to a unified enable flag, or different blocks in different images in the same image group correspond to a unified enable flag.
8. The method according to any one of claims 1 to 7, characterized in that, The process of enhancing the quality of the reconstructed block using a quality enhancement network, based on at least one of the predicted block and residual block of the current block, and the reconstructed block of the current block, includes: Based on at least one of the predicted block, the residual block, and the adjacent reconstructed blocks of the reconstructed block of the current block, and the reconstructed block of the current block, the reconstructed block is quality-enhanced by the quality enhancement network.
9. The method according to any one of claims 1 to 8, characterized in that, The step of enhancing the quality of the reconstructed block using a quality enhancement network based on at least one of the predicted block and residual block of the current block, and the reconstructed block of the current block, is performed before loop filtering.
10. The method according to any one of claims 1 to 9, characterized in that, The current block is a CTU block.
11. The method according to any one of claims 1 to 10, characterized in that, The quality enhancement network includes a first self-attention branch network and a first cross-attention branch network. The quality enhancement of the reconstructed block based on at least one of the predicted block and residual block of the current block, and the reconstructed block of the current block, through the quality enhancement network includes: Based on the reconstructed block, a first enhancement value of the reconstructed block is determined through the first self-attention branch network; Based on at least one of the prediction block and the residual block, and the reconstruction block, a second enhancement value of the reconstruction block is determined through the first cross-attention branch network; The quality of the reconstructed block is enhanced based on the first enhancement value and the second enhancement value.
12. The method according to claim 11, characterized in that, The first self-attention branch network includes a first encoding module and a first decoding module; the step of determining a first enhancement value of the reconstructed block through the first self-attention branch network based on the reconstructed block includes: The first deep features of the reconstructed block are obtained by extracting features from the reconstructed block using the first encoding module. The first deep feature is fused by the first decoding module to obtain the first enhancement value of the reconstructed block.
13. The method according to claim 12, characterized in that, The first encoding module includes a first convolutional layer and a first feature extraction layer. The step of extracting features from the reconstructed block using the first encoding module to obtain the first deep features of the reconstructed block includes: The reconstructed block is convolved by the first convolutional layer to obtain the first shallow layer features; Based on the first shallow features, feature extraction is performed through the first feature extraction layer to obtain the first deep features.
14. The method according to claim 13, characterized in that, The first feature extraction layer includes a first self-attention subnetwork and a first feedforward subnetwork. The step of extracting features from the first shallow features using the first feature extraction layer to obtain the first deep features includes: Based on the first shallow features, feature extraction and fusion are performed through the first self-attention subnetwork to obtain the first weighted features; Based on the first weighted feature, the first deep feature is obtained by feature fusion and nonlinear mapping through the first feedforward sub-network.
15. The method according to claim 12, characterized in that, The first decoding module includes a first feature fusion layer and a second convolutional layer. The step of fusing the first deep features through the first decoding module to obtain the first enhancement value of the reconstructed block includes: Based on the first deep feature, feature fusion is performed through the first feature fusion layer to obtain the first fused feature; The first fusion feature is convolved by the second convolutional layer to obtain the first enhancement value of the reconstructed block.
16. The method according to claim 15, characterized in that, The first feature fusion layer includes a second self-attention subnetwork and a second feedforward subnetwork. The step of fusing features based on the first deep features through the first feature fusion layer to obtain the first fused feature includes: Based on the first deep feature, the second weighted feature is obtained by feature extraction and fusion through the second self-attention subnetwork; Based on the second weighted feature, the first fused feature is obtained by feature fusion and nonlinear mapping through the second feedforward sub-network.
17. The method according to any one of claims 11 to 16, characterized in that, The first cross-attention branch network includes a second encoding module and a second decoding module; the step of determining a second enhancement value of the reconstructed block based on at least one of the predicted block and the residual block, and the reconstructed block, through the first cross-attention branch network, includes: Based on at least one of the prediction block and the residual block, and the reconstruction block, the second encoding module performs feature extraction on the reconstruction block to obtain the second deep feature of the reconstruction block; Based on at least one of the prediction block and the residual block, the second deep feature is fused by the second decoding module to obtain the second enhancement value of the reconstruction block.
18. The method according to claim 17, characterized in that, The second encoding module includes a third convolutional layer and a second feature extraction layer. Based on at least one of the predicted block and the residual block, and the reconstructed block, the second encoding module extracts features from the reconstructed block to obtain a second deep feature of the reconstructed block, including: The reconstruction block is convolved by the third convolutional layer to obtain the second shallow feature of the reconstruction block. At least one of the prediction block and the residual block is convolved by the third convolutional layer to obtain at least one of the third shallow feature of the prediction block and the fourth shallow feature of the residual block. Based on at least one of the third and fourth shallow features, and the second shallow feature, feature extraction is performed through the second feature extraction layer to obtain the second deep feature.
19. The method according to claim 18, characterized in that, The second feature extraction layer includes a first cross-attention subnetwork and a third feedforward subnetwork. The second deep feature is obtained by extracting features based on at least one of the third shallow features and the fourth shallow features, as well as the second shallow features. This includes: Based on at least one of the third shallow features and the fourth shallow features, as well as the second shallow features, feature extraction and fusion are performed through the first cross-attention subnetwork to obtain the third weighted feature; Based on the third weighted feature, the second deep feature is obtained by feature fusion and nonlinear mapping through the third feedforward sub-network.
20. The method according to claim 18, characterized in that, The second decoding module includes a second feature fusion layer and a fourth convolutional layer. The step of fusing features of the second deep features based on at least one of the predicted block and the residual block to obtain the second enhancement value of the reconstructed block includes: Based on at least one of the third shallow features and the fourth shallow features, and the second deep features, feature fusion is performed through the second feature fusion network to obtain the second fused feature; The second enhancement value of the reconstructed block is obtained by performing convolution processing on the second fused feature through the fourth convolutional layer.
21. The method according to claim 20, characterized in that, The second feature fusion layer includes a second cross-attention subnetwork and a fourth feedforward subnetwork. The second feature fusion layer performs feature fusion based on at least one of the third and fourth shallow features, and the second deep feature, to obtain a second fused feature, including: Based on at least one of the third and fourth shallow features, and the second deep feature, feature extraction and fusion are performed through the second cross-attention sub-network to obtain the fourth weighted feature; Based on the fourth weighted feature, feature fusion and nonlinear mapping are performed through the fourth feedforward sub-network to obtain the second fused feature.
22. The method according to any one of claims 11 to 21, characterized in that, The quality enhancement network is based on a lightweight processing of the teacher network; The teacher network includes a second self-attention branch network and a second cross-attention branch network; the network structure of the second self-attention branch network is a U-net structure, and the second self-attention branch network includes at least two third feature extraction layers and at least two third feature fusion layers; the network structure of the second cross-attention branch network is a U-net structure, and the second cross-attention branch network includes at least two fourth feature extraction layers and at least two fourth feature fusion layers. The first feature extraction layer in the quality enhancement network corresponds to the first third feature extraction layer among the at least two third feature extraction layers; the first feature fusion layer in the quality enhancement network corresponds to the last third feature fusion layer among the at least two third feature fusion layers. The second feature extraction layer in the quality enhancement network corresponds to the first fourth feature extraction layer among the at least two fourth feature extraction layers; the second feature fusion layer in the quality enhancement network corresponds to the last second feature fusion layer among the at least two fourth feature fusion layers.
23. The method according to claim 22, characterized in that, The quality enhancement network is trained based on the distillation loss between the quality enhancement network and the teacher network, and the mean square error loss between the reconstructed block and the original block after quality enhancement.
24. The method according to claim 23, characterized in that, The distillation losses include a first distillation loss, a second distillation loss, a third distillation loss, and a fourth distillation loss; The first distillation loss is determined based on the first deep feature of the sample output by the first feature extraction layer in the quality enhancement network and the third deep feature of the sample output by the first third feature extraction layer among the at least two third feature extraction layers; The second distillation loss is determined based on the first sample fusion feature output by the first feature fusion layer in the quality enhancement network and the third sample fusion feature output by the last of the at least two third feature fusion layers; The third distillation loss is determined based on the second sample deep features output by the second feature extraction layer in the quality enhancement network and the fourth sample deep features output by the first of the at least two fourth feature extraction layers. The fourth distillation loss is determined based on the second sample fusion feature output by the second feature fusion layer in the quality enhancement network and the fourth sample fusion feature output by the last of the at least two fourth feature fusion layers.
25. An image encoding method applied to an encoder, characterized in that, The method includes: Based on at least one of the predicted block and residual block of the current block, and the reconstructed block of the current block, the reconstructed block is quality-enhanced by a quality enhancement network.
26. The method according to claim 25, characterized in that, The method further includes: A first rate-distortion cost and a second rate-distortion cost are determined. The first rate-distortion cost is the rate-distortion cost of not performing quality enhancement on the reconstructed block. The second rate-distortion cost is the rate-distortion cost of performing quality enhancement on the reconstructed block through a quality enhancement network based on at least one of the predicted block and the residual block, and the reconstructed block of the current block. Based on the comparison result of the first rate-distortion cost and the second rate-distortion cost, the value of the enable flag is encoded into the bitstream, and the value of the enable flag is used to indicate whether the reconstructed block needs to be enhanced in quality.
27. The method according to claim 26, characterized in that, The value of the enable flag is used to indicate whether the reconstruction block of the current block needs quality enhancement, or whether the luminance reconstruction block of the current block needs quality enhancement, or whether the chroma reconstruction block of the current block needs quality enhancement, or whether both the luminance reconstruction block and the chroma reconstruction block of the current block need quality enhancement.
28. The method according to claim 27, characterized in that, The method further includes: A third rate distortion cost is determined, which is the rate distortion cost of enhancing the brightness reconstruction block through a brightness quality enhancement network based on at least one of the brightness prediction block and brightness residual block of the current block, and the brightness reconstruction block of the current block. Based on the comparison result of the first rate distortion cost and the third rate distortion cost, the value of the luminance enable flag is encoded into the bitstream.
29. The method according to claim 27, characterized in that, The method further includes: A fourth rate distortion cost is determined, which is the rate distortion cost of enhancing the chroma reconstruction block through a chroma quality enhancement network based on at least one of the chroma prediction block and the chroma residual block of the current block, and the chroma reconstruction block of the current block. Based on the comparison result of the first rate-distortion cost and the fourth rate-distortion cost, the value of the chroma enable flag is encoded into the bitstream.
30. The method according to claim 27, characterized in that, The method further includes: A fifth rate distortion cost is determined, wherein the fifth rate distortion cost is a rate distortion cost based on at least one of the chroma prediction block, chroma residual block, and luminance reconstruction block of the current block, and the chroma reconstruction block of the current block, wherein the chroma reconstruction block is quality-enhanced through a first chroma quality enhancement network; and / or, a rate distortion cost based on at least one of the chroma prediction block, chroma residual block, and quality-enhanced luminance reconstruction block of the current block, and the chroma reconstruction block of the current block, wherein the chroma reconstruction block is quality-enhanced through a second chroma quality enhancement network. Based on the comparison result of the first rate distortion cost and the fifth rate distortion cost, the value of the chroma enable flag is encoded into the bitstream.
31. The method according to any one of claims 26 to 30, characterized in that, Different blocks in the same image correspond to their respective enable flags, or different blocks within the same slice in the same image correspond to a unified enable flag, or different blocks in the same image correspond to a unified enable flag, or different blocks in different images in the same image group correspond to a unified enable flag.
32. The method according to any one of claims 25 to 31, characterized in that, The process of enhancing the quality of the reconstructed block using a quality enhancement network, based on at least one of the predicted block and residual block of the current block, and the reconstructed block of the current block, includes: Based on at least one of the predicted block, the residual block, and the adjacent reconstructed blocks of the reconstructed block of the current block, and the reconstructed block of the current block, the reconstructed block is quality-enhanced by the quality enhancement network.
33. The method according to any one of claims 25 to 32, characterized in that, The step of enhancing the quality of the reconstructed block using a quality enhancement network based on at least one of the predicted block and residual block of the current block, and the reconstructed block of the current block, is performed before loop filtering.
34. The method according to any one of claims 25 to 33, characterized in that, The current block is a CTU block.
35. The method according to any one of claims 25 to 34, characterized in that, The quality enhancement network includes a first self-attention branch network and a first cross-attention branch network. The quality enhancement of the reconstructed block based on at least one of the predicted block and residual block of the current block, and the reconstructed block of the current block, through the quality enhancement network includes: Based on the reconstructed block, a first enhancement value of the reconstructed block is determined through the first self-attention branch network; Based on at least one of the prediction block and the residual block, and the reconstruction block, a second enhancement value of the reconstruction block is determined through the first cross-attention branch network; The quality of the reconstructed block is enhanced based on the first enhancement value and the second enhancement value.
36. The method according to claim 35, characterized in that, The first self-attention branch network includes a first encoding module and a first decoding module; the step of determining a first enhancement value of the reconstructed block through the first self-attention branch network based on the reconstructed block includes: The first deep features of the reconstructed block are obtained by extracting features from the reconstructed block using the first encoding module. The first deep feature is fused by the first decoding module to obtain the first enhancement value of the reconstructed block.
37. The method according to claim 36, characterized in that, The first encoding module includes a first convolutional layer and a first feature extraction layer. The step of extracting features from the reconstructed block using the first encoding module to obtain the first deep features of the reconstructed block includes: The reconstructed block is convolved by the first convolutional layer to obtain the first shallow layer features; Based on the first shallow features, feature extraction is performed through the first feature extraction layer to obtain the first deep features.
38. The method according to claim 37, characterized in that, The first feature extraction layer includes a first self-attention subnetwork and a first feedforward subnetwork. The step of extracting features from the first shallow features using the first feature extraction layer to obtain the first deep features includes: Based on the first shallow features, feature extraction and fusion are performed through the first self-attention subnetwork to obtain the first weighted features; Based on the first weighted feature, the first deep feature is obtained by feature fusion and nonlinear mapping through the first feedforward sub-network.
39. The method according to claim 36, characterized in that, The first decoding module includes a first feature fusion layer and a second convolutional layer. The step of fusing the first deep features through the first decoding module to obtain the first enhancement value of the reconstructed block includes: Based on the first deep feature, feature fusion is performed through the first feature fusion layer to obtain the first fused feature; The first fusion feature is convolved by the second convolutional layer to obtain the first enhancement value of the reconstructed block.
40. The method according to claim 39, characterized in that, The first feature fusion layer includes a second self-attention subnetwork and a second feedforward subnetwork. The step of fusing features based on the first deep features through the first feature fusion layer to obtain the first fused feature includes: Based on the first deep feature, the second weighted feature is obtained by feature extraction and fusion through the second self-attention subnetwork; Based on the second weighted feature, the first fused feature is obtained by feature fusion and nonlinear mapping through the second feedforward sub-network.
41. The method according to any one of claims 35 to 40, characterized in that, The first cross-attention branch network includes a second encoding module and a second decoding module; the step of determining a second enhancement value of the reconstructed block based on at least one of the predicted block and the residual block, and the reconstructed block, through the first cross-attention branch network, includes: Based on at least one of the prediction block and the residual block, and the reconstruction block, the second encoding module performs feature extraction on the reconstruction block to obtain the second deep feature of the reconstruction block; Based on at least one of the prediction block and the residual block, the second deep feature is fused by the second decoding module to obtain the second enhancement value of the reconstruction block.
42. The method according to claim 41, characterized in that, The second encoding module includes a third convolutional layer and a second feature extraction layer. Based on at least one of the predicted block and the residual block, and the reconstructed block, the second encoding module extracts features from the reconstructed block to obtain a second deep feature of the reconstructed block, including: The reconstruction block is convolved by the third convolutional layer to obtain the second shallow feature of the reconstruction block. At least one of the prediction block and the residual block is convolved by the third convolutional layer to obtain at least one of the third shallow feature of the prediction block and the fourth shallow feature of the residual block. Based on at least one of the third and fourth shallow features, and the second shallow feature, feature extraction is performed through the second feature extraction layer to obtain the second deep feature.
43. The method according to claim 42, characterized in that, The second feature extraction layer includes a first cross-attention subnetwork and a third feedforward subnetwork. The second deep feature is obtained by extracting features based on at least one of the third shallow features and the fourth shallow features, as well as the second shallow features. This includes: Based on at least one of the third shallow features and the fourth shallow features, as well as the second shallow features, feature extraction and fusion are performed through the first cross-attention subnetwork to obtain the third weighted feature; Based on the third weighted feature, the second deep feature is obtained by feature fusion and nonlinear mapping through the third feedforward sub-network.
44. The method according to claim 42, characterized in that, The second decoding module includes a second feature fusion layer and a fourth convolutional layer. The step of fusing features of the second deep features based on at least one of the predicted block and the residual block to obtain the second enhancement value of the reconstructed block includes: Based on at least one of the third shallow features and the fourth shallow features, and the second deep features, feature fusion is performed through the second feature fusion network to obtain the second fused feature; The second enhancement value of the reconstructed block is obtained by performing convolution processing on the second fused feature through the fourth convolutional layer.
45. The method according to claim 44, characterized in that, The second feature fusion layer includes a second cross-attention subnetwork and a fourth feedforward subnetwork. The second feature fusion layer performs feature fusion based on at least one of the third and fourth shallow features, and the second deep feature, to obtain a second fused feature, including: Based on at least one of the third and fourth shallow features, and the second deep feature, feature extraction and fusion are performed through the second cross-attention sub-network to obtain the fourth weighted feature; Based on the fourth weighted feature, feature fusion and nonlinear mapping are performed through the fourth feedforward sub-network to obtain the second fused feature.
46. The method according to any one of claims 35 to 45, characterized in that, The quality enhancement network is based on a lightweight processing of the teacher network; The teacher network includes a second self-attention branch network and a second cross-attention branch network; the network structure of the second self-attention branch network is a U-net structure, and the second self-attention branch network includes at least two third feature extraction layers and at least two third feature fusion layers; the network structure of the second cross-attention branch network is a U-net structure, and the second cross-attention branch network includes at least two fourth feature extraction layers and at least two fourth feature fusion layers. The first feature extraction layer in the quality enhancement network corresponds to the first third feature extraction layer among the at least two third feature extraction layers; the first feature fusion layer in the quality enhancement network corresponds to the last third feature fusion layer among the at least two third feature fusion layers. The second feature extraction layer in the quality enhancement network corresponds to the first fourth feature extraction layer among the at least two fourth feature extraction layers; the second feature fusion layer in the quality enhancement network corresponds to the last second feature fusion layer among the at least two fourth feature fusion layers.
47. The method according to claim 46, characterized in that, The quality enhancement network is trained based on the distillation loss between the quality enhancement network and the teacher network, and the mean square error loss between the reconstructed block and the original block after quality enhancement.
48. The method according to claim 47, characterized in that, The distillation losses include a first distillation loss, a second distillation loss, a third distillation loss, and a fourth distillation loss; The first distillation loss is determined based on the first deep feature of the sample output by the first feature extraction layer in the quality enhancement network and the third deep feature of the sample output by the first third feature extraction layer among the at least two third feature extraction layers; The second distillation loss is determined based on the first sample fusion feature output by the first feature fusion layer in the quality enhancement network and the third sample fusion feature output by the last of the at least two third feature fusion layers; The third distillation loss is determined based on the second sample deep features output by the second feature extraction layer in the quality enhancement network and the fourth sample deep features output by the first of the at least two fourth feature extraction layers. The fourth distillation loss is determined based on the second sample fusion feature output by the second feature fusion layer in the quality enhancement network and the fourth sample fusion feature output by the last of the at least two fourth feature fusion layers.
49. A decoding device, characterized in that, The device includes: A decoding unit is configured to enhance the quality of the reconstructed block using a quality enhancement network based on at least one of the predicted block and residual block of the current block, and the reconstructed block of the current block.
50. An encoding device, characterized in that, The device includes: An encoding unit is configured to enhance the quality of a reconstructed block using a quality enhancement network, based on at least one of a predicted block and a residual block of the current block, and a reconstructed block of the current block.
51. A decoder, characterized in that, The decoder includes a memory and a processor, the memory being used to store a computer program running on the processor; the processor being used to execute the image decoding method as described in any one of claims 1 to 24 when running the computer program.
52. An encoder, characterized in that, The encoder includes a memory and a processor, the memory being used to store a computer program running on the processor; the processor being used to execute the image encoding method as described in any one of claims 25 to 48 when running the computer program.
53. A non-volatile computer-readable storage medium for storing a bitstream, characterized in that, The bitstream is generated by using an image encoding method of an encoder, or the bitstream is decoded by using an image decoding method of a decoder, wherein the image decoding method includes the image decoding method as described in any one of claims 1 to 24, or the image encoding method includes the image encoding method as described in any one of claims 25 to 48.
54. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium, a processor retrieving the computer instructions from the computer-readable storage medium, and the processor executing the computer instructions to implement the image decoding method as described in any one of claims 1 to 24, or the image encoding method as described in any one of claims 25 to 48.
