Image processing method and system, electronic device, storage medium, and program product
By using a feature transformation network based on nonlinear aggregation and weighted fitting, the problem of high complexity of transformation modules in deep learning image compression technology is solved, achieving more efficient image compression and decoding, which is suitable for scenarios such as real-time communication, live streaming, and video conferencing.
Patent Information
- Application Number
- PCT/CN2026/094276
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-13
- Filing Date
- 2026-04-30
- Publication Date
- 2026-11-19
AI Technical Summary
In existing deep learning-based image compression technologies, the high complexity of the transformation module limits the deployment and application of image compression algorithms.
A feature transformation network employing nonlinear aggregation and weighted fitting is used to generate more representative and diverse feature representations by performing nonlinear transformation and aggregation on the features to be transformed, and weighted fitting is performed on the aggregated feature elements to reduce redundant information.
With low complexity and lightweight design, it improves image compression efficiency and reduces encoding or decoding overhead, making it suitable for real-time high-definition video streaming and other image processing scenarios on mobile devices.
Smart Images

Figure CN2026094276_19112026_PF_FP_ABST
Abstract
Description
Image processing methods, systems, electronic devices, storage media, and software products
[0001] This disclosure claims priority to Chinese Patent Application No. 202510615554.0, filed on May 13, 2025, entitled "Image Processing Method, System, Electronic Device, Storage Medium and Program Product", the entire contents of which are incorporated herein by reference. Technical Field
[0002] This disclosure relates to the field of image processing technology, and in particular to image processing methods, systems, electronic devices, storage media and program products, which can be applied to the field of artificial intelligence. Background Technology
[0003] With the development of network and mobile communication technologies, the demand for image processing technology in scenarios such as real-time communication (RTC), live streaming, and video conferencing has increased dramatically. In recent years, end-to-end image compression technology based on deep learning has gradually become a mainstream solution due to its adaptive learning capabilities and global optimization characteristics. This type of method uses an encoder-decoder architecture, where the transformation module primarily achieves its function by stacking a large number of convolutional neural networks (CNNs) and self-attention transformation networks. This transformation module typically has high complexity, which limits the deployment and application of image compression algorithms. Summary of the Invention
[0004] This disclosure provides an image processing method, an image processing system, an electronic device, a computer-readable storage medium, and a computer program product to alleviate or solve one or more technical problems existing in the prior art.
[0005] In a first aspect, embodiments of this disclosure provide an image processing method, comprising: generating features to be transformed based on a first image, wherein the features to be transformed include feature elements at multiple spatial locations; performing nonlinear aggregation on the features to be transformed to obtain aggregated features; performing weighted fitting on the feature elements in the aggregated features to obtain fitted features; and generating a bitstream corresponding to the first image using the fitted features.
[0006] Secondly, embodiments of this disclosure provide an image processing method, comprising: generating features to be transformed based on a bitstream, wherein the bitstream is generated based on a first image, and the features to be transformed include feature elements at multiple spatial locations; performing nonlinear aggregation on the features to be transformed to obtain aggregated features; performing weighted fitting on the feature elements in the aggregated features to obtain fitted features; and reconstructing the first image using the fitted features to obtain a second image.
[0007] Thirdly, embodiments of this disclosure provide an image processing system, including: an encoding end for executing the method of the first aspect to generate a bitstream; and a decoding end for obtaining the bitstream from the encoding end and executing the method of the second aspect.
[0008] Fourthly, embodiments of this disclosure provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements the method of any one of the embodiments of this disclosure when executing the computer program.
[0009] Fifthly, embodiments of this disclosure provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method of any one of the embodiments of this disclosure.
[0010] Sixthly, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the method of any one of the embodiments of this disclosure.
[0011] According to the technical solution of this disclosure, in the process of image encoding (compression) or decoding (decompression), firstly, by performing a nonlinear transformation on the features to be transformed, image details and structural information are better captured, and spatially adjacent feature elements are aggregated to generate more representative, diverse, and less redundant feature representations, providing high-quality input for subsequent weighted fitting. Furthermore, weighted fitting is performed on the feature elements in the aggregated features to highlight key elements, suppress redundant elements, and optimize feature distribution. Compared to encoding and decoding by stacking a large number of convolutional neural networks and self-attention transformation networks, this disclosure, through nonlinear aggregation and weighted fitting processing during the encoding or decoding process, can remove more redundancy and reduce encoding or decoding overhead under the premise of lower complexity and lightweight design, achieving higher compression or decompression efficiency.
[0012] The above description is only an overview of the technical solution of this disclosure. In order to better understand the technical means of this disclosure, it can be implemented in accordance with the contents of the specification. In order to make the above and other objects, features and advantages of this disclosure more easy to understand, specific embodiments of this disclosure are given below. Attached Figure Description
[0013] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments according to this disclosure and should not be construed as limiting the scope of this disclosure.
[0014] Figure 1 shows an architecture diagram of the image processing system provided in an embodiment of this disclosure.
[0015] Figure 2 is a flowchart of the image processing method provided in Embodiment 1 of this disclosure.
[0016] Figure 3 shows an application example diagram of the encoding network and encoding process according to an embodiment of the present disclosure.
[0017] Figure 4 shows an example of the application of the Residual Downsampling (ResDown) layer in an embodiment of this disclosure.
[0018] Figure 5 shows an example of the application of the nonlinear aggregation module in an embodiment of this disclosure.
[0019] Figure 6 shows an example of the application of the Deep Residual Bottleneck Layer (DepthRB) in an embodiment of this disclosure.
[0020] Figure 7 shows an example of the application of the feature gating module in an embodiment of this disclosure.
[0021] Figure 8 shows an example of the application of the gate transform network in the embodiments of this disclosure.
[0022] Figure 9 is a flowchart of the image processing method provided in Embodiment 2 of this disclosure.
[0023] Figure 10 shows an application example diagram of the decoding network and decoding process according to an embodiment of the present disclosure.
[0024] Figure 11 shows an example of the application of the Residual Upsampling (ResUp) layer in an embodiment of this disclosure.
[0025] Figure 12 shows a block diagram of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation
[0026] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of this disclosure. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.
[0027] Application scenarios
[0028] With the continuous advancement of deep learning technology, encoder-decoder architectures based on this technology are increasingly widely used in image compression. The transform module in an encoder-decoder architecture is typically composed of deep transform networks. Compared to traditional hand-designed image compression methods, deep network-based image compression methods can achieve end-to-end optimization and, due to learning from large amounts of data, possess greater flexibility and better compression efficiency. Currently, the transform module in an encoder-decoder architecture mainly achieves its function by stacking a large number of convolutional neural networks and self-attention transform networks. However, existing transform modules often come with high complexity, which to some extent limits the practical application scope of deep network-based image compression methods.
[0029] In view of this, the present disclosure aims to provide an image processing method that, while ensuring compression performance, constructs a lightweight and low-complexity feature extraction module and improves the nonlinear expression and fitting capabilities of the encoding and decoding networks, thereby effectively eliminating redundant information and reducing the overhead required for encoding in feature extraction.
[0030] Figure 1 shows the architecture of the image processing system provided in this embodiment. As shown in Figure 1, the image processing system includes an encoding end and a decoding end. The encoding end represents a first image as features to be transformed, performs nonlinear aggregation and weighted fitting on the features to be transformed to extract features to be quantized, and then quantizes and entropy-encodes the features to be quantized to generate a bitstream corresponding to the first image. The bitstream corresponding to the first image is transmitted from the encoding end to the decoding end via a network or other communication method. The decoding end decodes the bitstream and performs probability estimation on the quantized features to obtain the corresponding quantized features, then performs nonlinear aggregation and weighted fitting on the quantized features to extract the target image features of the first image, and then reconstructs the second image using the target image features.
[0031] For example, the encoding and decoding ends can be applications, services, instances, software modules, virtual machines (VMs), containers, or cloud servers, or hardware devices (such as servers or terminal devices) or hardware chips with data processing capabilities. Terminal devices can be fixed terminals, such as personal computers, or mobile terminals, such as mobile phones or tablets. The encoding and decoding ends can be deployed on different sides, for example, on different terminal devices or servers. They can also be deployed on the same side; for example, they can be deployed as software modules on the same terminal device, or as cloud products on the same cloud server or in the same cluster.
[0032] The image processing method provided in this disclosure can be widely applied to scenarios such as real-time communication (RTC), live streaming, video conferencing, mobile terminal image transmission, and remote medical image transmission. For example, in real-time communication, this solution uses a lightweight transformation module (a feature transformation network for implementing nonlinear aggregation and weighted fitting) to significantly reduce computational complexity while ensuring compression efficiency, supporting real-time encoding and decoding of high-definition video streams, and is particularly suitable for bandwidth-constrained environments of mobile devices. Furthermore, the image processing method provided in this disclosure can also be applied to fields such as intelligent security monitoring and autonomous driving perception systems. For example, in security monitoring, the encoding end can perform dynamic feature nonlinear aggregation and weighted fitting on the monitoring video, reducing redundant information and improving the compression ratio; in autonomous driving, images captured by onboard cameras are compressed using this solution and transmitted to edge computing nodes, meeting real-time road condition processing requirements while reducing the onboard computing load.
[0033] It should be noted that the application scenarios or examples provided in this disclosure are for ease of understanding, and this disclosure does not specifically limit the application of the technical solutions. Furthermore, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. The collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.
[0034] The technical solutions of this disclosure and how they solve the aforementioned technical problems are described in detail below with specific embodiments. The listed specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.
[0035] Example 1
[0036] Figure 2 is a flowchart of an image processing method provided in an embodiment of this disclosure. This method can be applied to an image processing device, which can be deployed at an encoding end. As shown in Figure 2, the image processing method includes steps S201, S202, S203, and S204.
[0037] Step S201: Generate features to be transformed based on the first image, wherein the features to be transformed include feature elements at multiple spatial locations.
[0038] The first image can be an image to be compressed or encoded at the encoding end. It can be an image captured by an image sensor, a computer-generated image, or a single frame extracted from a video. This embodiment does not specifically limit the source of the first image. The first image typically includes one or more channels, which can include color channels, depth channels, or other types of channels. Therefore, the first image can be represented as a multidimensional array, where multidimensionality refers to multiple channel dimensions.
[0039] The features to be transformed are used to represent the first image. That is, feature extraction and representation of the first image can generate the features to be transformed. Therefore, the features to be transformed include feature elements at multiple spatial locations. For example, the first image can be downsampled to reduce its spatial resolution or size, thereby obtaining the features to be transformed. Downsampling includes, but is not limited to, convolutional downsampling (ConvDown), bilinear sampling, average pooling, max pooling, and residual downsampling (ResDown). Compared to the first image, the features to be transformed can reduce the amount of data, thereby reducing the computational cost of the encoding process. For example, one or more downsampling layers can be deployed in the encoding network.
[0040] It should be noted that generating the feature to be transformed based on the first image can include directly downsampling the first image to generate the feature to be transformed, or it can include downsampling the intermediate features obtained after processing the first image to generate the feature to be transformed. For example, the intermediate features can be the fitted features obtained in step S203.
[0041] Step S202: Perform nonlinear aggregation on the features to be transformed to obtain aggregated features.
[0042] Nonlinear aggregation includes nonlinear transformation and feature aggregation.
[0043] Nonlinear transformation refers to nonlinear feature mapping. For example, nonlinear activation functions are used to perform nonlinear feature mapping on the features input to the function, thereby enhancing the nonlinear expressive power of the mapped features and better capturing details and structural information in the image. Nonlinear activation functions include, but are not limited to, the Gaussian Error Linear Unit (GELU) function or the Rectified Linear Unit (ReLU) function.
[0044] Feature aggregation refers to aggregating information from spatially adjacent elements, that is, combining elements with adjacent spatial positions in a feature set to reduce redundancy between adjacent elements in the spatial domain. The feature element of an aggregated feature is the element obtained after aggregating elements with adjacent spatial positions. For example, a deep separable convolution (DWConv) network with a large receptive field can be used for feature aggregation.
[0045] For example, one or more nonlinear aggregation modules can be deployed in the encoding network to perform nonlinear aggregation on the features (including the features to be transformed) of the input nonlinear aggregation module. The nonlinear aggregation module includes one or more nonlinear transformation layers and one or more depthwise separable convolutional (DWConv) layers to perform nonlinear aggregation on the input data (the input features of the nonlinear aggregation module). The nonlinear transformation layer can be constructed based on the GELU function.
[0046] Step S203: Perform weighted fitting on the feature elements in the aggregated features to obtain the fitted features.
[0047] Weighted fitting includes weighted transformation and fitting. Weighted transformation refers to dynamically assigning weights to each feature element in the aggregated feature during feature extraction, and then weighting the feature elements of the aggregated feature based on the weights of each feature element to generate a weighted feature. This highlights key elements (such as high-frequency edge elements or texture detail elements) and suppresses redundant elements (such as background elements). Fitting refers to fitting the weighted feature to the aggregated feature to preserve the original feature details and avoid information loss.
[0048] For example, one or more feature gating modules can be deployed in the encoding network to perform weighted fitting of the features (including aggregated features) input to the feature gating modules. The feature gating module includes a gating transformation network and a residual connection layer. The gating transformation network performs a weighted transformation on the feature elements in the aggregated features through a gating mechanism to generate weighted features. The residual connection layer is used to perform residual connections between the weighted features output by the gating transformation layer and the features (aggregated features) input to the feature gating module. These residual connections include, but are not limited to, addition, subtraction, or concatenation along spatial channels.
[0049] Step S204: Generate a bitstream corresponding to the first image using the fitted features.
[0050] The bitstream, also known as the code stream, is typically a data stream represented in binary form (i.e., a sequence of "0"s and "1"s). For example, fitted features can be downsampled to generate features to be quantized. Further, the features to be quantized are quantized and entropy encoded to obtain the bitstream.
[0051] The quantization process converts the features to be quantized into integers, resulting in quantized features. This maps continuous feature values to a finite discrete set, further reducing the amount of data. Entropy coding can be, for example, arithmetic encoding (AE) or interval coding. For instance, a deep probabilistic estimation network can be used to estimate the probability distribution of the quantized features and entropy-encode each quantized symbol in the features according to these probability distributions. Symbols with higher probabilities are encoded using shorter codewords, thus outputting a binary bitstream to compress the first image. The deep probabilistic estimation network can use various forms of probability estimation networks, including but not limited to convolutional layers and self-attention networks.
[0052] According to the technical solution of this disclosure, in the process of image encoding (compression), firstly, through nonlinear transformation of the features to be transformed, the rich details and structural information in the image are better captured, and spatially adjacent feature elements are aggregated to generate more representative, diverse, and less redundant feature representations, providing high-quality input for subsequent weighted fitting. Furthermore, weighted fitting is performed on the feature elements in the aggregated features to highlight key elements, suppress redundant elements, and optimize feature distribution. Compared to encoding by stacking a large number of convolutional neural networks and self-attention transformation networks, this disclosure, through nonlinear aggregation and weighted fitting processing during the encoding process, can remove more redundancy, reduce encoding overhead, and achieve higher compression efficiency with lower complexity and lightweight design.
[0053] In one embodiment, in step S201, generating the feature to be transformed based on the first image may include: performing residual downsampling (ResDown) on the first image or the fitted features obtained based on the first image to obtain the feature to be transformed.
[0054] In other words, step S201 can be implemented using a residual downsampling layer. The input features of the residual downsampling layer include the first image or the aforementioned fitted features obtained based on the first image. For example, the residual downsampling layer can be used to directly perform residual downsampling on the first image to generate the features to be transformed, or the residual downsampling layer can be used to perform residual downsampling on the intermediate features obtained after processing the first image to generate the features to be transformed. These intermediate features can be fitted features obtained after performing one or more residual downsampling operations (as in step S201), nonlinear aggregation (as in step S202), and weighted fitting (as in step S203) on the first image. That is to say, steps S201, S202, and S203 can all be executed multiple times.
[0055] For example, the residual downsampling layer can make the width and height of the output feature half the width and height of the input feature by using a convolutional downsampling layer, and then use a residual connection layer to perform residual connection on the output of the convolutional downsampling layer and the input feature of the residual downsampling layer to generate the feature to be transformed.
[0056] Understandably, in this implementation, residual downsampling introduces skip connections during the downsampling process, allowing the input features of the residual downsampling layer to skip certain layers and be directly passed to deeper layers (residual connection layers), thereby alleviating the gradient vanishing problem in the downsampling feature extraction process and improving the feature representation capability of the features to be transformed.
[0057] In a specific application example, multiple downsampling layers and multiple tokenized feature transformation (TokenMixing) networks are deployed in the encoding network. Figure 3 illustrates an application example of the encoding network and encoding process according to an embodiment of this disclosure. The multiple downsampling layers include multiple residual downsampling (ResDown) layers and one convolutional downsampling (ConvDown) layer. The tokenized feature transformation network is used to perform feature transformations (including nonlinear aggregation and weighted fitting) on the input features. This will be described in detail below.
[0058] As shown in Figure 3, when the first image x is input into the encoding network, it undergoes three feature extractions and one downsampling. Each feature extraction is implemented through a feature extraction module; that is, the first feature extraction module is used for the first feature extraction, the second feature extraction module is used for the second feature extraction, and the third feature extraction module is used for the third feature extraction.
[0059] Furthermore, the feature extraction module includes a residual downsampling (ResDown) layer and one or more feature transformation (TokenMixing) networks. The residual downsampling layer is used to implement step S201, and the feature transformation network includes the aforementioned nonlinear aggregation module (used to implement step S202) and the aforementioned feature gating module (used to implement step S203). After three feature extractions, the transformed features are input into a 3×3 convolutional downsampling (ConvDown) layer to obtain the feature y to be quantized.
[0060] Furthermore, the quantized feature y is obtained by quantizing (Q) the feature to be quantized into an integer.
[0061] The encoding method shown in Figure 3 can be represented by the following formulas: y0 = ResDown(x), y1 = TokenMixing(y0), y2 = ResDown(y1), y3 = TokenMixing(y2), y4 = ResDown(y3), y5 = TokenMixing(y4), y = ConvDown(y5).
[0062] Where y0 and y1 represent the features to be transformed and the fitted features during the first feature extraction, respectively; y2 and y3 represent the features to be transformed and the fitted features during the second feature extraction, respectively; and y4 and y5 represent the features to be transformed and the fitted features during the third feature extraction, respectively.
[0063] It is understood that three feature extraction modules are deployed in the coding network shown in Figure 3 for three-stage feature extraction. Each feature extraction module includes a residual downsampling layer and a feature transformation network. However, the embodiments of this disclosure do not specifically limit the number of feature extraction modules or the number of feature transformation networks in each feature extraction module.
[0064] Figure 4 illustrates an application example of the residual downsampling (ResDown) layer in an embodiment of this disclosure. The residual downsampling layer includes: a convolutional downsampling (ConvDown) layer, used to downsample the input feature y00 of the residual downsampling layer to generate an intermediate feature y01; a GELU function, used to perform non-linear activation on the intermediate feature y01 to generate an intermediate feature y02; a convolution (Conv) layer, used to perform a convolution operation on the intermediate feature y02 to generate an intermediate feature y03; a normalization (Norm) layer, used to normalize the feature distribution of the intermediate feature y03 to generate an intermediate feature y04; and a residual connection layer, used to perform a residual connection between the intermediate feature y04 and the input feature y00 to generate the output feature of the residual downsampling layer.
[0065] The residual downsampling method shown in Figure 4 can be expressed by the following formulas: y01 = ConvDown(y00), y02 = GELU(y01), y03 = Conv(y02), y04 = Norm(y03).
[0066] It is understandable that when the residual downsampling layer is deployed at different locations in the coding network, its input feature y00 and output feature y00 will differ. The meanings are different. For example, when the residual downsampling layer is deployed in the first feature extraction module shown in Figure 3, the input feature y00 and the output feature... These represent the first image and the features to be transformed during the first feature extraction, respectively; when the residual downsampling layer is deployed in the second feature extraction module shown in Figure 3, the input feature y00 and the output feature y00 are respectively... These represent the fitted features obtained from the first feature extraction and the features to be transformed during the second feature extraction, respectively; when the residual downsampling layer is deployed in the third feature extraction module shown in Figure 3, the input feature y00 and the output feature y00 are respectively... These represent the fitted features obtained from the second feature extraction and the features to be transformed during the third feature extraction, respectively.
[0067] It should be noted that Figure 4 uses element-wise addition (+) of residual connections as an example, but the embodiments of this disclosure do not specifically limit the method of residual connections. For example, it can also be element-wise subtraction or splicing in the spatial channel dimension. In addition, Figure 4 is an example of a residual downsampling layer and residual downsampling method, but it is not limited to this.
[0068] The following sections provide a detailed description of the nonlinear aggregation module in the feature transformation network and the specific implementation of step S202 (implementing nonlinear aggregation). A detailed description is also provided of the feature gating module in the feature transformation network (which can be used to implement step S203).
[0069] In one embodiment, step S202, performing nonlinear aggregation on the features to be transformed to obtain aggregated features, may include: performing nonlinear feature mapping on the features to be transformed to obtain mapped features; aggregating feature elements with adjacent spatial positions in the mapped features to obtain candidate features; and performing residual connection between the candidate features and the features to be transformed to generate aggregated features.
[0070] For example, before performing nonlinear feature mapping on the features to be transformed, the features can be standardized (normated) to normalize their feature distribution. For instance, the nonlinear aggregation module may include a standardization layer, which is used to standardize the features input to the standardization layer. Further, the standardized features are then subjected to nonlinear feature mapping to obtain the mapped features.
[0071] The nonlinear feature mapping is the aforementioned nonlinear transformation. For example, the nonlinear aggregation module may include a deep residual bottleneck (DepthRB) layer, which is used to perform nonlinear feature mapping on the features input to the deep residual bottleneck layer.
[0072] Furthermore, aggregating feature elements with adjacent spatial locations within the mapped features, i.e., feature aggregation of the mapped features, reduces redundant relationships between adjacent elements in the spatial domain. For example, the nonlinear aggregation module may include a depthwise separable convolutional (DWConv) layer and a linear layer. The DWConv layer effectively aggregates feature elements with adjacent spatial locations by performing depthwise convolutions (applying filters individually to each channel) and point convolutions with a large receptive field (1×1) on the features from the DWConv layer. The linear layer further processes the output from the DWConv layer, for example, by integrating the features output by the DWConv layer through fully connected layers or other forms of linear transformations to generate candidate features.
[0073] For example, residual connections include, but are not limited to, addition, subtraction, or splicing in the spatial channel dimension, etc., to perform residual connections between the candidate features and the original features to be transformed (the features to be transformed without standardization) to alleviate the gradient vanishing problem in the nonlinear aggregation process and generate aggregated features.
[0074] Figure 5 illustrates an application example of the nonlinear aggregation module in an embodiment of this disclosure. In this application example, the nonlinear aggregation module in the TokenMixing network can be used to implement step S202, that is, to perform nonlinear aggregation on the features to be transformed to obtain aggregated features.
[0075] As shown in Figure 5, the nonlinear aggregation module can include a two-step transformation. The first transformation includes a normalization layer, which is used to normalize the feature f00 to be transformed output by the residual downsampling layer, that is, to normalize the feature distribution of the feature f00 to be transformed, and generate the normalized feature to be transformed. The Deep Residual Bottleneck (DepthRB) layer is used to transform the standardized features. Perform nonlinear mapping to generate mapping features Depthwise separable convolutional (DWConv) layers are used to aggregate mapped features. Intermediate features are generated from feature elements with adjacent spatial locations. Linear layers are used for processing intermediate features. Further linear transformation generates candidate features. The second transformation step includes a residual connection layer, used to transform the candidate features... The aggregated feature is generated by performing a residual concatenation with the untransformed feature f00 output from the residual downsampling layer.
[0076] It should be noted that Figure 5 uses element-wise addition (+) of residual connections as an example, but the embodiments of this disclosure do not specifically limit the method of residual connections. For example, it can also be element-wise subtraction or splicing in the spatial channel dimension. In addition, the deployment of the standardization layer and the standardization processing of the features to be transformed output by the residual downsampling layer are only examples and not limitations. For example, the nonlinear aggregation module may not deploy a standardization layer, and the features to be transformed output by the residual downsampling layer can be directly input into the depth residual bottleneck layer.
[0077] The nonlinear aggregation method shown in Figure 5 can be represented by the following formula:
[0078] It is understandable that multiple feature extraction modules can be deployed in the encoding network. Each feature extraction module includes a residual downsampling (ResDown) layer and one or more feature transformation (TokenMixing) networks. The nonlinear aggregation module shown in Figure 5 is used to perform nonlinear aggregation on the features to be transformed in the same feature extraction module to obtain the aggregated features of the feature extraction module.
[0079] Based on this implementation, in the nonlinear aggregation process, the nonlinear representation capability of features is enhanced by nonlinear feature mapping, thereby increasing the depth and breadth of feature extraction. Redundancy between adjacent elements in the spatial domain is reduced by feature aggregation. Then, the gradient vanishing problem in the nonlinear aggregation process is alleviated by residual connection. Furthermore, this nonlinear aggregation method can also reduce the complexity and coding overhead of the coding network.
[0080] In one implementation, performing nonlinear feature mapping on the feature to be transformed to obtain mapped features may include: expanding the spatial channel dimension of the feature to be transformed through a linear transformation operation to generate expanded features; enhancing the feature elements at some spatial locations in the expanded features based on a nonlinear activation function and a depthwise separable convolutional network to obtain enhanced features; restoring the spatial channel dimension of the enhanced features to the spatial channel dimension of the feature to be transformed, and then performing a residual connection with the feature to be transformed to generate mapped features.
[0081] The linear transformation operation can be implemented using a 1×1 convolution. This linear transformation expands the spatial channel dimension of the feature to be transformed (or the standardized feature to be transformed), i.e., increases the number of spatial channels (dimensionality increase). For example, the spatial channel dimension of the feature to be transformed is C. in The spatial channel dimension of the extended feature is C. out .
[0082] It is understandable that extended features include feature elements at multiple spatial locations. Feature enhancement is performed on feature elements at some spatial locations, i.e., nonlinear enhancement is applied to the extended features, resulting in enhanced features. For example, nonlinear enhancement of extended features can be performed based on nonlinear activation functions and a Deeply Separable Convolutional (DWConv) network. The nonlinear activation function is used to nonlinearly activate the input features; nonlinear activation functions include, but are not limited to, Gaussian Error Linear Unit (GELU) functions or ReLU functions.
[0083] It should be noted that multiple nonlinear activation functions can be used to perform multiple nonlinear activations. For example, after the first nonlinear activation function performs a nonlinear activation on the extended features, the output features are input into the DWConv network for feature extraction. The extracted features are then input into the second nonlinear activation function for a second nonlinear activation, and the output is the enhanced features.
[0084] Furthermore, the spatial channel dimension of the enhanced feature is restored to the spatial channel dimension of the feature to be transformed (dimensionality reduction). Then, the dimensionality-reduced enhanced feature is residually joined with the feature to be transformed (or the standardized feature to be transformed) to generate a mapped feature. The dimensionality reduction operation can be achieved through a linear transformation operation, that is, by reducing the number of spatial channels through a linear transformation operation. For example, the spatial channel dimension of the enhanced feature is C. in After dimensionality reduction, the spatial channel dimension of the enhanced features is restored to C. in Residual joins include, but are not limited to, addition, subtraction, or splicing along spatial channel dimensions.
[0085] In a specific application example, the Deep Residual Bottleneck (DepthRB) layer can be deployed in the nonlinear aggregation module of the TokenMixing network to transform the feature f00 output by the ResDown layer or the standardized feature f00. Perform nonlinear feature mapping to obtain the mapped features. Figure 6 illustrates an application example of the Deep Residual Bottleneck Layer (DepthRB) in an embodiment of this disclosure. Figure 6 shows the features to be transformed after normalization. As input to the deep residual bottleneck layer.
[0086] As shown in Figure 6, the deep residual bottleneck layer may include: a first linear layer, used to expand the normalized features to be transformed through linear transformation operations. Spatial channel dimension, generating extended features The first nonlinear activation function (such as the GELU function) is used to expand the features. Perform a non-linear activation to generate intermediate features. Depthwise separable convolutional (DWConv) layers are used to process intermediate features from the output of the first nonlinear activation function. Perform feature extraction to generate intermediate features A second nonlinear activation function (such as the GELU function) is used to apply intermediate features from the output of depthwise separable convolutional layers. Perform further nonlinear activation to generate enhanced features. The second linear layer is used to enhance features. Spatial channel dimension recovery to the feature to be transformed Spatial channel dimension, generating intermediate features Residual connection layer, used to process intermediate features output by the second linear layer. Features to be transformed after standardization Perform residual connections to generate mapped features.
[0087] The nonlinear feature mapping method shown in Figure 6 can be represented by the following formula:
[0088] Compared to the traditional method of linearly stacking convolutional layers for feature transformation, the aforementioned nonlinear feature mapping implementation, through a nonlinear activation function, can provide smooth nonlinear transformations across different input value ranges while maintaining lightweight design. This allows for the modeling of complex spatial relationships between feature elements. The Depthily Separable Convolutional (DWConv) network can effectively expand the receptive field and reduce computational cost during feature extraction. Furthermore, the dimensionality increase before nonlinear enhancement increases feature dimensions and enhances feature expressiveness, while the dimensionality reduction after nonlinear enhancement restores the number of spatial channels, reducing the complexity of subsequent computations.
[0089] The following details the specific implementation of the feature gating module in the feature transformation network and step S203 (implementing weighted fitting).
[0090] In one embodiment, in step S203, the feature elements in the aggregated feature are weighted and fitted to obtain the fitted feature, including: dividing the aggregated feature into a first aggregated sub-feature and a second aggregated sub-feature, wherein the first aggregated sub-feature, the second aggregated sub-feature, and the aggregated feature have the same spatial channel dimension; generating a weighted feature using the first aggregated sub-feature, wherein the weighted feature includes the weights of the feature elements in the aggregated feature; weighting the feature elements in the second aggregated sub-feature based on the weighted feature to generate a weighted feature; and performing a residual connection between the weighted feature and the aggregated feature to generate the fitted feature.
[0091] One approach is to use convolution (Conv) to increase the dimensionality of the aggregated feature. The spatial channel dimension of the aggregated feature after the increase is twice that of the aggregated feature before the increase. The aggregated feature after the increase is then divided into two parts in the spatial channel dimension, with each part having the same spatial channel dimension. This results in a first aggregated sub-feature and a second aggregated sub-feature with the same spatial channel dimension as the aggregated feature.
[0092] Furthermore, weight features are generated using the first aggregate feature, that is, the weights of each feature element in the first aggregate feature are determined. The weights of each feature element obtained constitute the weight feature, which is used to represent the weights of each feature element in the aggregate feature.
[0093] Furthermore, the weights of each feature element in the first aggregate feature are used to weight each feature element in the second aggregate feature. The resulting weighted feature can highlight key elements (such as high-frequency edge elements or texture detail elements) and suppress redundant elements (such as background elements).
[0094] Furthermore, residual connections are performed on the weighted features and aggregated features to generate fitted features. Residual connection methods include, but are not limited to, element-wise addition, element-wise subtraction, or splicing along spatial channel dimensions.
[0095] Compared to traditional attention mechanisms, this implementation divides the aggregated features into a first aggregated feature and a second aggregated feature. It uses the first aggregated feature to assign weights to feature elements and then weights the second aggregated feature. This simplifies the computational complexity of the weighted fitting process, thereby reducing the complexity of the encoding network and improving encoding (i.e., compression) efficiency. Furthermore, this segmentation-based weight determination method can rely on internal feature transformations to allocate weights, avoiding the need for an external weight generation module.
[0096] In one implementation, generating weighted features using first aggregate features includes: inputting feature elements from the first aggregate features into a nonlinear activation function to obtain weighted features.
[0097] The nonlinear activation functions include, but are not limited to, the GELU function or the ReLU function. A nonlinear transformation is performed on the first aggregate feature based on the nonlinear activation function to determine the weights of each feature element in the first aggregate feature.
[0098] This nonlinear transformation method for determining weights can further break the limitations of traditional attention mechanisms in determining the correlation between feature elements, so that no attention mechanism is needed in the entire encoding process, thereby further reducing the complexity of the encoding network.
[0099] In one implementation, weighting the feature elements in the second aggregate feature based on the weight feature to generate a weighted feature includes: performing a feature element-wise multiplication operation between the weight feature and the second aggregate feature to generate the weighted feature.
[0100] The element-wise multiplication operation can be implemented based on the Hadamard product. The Hadamard product is the operation of multiplying corresponding elements of two matrices of the same shape (or multidimensional arrays of the same channel dimension) to obtain a new matrix of the same shape.
[0101] Since the element-wise multiplication operation is performed between tensors of the same shape, using element-wise multiplication in the weighted fitting process can preserve more feature element information of the original features and enhance the feature representation ability of the encoding network.
[0102] Figure 7 illustrates an application example of the feature gating module in an embodiment of this disclosure. In this application example, the feature gating module in the TokenMixing network can be used to implement step S203, that is, to perform weighted fitting on the feature elements in the aggregated features to obtain the fitted features.
[0103] As shown in Figure 7, the feature gating module can implement a two-step transformation. The network structure implementing the first-step transformation includes a normalization layer, which normalizes the input feature (i.e., the aggregated feature output by the nonlinear aggregation module) f2, that is, normalizes the feature distribution of the aggregated feature f2, generating the normalized aggregated feature. Gated transform networks are used to implement weighted transforms, that is, to aggregate standardized features through a gating mechanism. The feature elements in the data are weighted to generate weighted features. The network structure that implements the second-step transformation includes: a residual connection layer, used to process the weighted features. and aggregation features Perform residual connections to generate fitted features.
[0104] The weighted fitting method shown in Figure 7 can be represented by the following formula:
[0105] Figure 8 illustrates an application example of the gated transformation network in this embodiment of the present disclosure. This gated transformation network can be deployed within the feature gate module of a tokenized transformation network.
[0106] As shown in Figure 8, the gated transformation network may include: a linear layer, used to process its input features (standardized aggregate features). Perform dimensionality increase to generate dimensionality-increased features. Features after dimensionality upgrade The spatial channel dimension is the input feature Twice the spatial channel dimension; a split layer is used to divide the increased-dimensional features. The spatial channel dimension is divided into two parts, each with the same spatial channel dimension, thus obtaining a first aggregated sub-feature with the same spatial channel dimension as the aggregated feature f2. Second polymer features The GELU function is used to analyze the features of the first aggregate. Nonlinear activation is performed to determine the weights of each feature element in the first aggregate feature, generating a weighted feature; a Hadamard layer is used to perform element-wise multiplication of the weighted feature with the second aggregate feature to generate intermediate features. Residual connection layer, used to connect intermediate features Compared with input features (aggregated features after standardization) Residual connections are performed to mitigate the gradient vanishing problem during the weighted transformation process, generating output features (i.e., weighted features).
[0107] The weighted transformation method shown in Figure 8 can be expressed by the following formula:
[0108] Example 2
[0109] Figure 9 is a flowchart of an image processing method provided in an embodiment of this disclosure. This method can be applied to an image processing device, which can be deployed at a decoding end. As shown in Figure 9, the image processing method includes steps S901, S902, S903, and S904.
[0110] Step S901: Generate features to be transformed based on the bitstream, wherein the bitstream is generated based on the first image, and the features to be transformed include feature elements at multiple spatial locations.
[0111] The bitstream can be obtained from the encoder. For example, a deep probabilistic estimation network can be used to estimate the probability distribution of the quantized features, and entropy decoding can be performed on the bitstream based on these probability distributions to obtain the quantized features. Entropy decoding can be, for example, arithmetic decoding (AD) or interval decoding.
[0112] For example, quantized features can be directly used as the features to be transformed, or intermediate features after processing quantized features can be upsampled to generate the features to be transformed. These intermediate features can be, for example, the fitted features obtained in step S903. Upsampling includes, but is not limited to, convolutional upsampling (ConvUp), subpixel convolution, bilinear sampling, and residual upsampling (ResUp), where convolutional upsampling can also be called deconvolution. For example, one or more upsampling layers can be deployed in the decoding network to generate the features to be transformed. Compared to the bitstream, the features to be transformed can reduce the amount of data, thereby reducing the computational load of the decoding process.
[0113] Step S902: Perform nonlinear aggregation on the features to be transformed to obtain aggregated features. Step S902 can be implemented in a manner similar to or the same as step S202.
[0114] Step S903: Perform weighted fitting on the feature elements in the aggregated features to obtain the fitted features. Step S903 can be implemented in a manner similar to or the same as step S203.
[0115] Step S904: Reconstruct the first image using the fitted features to obtain the second image.
[0116] The bitstream is generated based on the first image, and the second image is generated by reconstructing the first image. For example, the fitted features are deconvolved to recover and restore details in the first image, thereby reconstructing the first image and generating the second image.
[0117] According to the technical solution of this disclosure, in the process of image decoding (decompression), firstly, through nonlinear transformation of the features to be transformed, the rich details and structural information in the image are better captured, and spatially adjacent feature elements are aggregated to generate more representative, diverse, and less redundant feature representations, providing high-quality input for subsequent weighted fitting. Further, feature fitting is performed based on the weights of each feature element in the aggregated features to highlight key elements, suppress redundant elements, and optimize feature distribution. Compared to decoding by stacking a large number of convolutional neural networks and self-attention transformation networks, this disclosure, through nonlinear aggregation and weighted fitting processing during the decoding process, can remove more redundancy, reduce decoding overhead, and achieve higher decompression efficiency with lower complexity and lightweight design.
[0118] In one implementation, step S901, generating the feature to be transformed based on the code stream, includes: generating quantized features based on the code stream and using the quantized features as the feature to be transformed; or, performing residual upsampling on the fitted features obtained based on the code stream to obtain the feature to be transformed.
[0119] In other words, step S901 can be implemented using a residual upsampling layer. The input to the residual upsampling layer includes quantized features or the fitted features obtained based on the bitstream. Therefore, steps S901, S902, and S903 can all be executed multiple times. Specifically, when step S901 is executed for the first time, the quantized features are directly used as the features to be transformed, and then steps S902 and S903 are executed for the first time. Subsequently, when step S901 is executed again, the fitted features obtained from steps S902 and S903 are input into the residual upsampling layer to obtain the features to be transformed, and then steps S902 and S903 are executed again.
[0120] For example, the residual upsampling layer can restore the resolution of the input features by making the width and height of the output features twice that of the input features through deconvolution, and generate the features to be transformed by performing residual connection on the output of deconvolution and the input features of the residual upsampling layer through residual connection layer.
[0121] Understandably, in this implementation, the residual upsampling layer introduces skip connections during the upsampling process, allowing the input features of the residual upsampling layer to skip certain layers and be directly passed to deeper layers (residual connection layers), thereby alleviating the gradient vanishing problem in the upsampling feature extraction process and improving the feature representation capability of the features to be transformed.
[0122] In a specific application example, multiple upsampling layers and multiple token mixing networks are deployed in the decoding network. Figure 10 shows an application example diagram of the decoding network and decoding process according to an embodiment of this disclosure. The multiple upsampling layers include multiple residual upsampling (ResUp) layers and one deconvolution (ConvUp) layer. The token mixing network is used to perform feature transformation on the input features (features to be transformed) (including the nonlinear aggregation in step S902 and the weighted fitting in step S903).
[0123] As shown in Figure 10, quantization features are obtained based on the bitstream. Then, the quantified features will be... When input into the decoding network, features are extracted three times. Each feature extraction is implemented through a feature extraction module; that is, the first feature extraction module is used for the first feature extraction, the second feature extraction module for the second feature extraction, and the third feature extraction module for the third feature extraction. Each feature extraction module includes one or more feature transformation (TokenMixing) networks and a residual upsampling (ResUp) layer.
[0124] In the first feature extraction, the feature transformation network of the first feature extraction module inputs the quantized features to be transformed. Output Fitting Features The residual upsampling layer of the first feature extraction module is based on fitted features. Generate the features to be transformed required for the second feature extraction. During the second feature extraction, the feature transformation network of the second feature extraction module inputs the features to be transformed. Output Fitting Features The residual upsampling layer of the second feature extraction module is based on fitted features. Generate the features to be transformed required for the second feature extraction. During the third feature extraction, the feature transformation network of the third feature extraction module inputs the features to be transformed. Output Fitting Features The residual upsampling layer of the third feature extraction module is based on fitted features. Features to be transformed required when generating the restored features
[0125] Furthermore, the features to be transformed obtained from the three feature extractions are... The input feature restoration module performs feature restoration to generate restored features for reconstructing the first image (generating the second image). The feature restoration module includes a TokenMixing network and a ConvUp layer. The TokenMixing network is used to transform the features obtained from the three feature extraction steps. Perform nonlinear aggregation (step S902) and weighted fitting (step S903) to generate fitting features. Deconvolution (ConvUp) layers are used for deconvolution to fit features. Restore spatial resolution and generate restored features Based on reduction features The image data is constructed, and then a second image is generated.
[0126] The decoding method shown in Figure 10 can be represented by the following formula:
[0127] It should be noted that the embodiments disclosed herein do not specifically limit the number of feature extraction modules or the number of feature transformation networks in the decoding network.
[0128] Figure 11 illustrates an application example of the residual upsampling (ResUp) layer in an embodiment of this disclosure. The residual upsampling layer includes a deconvolution (ConvUp) layer for processing the input features. Upsampling to generate intermediate features The GELU function is used to process intermediate features. Perform nonlinear activation to generate intermediate features Convolutional (Conv) layers are used for intermediate features Perform convolution operations to generate intermediate features The normalization layer is used to normalize intermediate features. The feature distribution generates intermediate features. Residual connection layer, used for intermediate features and input features Perform residual connections to generate output features.
[0129] The residual upsampling method shown in Figure 11 can be expressed by the following formula:
[0130] It is understandable that when the residual upsampling layer is deployed at different locations in the decoding network, its input features will vary. and output features The meanings differ. For example, when the residual upsampling layer is deployed in the first feature extraction module shown in Figure 10, the input features... and output features These represent the fitted features generated during the first feature extraction and the features to be transformed for the second feature extraction, respectively; when the residual upsampling layer is deployed in the second feature extraction module shown in Figure 10, the input features... and output features These represent the fitted features obtained from the second feature extraction and the features to be transformed for the third feature extraction, respectively; when the residual upsampling layer is deployed in the third feature extraction module shown in Figure 10, the input features... and output features These represent the fitted features obtained during the third feature extraction and the features to be transformed during feature restoration, respectively.
[0131] It should be noted that Figure 11 uses an element-wise addition (+) residual connection method as an example, but the embodiments of this disclosure do not specifically limit the residual connection method. For example, it can also be element-wise subtraction or splicing in the spatial channel dimension. In addition, Figure 11 is an example of a residual upsampling layer and residual upsampling method, but it is not limited to this.
[0132] In one embodiment, step S902 may include: performing nonlinear feature mapping on the feature to be transformed to obtain mapped features; aggregating feature elements with adjacent spatial positions in the mapped features to obtain candidate features; and performing residual connection between the candidate features and the feature to be transformed to generate aggregated features.
[0133] In one implementation, performing nonlinear feature mapping on the feature to be transformed to obtain mapped features may include: expanding the spatial channel dimension of the feature to be transformed through a linear transformation operation to generate expanded features; enhancing the feature elements at some spatial locations in the expanded features based on a nonlinear activation function and a depthwise separable convolutional network to obtain enhanced features; restoring the spatial channel dimension of the enhanced features to the spatial channel dimension of the feature to be transformed, and then performing a residual connection with the feature to be transformed to generate mapped features.
[0134] The above implementation method can be found in the relevant description of step S202 in the encoding process. The nonlinear aggregation module used to implement step S902 can be found in the relevant description of the encoding network, and will not be repeated here.
[0135] In one implementation, step S903 may include: dividing the aggregated feature into a first aggregated sub-feature and a second aggregated sub-feature, wherein the first aggregated sub-feature, the second aggregated sub-feature, and the aggregated feature have the same spatial channel dimension; generating a weighted feature using the first aggregated sub-feature, wherein the weighted feature includes the weights of the feature elements in the aggregated feature; performing weighted processing on the feature elements in the second aggregated sub-feature based on the weighted feature to generate a weighted feature; and fitting the weighted feature and the aggregated feature to generate a fitted feature.
[0136] In one implementation, generating weighted features using first aggregate features includes: inputting feature elements from the first aggregate features into a nonlinear activation function to obtain weighted features.
[0137] In one implementation, weighting the feature elements in the second aggregate feature based on the weight feature to generate a weighted feature includes: performing a feature element-wise multiplication operation between the weight feature and the second aggregate feature to generate the weighted feature.
[0138] The above implementation method can be found in the relevant description of step S203 in the encoding process. The feature gating module used to implement step S903 can be found in the relevant description of the encoding network, and will not be repeated here.
[0139] Example 3
[0140] Corresponding to the application scenario and method provided in Embodiment 1 of this disclosure, this disclosure also provides an image processing apparatus, comprising: a first feature generation module for generating features to be transformed based on a first image, wherein the features to be transformed include feature elements at multiple spatial locations; an aggregation feature generation module for performing nonlinear aggregation on the features to be transformed to obtain aggregated features; a fitting feature generation module for performing weighted fitting on the feature elements in the aggregated features to obtain fitted features; and a bitstream generation module for generating a bitstream corresponding to the first image using the fitted features.
[0141] In one embodiment, the aggregated feature generation module is specifically used to: perform nonlinear feature mapping on the feature to be transformed to obtain mapped features; aggregate feature elements with adjacent spatial positions in the mapped features to obtain candidate features; and perform residual connection between the candidate features and the feature to be transformed to generate the aggregated features.
[0142] In one embodiment, the aggregation feature generation module is further configured to: expand the spatial channel dimension of the feature to be transformed through a linear transformation operation to generate an expanded feature; enhance the feature elements at some spatial locations in the expanded feature based on a nonlinear activation function and a depthwise separable convolutional network to obtain an enhanced feature; restore the spatial channel dimension of the enhanced feature to the spatial channel dimension of the feature to be transformed, and then perform a residual connection with the feature to be transformed to generate the mapped feature.
[0143] In one implementation, the fitting feature generation module is specifically used to: segment the aggregated feature into a first aggregated sub-feature and a second aggregated sub-feature, wherein the first aggregated sub-feature, the second aggregated sub-feature, and the aggregated feature have the same spatial channel dimension; generate a weighted feature using the first aggregated sub-feature, wherein the weighted feature includes the weights of the feature elements in the aggregated feature; perform weighted processing on the feature elements in the second aggregated sub-feature based on the weighted feature to generate a weighted feature; and fit the weighted feature and the aggregated feature to generate the fitting feature.
[0144] In one implementation, the fitting feature generation module is further configured to: input the feature elements in the first aggregated sub-features into a nonlinear activation function to obtain the weighted features.
[0145] In one embodiment, the fitting feature generation module is further configured to: perform element-wise multiplication of the weighted features with the second aggregated features to generate the weighted features.
[0146] In one implementation, the first feature generation module is specifically used to: perform residual downsampling on the first image or the fitted features obtained based on the first image to obtain the feature to be transformed.
[0147] Example 4
[0148] Corresponding to the application scenario and method provided in Embodiment 2 of this disclosure, this disclosure also provides an image processing apparatus, including: a second feature generation module for generating features to be transformed based on a bitstream, wherein the bitstream is generated based on a first image, and the features to be transformed include feature elements at multiple spatial locations; an aggregation feature generation module for performing nonlinear aggregation on the features to be transformed to obtain aggregated features; a fitting feature generation module for performing weighted fitting on the feature elements in the aggregated features to obtain fitted features; and a second image generation module for reconstructing the first image using the fitted features to obtain a second image.
[0149] In one embodiment, the aggregated feature generation module is specifically used to: perform nonlinear feature mapping on the feature to be transformed to obtain mapped features; aggregate feature elements with adjacent spatial positions in the mapped features to obtain candidate features; and perform residual connection between the candidate features and the feature to be transformed to generate the aggregated features.
[0150] In one implementation, the fitting feature generation module is specifically used to: segment the aggregated feature into a first aggregated sub-feature and a second aggregated sub-feature, wherein the first aggregated sub-feature, the second aggregated sub-feature, and the aggregated feature have the same spatial channel dimension; generate a weighted feature using the first aggregated sub-feature, wherein the weighted feature includes the weights of the feature elements in the aggregated feature; perform weighted processing on the feature elements in the second aggregated sub-feature based on the weighted feature to generate a weighted feature; and fit the weighted feature and the aggregated feature to generate the fitting feature.
[0151] In one implementation, the second feature generation module is specifically used to: generate quantized features based on the bitstream and use the quantized features as the feature to be transformed; or, perform residual upsampling on the fitted features obtained based on the bitstream to obtain the feature to be transformed.
[0152] The functions of each module in the apparatus of this embodiment can be found in the corresponding description in the above method, and they have corresponding beneficial effects, which will not be repeated here.
[0153] Figure 12 is a block diagram of an electronic device used to implement embodiments of the present disclosure. As shown in Figure 12, the electronic device includes a memory 1201 and a processor 1202. The memory 1201 stores a computer program that can run on the processor 1202. When the processor 1202 executes the computer program, it implements the methods in the above embodiments. The number of memories 1201 and processors 1202 can be one or more. In a specific implementation, the electronic device may also include a communication interface 1203 for communicating with external devices and performing data exchange and transmission.
[0154] In practical implementation, if the memory 1201, processor 1202, and communication interface 1203 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in Figure 12, but this does not indicate that there is only one bus or one type of bus.
[0155] Optionally, in a specific implementation, if the memory 1201, processor 1202, and communication interface 1203 are integrated on a single chip, then the memory 1201, processor 1202, and communication interface 1203 can communicate with each other through an internal interface.
[0156] This disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods provided in this disclosure.
[0157] This disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the methods provided in this disclosure.
[0158] This disclosure also provides a chip including a processor for calling and executing instructions stored in a memory, causing a communication device on which the chip is installed to perform the methods provided in this disclosure.
[0159] This disclosure also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in the application embodiment.
[0160] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.
[0161] Further, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).
[0162] In the above embodiments, implementation can be achieved, in whole or in part, by software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this disclosure are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.
[0163] In the description of this disclosure, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this disclosure, as well as the features of those different embodiments or examples.
[0164] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means two or more, unless otherwise explicitly specified.
[0165] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.
[0166] The logic and / or steps described in the flowchart or otherwise herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).
[0167] It should be understood that various parts of this disclosure can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.
[0168] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.
[0169] The above description is merely an exemplary embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this disclosure, and these should all be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.
Claims
1. An image processing method, comprising: The transformation features are generated based on the first image, wherein the transformation features include feature elements at multiple spatial locations; The features to be transformed are nonlinearly aggregated to obtain aggregated features; The feature elements in the aggregated features are weighted and fitted to obtain the fitted features; The fitted features are used to generate a bitstream corresponding to the first image.
2. The method according to claim 1, wherein, The nonlinear aggregation of the features to be transformed to obtain aggregated features includes: The features to be transformed are subjected to nonlinear feature mapping to obtain the mapped features; By aggregating feature elements with adjacent spatial positions in the mapping features, candidate features are obtained; The candidate feature and the feature to be transformed are residually joined to generate the aggregated feature.
3. The method according to claim 2, wherein, The process of performing nonlinear feature mapping on the features to be transformed to obtain mapped features includes: The spatial channel dimension of the feature to be transformed is expanded by a linear transformation operation to generate an expanded feature. Based on a nonlinear activation function and a depthwise separable convolutional network, feature enhancement is performed on feature elements at some spatial locations in the extended features to obtain enhanced features. After restoring the spatial channel dimension of the enhanced feature to the spatial channel dimension of the feature to be transformed, a residual connection is performed with the feature to be transformed to generate the mapped feature.
4. The method according to any one of claims 1 to 3, wherein, The weighted fitting of the feature elements in the aggregated features to obtain the fitted features includes: The aggregated feature is divided into a first aggregated sub-feature and a second aggregated sub-feature, wherein the first aggregated sub-feature, the second aggregated feature, and the aggregated feature have the same spatial channel dimension; A weighted feature is generated using the first aggregated feature, wherein the weighted feature includes the weights of the feature elements in the aggregated feature; Based on the weighted features, the feature elements in the second aggregate feature are weighted to generate a weighted feature; The weighted features and the aggregated features are joined by residual connections to generate the fitted features.
5. The method according to claim 4, wherein, The step of generating weighted features using the first aggregated feature includes: The feature elements in the first aggregate feature are input into a nonlinear activation function to obtain the weighted feature.
6. The method according to claim 4, wherein, The step of weighting the feature elements in the second aggregated sub-feature based on the weighted features to generate weighted features includes: The weighted feature is generated by performing element-wise multiplication of the weighted feature with the second aggregate feature.
7. The method according to any one of claims 1 to 3, wherein, The process of generating the features to be transformed based on the first image includes: The first image or the fitted features obtained based on the first image are subjected to residual downsampling to obtain the features to be transformed.
8. An image processing method, comprising: The feature to be transformed is generated based on the bitstream, wherein the bitstream is generated based on the first image, and the feature to be transformed includes feature elements at multiple spatial locations; The features to be transformed are nonlinearly aggregated to obtain aggregated features; The feature elements in the aggregated features are weighted and fitted to obtain the fitted features; The first image is reconstructed using the fitted features to obtain the second image.
9. The method according to claim 8, wherein, The nonlinear aggregation of the features to be transformed to obtain aggregated features includes: The features to be transformed are subjected to nonlinear feature mapping to obtain the mapped features; By aggregating feature elements with adjacent spatial positions in the mapping features, candidate features are obtained; The candidate feature and the feature to be transformed are residually joined to generate the aggregated feature.
10. The method according to claim 8 or 9, wherein, The weighted fitting of the feature elements in the aggregated features to obtain the fitted features includes: The aggregated feature is divided into a first aggregated sub-feature and a second aggregated sub-feature, wherein the first aggregated sub-feature, the second aggregated feature, and the aggregated feature have the same spatial channel dimension; A weighted feature is generated using the first aggregated feature, wherein the weighted feature includes the weights of the feature elements in the aggregated feature; Based on the weighted features, the feature elements in the second aggregate feature are weighted to generate a weighted feature; The weighted features and the aggregated features are fitted together to generate the fitted features.
11. The method according to claim 8 or 9, wherein, The generation of features to be transformed based on the code stream includes: Quantization features are generated based on the bitstream, and these quantization features are used as the features to be transformed; or... The fitted features obtained based on the bitstream are subjected to residual upsampling to obtain the features to be transformed.
12. An image processing method, comprising: The transformation features are generated based on the first image, wherein the transformation features include feature elements at multiple spatial locations; The features to be transformed are nonlinearly aggregated to obtain aggregated features; The feature elements in the aggregated features are weighted and fitted to obtain the fitted features; The fitted features are used to generate a bitstream corresponding to the first image; The first image is reconstructed based on the bitstream to obtain the second image.
13. The method according to claim 12, wherein, The nonlinear aggregation of the features to be transformed to obtain aggregated features includes: The features to be transformed are subjected to nonlinear feature mapping to obtain the mapped features; By aggregating feature elements with adjacent spatial positions in the mapping features, candidate features are obtained; The candidate feature and the feature to be transformed are residually joined to generate the aggregated feature.
14. The method according to claim 12 or 13, wherein, The weighted fitting of the feature elements in the aggregated features to obtain the fitted features includes: The aggregated feature is divided into a first aggregated sub-feature and a second aggregated sub-feature, wherein the first aggregated sub-feature, the second aggregated feature, and the aggregated feature have the same spatial channel dimension; A weighted feature is generated using the first aggregated feature, wherein the weighted feature includes the weights of the feature elements in the aggregated feature; Based on the weighted features, the feature elements in the second aggregate feature are weighted to generate a weighted feature; The weighted features and the aggregated features are joined by residual connections to generate the fitted features.
15. An image processing apparatus, comprising: The first feature generation module is used to generate features to be transformed based on the first image, wherein the features to be transformed include feature elements at multiple spatial locations; The aggregation feature generation module is used to perform nonlinear aggregation on the features to be transformed to obtain aggregated features; The fitting feature generation module is used to perform weighted fitting on the feature elements in the aggregated features to obtain the fitting features; The bitstream generation module is used to generate a bitstream corresponding to the first image using the fitted features.
16. An image processing apparatus, comprising: The second feature generation module generates features to be transformed based on the bitstream, wherein the bitstream is generated based on the first image, and the features to be transformed include feature elements at multiple spatial locations. The aggregation feature generation module is used to perform nonlinear aggregation on the features to be transformed to obtain aggregated features; The fitting feature generation module is used to perform weighted fitting on the feature elements in the aggregated features to obtain the fitting features; The second image generation module is used to reconstruct the first image using the fitted features to obtain the second image.
17. An image processing system, comprising: The encoding end is used to perform the method according to any one of claims 1 to 7 to generate a bitstream; The decoding end is used to obtain the bitstream from the encoding end and execute the method according to any one of claims 8 to 11.
18. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method of any one of claims 1 to 14.
19. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method of any one of claims 1 to 14.
20. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 14.
