Picture processing method and apparatus, device, and storage medium
By fusing features in the encoder and feature memory network and iteratively updating parameters in the online backpropagation network, the scene compatibility and efficiency issues of multi-task models are solved, achieving more efficient image processing and higher precision parameter updates.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- BEIJING XIAOYU INTELLISYS CO LTD
- Filing Date
- 2025-12-23
- Publication Date
- 2026-07-02
Smart Images
Figure CN2025144941_02072026_PF_FP_ABST
Abstract
Description
Image processing methods, apparatus, equipment and storage media
[0001] Cross-reference of related applications
[0002] This disclosure is based on and claims priority to Chinese Patent Application No. 202411920700.2, filed on December 25, 2024, the entire contents of which are incorporated herein by reference. Technical Field
[0003] This disclosure relates to the field of artificial intelligence technology, and in particular to an image processing method, apparatus, device, and storage medium. Background Technology
[0004] Task awareness refers to the ability of a system to dynamically adjust its operational strategies and parameters based on current task requirements and environmental changes during task execution, adapting to different task scenarios and objectives. This capability is particularly important in fields such as autonomous driving, robot control, and object detection.
[0005] In scenarios such as robotics, autonomous driving, and drones, it is necessary to complete perception tasks such as real-time new perspective synthesis, monocular estimation, mapping, localization, object classification, recognition, segmentation, tracking, and feature point extraction.
[0006] Currently, multi-task models can only perform a limited number of perception tasks, have low scene compatibility, and low image processing efficiency. Summary of the Invention
[0007] This disclosure provides an image processing method, apparatus, device, and storage medium to at least solve the problems of low compatibility and low image processing efficiency in existing multi-task model scenarios.
[0008] The technical solution disclosed herein is as follows:
[0009] This disclosure provides an image processing method, including:
[0010] The original image is input into the encoder for feature encoding to obtain the key features of the original image;
[0011] The key features of the original image and the key features of the historical image are input into the feature memory network for feature fusion to obtain feature fusion information, which includes: Gaussian parameters, point cloud information and depth estimation information;
[0012] The Gaussian parameters, point cloud information, and depth estimation information are input into the online backpropagation network for backpropagation iteration to obtain the updated Gaussian parameters, updated point cloud information, and updated depth estimation information.
[0013] The updated Gaussian parameters are input into the predictor corresponding to each task type to obtain the prediction results. The accuracy of the updated Gaussian parameters is higher than that of the original Gaussian parameters.
[0014] Optionally, the feature fusion information further includes: object recognition information, object segmentation information, object tracking information, and feature point extraction information; the task types include: segmentation, detection, recognition, monocular estimation, mapping, localization, new perspective synthesis, and feature point extraction.
[0015] Optionally, the encoder includes: a tile embedding layer, a position encoding layer, and multiple encoder layers, wherein the multiple encoder layers are connected serially; the step of inputting the original image into the encoder for feature encoding to obtain key features of the original image includes:
[0016] Inside the encoder, the original image is input into the tile embedding layer to obtain the tile embedding;
[0017] The image patch embedding is input into the location encoding layer to obtain the location-encoded image patch embedding;
[0018] The location-encoded tiles are embedded into the encoder layer to obtain the key features of the original image.
[0019] Optionally, each encoder layer includes: a multi-head self-attention mechanism layer, a first residual connection and layer normalization layer, a feedforward neural network layer, and a second residual connection and layer normalization layer; the step of embedding the position-encoded image patch into the encoder layer to obtain the key features of the original image includes:
[0020] For the target encoder layer, the position-encoded patches are embedded into the multi-head self-attention mechanism layer of the target encoder layer to obtain self-attention output features; wherein, the target encoder layer is any one of the multiple encoder layers;
[0021] The self-attention output features and the position-encoded patches are embedded into the first residual connection and layer normalization layer of the target encoder layer to obtain the first normalized features;
[0022] The first normalized feature is input into the feedforward neural network layer of the target encoder layer to obtain the feedforward network output feature.
[0023] The feedforward network output features and the first normalized features are input into the second residual connection and layer normalization layer of the target encoder layer to obtain the second normalized features.
[0024] Optionally, the feature memory network includes: multiple serially connected base block modules and a head module. The step of inputting the key features of the original image and the key features of historical images into the feature memory network for feature fusion to obtain feature fusion information includes:
[0025] The key features of the original image and the key features of the historical image are input into multiple base block modules to obtain the initial fusion features;
[0026] The initial fusion features are input into the head module to obtain the feature fusion information.
[0027] Optionally, the step of inputting the Gaussian parameters, the point cloud information, and the depth estimation information into an online backpropagation network for backpropagation iteration to obtain updated Gaussian parameters, updated point cloud information, and updated depth estimation information includes:
[0028] The Gaussian parameters, the point cloud information, and the depth estimation information are input into the online backpropagation network to render the original image, thereby obtaining the rendered image corresponding to the original image.
[0029] Determine the loss function based on the rendered image and the original image;
[0030] Backpropagation is performed based on the loss function to update the Gaussian parameters, the point cloud information, and the depth estimation information, resulting in updated Gaussian parameters, updated point cloud information, and updated depth estimation information.
[0031] This disclosure also provides an image processing apparatus, including:
[0032] The encoding module is used to input the original image into the encoder for feature encoding to obtain the key features of the original image;
[0033] The fusion module is used to input the key features of the original image and the key features of the historical image into the feature memory network for feature fusion to obtain feature fusion information, wherein the feature fusion information includes: Gaussian parameters, point cloud information and depth estimation information;
[0034] The rendering module is used to input the Gaussian parameters, the point cloud information, and the depth estimation information into the online backpropagation network for backpropagation to obtain updated Gaussian parameters, updated point cloud information, and updated depth estimation information.
[0035] The prediction module is used to input the updated Gaussian parameters into the predictor corresponding to each task type to obtain the prediction result, wherein the accuracy of the updated Gaussian parameters is higher than that of the original Gaussian parameters.
[0036] This disclosure also provides an electronic device, including:
[0037] processor;
[0038] Memory used to store the processor's executable instructions;
[0039] The processor is configured to execute the instructions to implement the steps in the above method.
[0040] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.
[0041] This disclosure also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the method described above.
[0042] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:
[0043] In some embodiments of this disclosure, the original image is input into an encoder for feature encoding to obtain key features of the original image; the key features of the original image and key features of historical images are input into a feature memory network for feature fusion to obtain feature fusion information, wherein the feature fusion information includes: Gaussian parameters, point cloud information and depth estimation information. This disclosure predicts Gaussian parameters through a network; inputting high-quality Gaussian parameters, point cloud information and depth estimation information into an online backpropagation network can reduce the number of backpropagations, improve image processing efficiency, and obtain more accurate updated Gaussian parameters by using the backpropagation method, which can be applied to multiple task models and improve the model's compatibility with various task scenarios.
[0044] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description
[0045] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.
[0046] Figure 1 is a schematic flowchart of an image processing method provided by an exemplary embodiment of this disclosure;
[0047] Figure 2 is a schematic diagram of the structure of an image processing system provided by an exemplary embodiment of the present disclosure;
[0048] Figure 3 is a schematic diagram of the structure of an online backpropagation network provided by an exemplary embodiment of this disclosure;
[0049] Figure 4 is a schematic diagram of the structure of an image processing apparatus provided in an exemplary embodiment of the present disclosure;
[0050] Figure 5 is a schematic diagram of the structure of an electronic device provided by an exemplary embodiment of this disclosure. Detailed Implementation
[0051] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.
[0052] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure.
[0053] It should be noted that the user information involved in this disclosure includes, but is not limited to, user device information and user personal information; the collection, storage, use, processing, transmission, provision and disclosure of user information in this disclosure all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0054] The following describes the existing solutions for perception tasks.
[0055] One approach involves mapping and feature point extraction. The input consists of multiple images. First, feature points are extracted from each image. Then, through feature point matching, the depth of the feature points and the pose of each image are calculated to complete mapping and localization. However, this approach is not applicable to many scenarios and has low versatility.
[0056] Second, a monocular estimation scheme. The input is a single image, and the network structure includes an encoder and a decoder. The output is a depth map, where the encoder is a feature extraction module and the decoder is a decoding module used to predict depth based on features. This scheme is not applicable to many scenarios and has low universality.
[0057] Third, the new perspective synthesis scheme. The input consists of multiple images and an initial sparse point cloud. The Gaussian parameters are then initialized to a preset value, for example, the opcity parameter is initialized to 0.1. The parameters are then updated through backpropagation. Each backpropagation iteration updates the parameters to make them "more accurate." Typically, 30,000 backpropagation iterations are needed to obtain sufficiently accurate parameters. Based on these parameters, the new perspective image can be rendered using a Gaussian rendering engine. Image rendering takes a long time, typically 10 minutes to 1 hour.
[0058] To address the aforementioned technical issues, in some embodiments of this disclosure, the original image is input into an encoder for feature encoding to obtain key features of the original image; the key features of the original image and key features of historical images are input into a feature memory network for feature fusion to obtain feature fusion information, wherein the feature fusion information includes: Gaussian parameters, point cloud information, and depth estimation information. This disclosure predicts Gaussian parameters through a network; inputting high-quality Gaussian parameters, point cloud information, and depth estimation information into an online backpropagation network can reduce the number of backpropagations, improve image processing efficiency, and obtain more accurate updated Gaussian parameters using the backpropagation method, which can be applied to various task models and improve the model's compatibility with various task scenarios.
[0059] The technical solutions provided by the embodiments of this disclosure are described in detail below with reference to the accompanying drawings.
[0060] Figure 1 is a flowchart illustrating an image processing method provided by an exemplary embodiment of this disclosure. As shown in Figure 1, the method includes:
[0061] S101: Input the original image into the encoder for feature encoding to obtain the key features of the original image;
[0062] S102: Input the key features of the original image and the key features of the historical image into the feature memory network for feature fusion to obtain feature fusion information, which includes: Gaussian parameters, point cloud information and depth estimation information;
[0063] S103: Input the Gaussian parameters, point cloud information and depth estimation information into the online backpropagation network for backpropagation iteration to obtain the updated Gaussian parameters, updated point cloud information and updated depth estimation information;
[0064] S104: Input the updated Gaussian parameters into the predictor corresponding to each task type to obtain the prediction results. The accuracy of the updated Gaussian parameters is higher than that of the original Gaussian parameters.
[0065] In this embodiment of the disclosure, the subject executing the above method can be a terminal device or a server.
[0066] The terminal device includes, but is not limited to, mobile stations (MS), mobile terminals, mobile phones, handsets, and portable equipment. This terminal device can communicate with one or more core networks via a radio access network (RAN). For example, the terminal device can be a mobile phone (or "cellular" phone), a computer with wireless communication capabilities, a computer with wireless transceiver capabilities, a virtual reality (VR) terminal device, an AR terminal device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical care, a wireless terminal in a smart grid, a wireless terminal in transportation safety, a wireless terminal in a smart city, a wireless terminal in a smart home, etc. The operating systems installed on the terminal device include, but are not limited to, iOS, Android, Windows, Linux, and Mac OS. In different networks, a terminal may be called by different names, such as: user equipment, mobile station, user unit, station, cellular phone, personal digital assistant, wireless modem, wireless communication device, handheld device, laptop computer, cordless phone, wireless local loop station, television, etc. For ease of description, it is simply referred to as a terminal device in this embodiment of the disclosure.
[0067] In this embodiment, the implementation form of the server is not limited. For example, the server can be a conventional server, a cloud server, a cloud host, a virtual center, or other server equipment. The server mainly consists of a processor, hard disk, memory, system bus, and other common computer architecture types.
[0068] In some embodiments of this disclosure, the original image is input into an encoder for feature encoding to obtain key features of the original image; the key features of the original image and key features of historical images are input into a feature memory network for feature fusion to obtain feature fusion information, wherein the feature fusion information includes: Gaussian parameters, point cloud information and depth estimation information. This disclosure predicts Gaussian parameters through a network; inputting high-quality Gaussian parameters, point cloud information and depth estimation information into an online backpropagation network can reduce the number of backpropagations, improve image processing efficiency, and obtain more accurate updated Gaussian parameters by using the backpropagation method, which can be applied to multiple task models and improve the model's compatibility with various task scenarios.
[0069] In this embodiment, the original image is input into an encoder for feature encoding to obtain key features of the original image; the key features of the original image and key features of historical images are input into a feature memory network for feature fusion to obtain feature fusion information, wherein the feature fusion information includes: Gaussian parameters, point cloud information and depth estimation information. This disclosure predicts Gaussian parameters through a network; the high-quality Gaussian parameters, point cloud information and depth estimation information are input into an online backpropagation network to render the original image to obtain the rendered image corresponding to the original image. This can reduce the number of backpropagations, improve image processing efficiency, and improve the compatibility of the model with various scenarios by using the backpropagation method to update parameters.
[0070] It should be noted that Gaussian parameters include: mean vector, covariance matrix, weights, magnitude, and scale parameters.
[0071] The mean vector, denoted by μ, defines the center position of the Gaussian distribution. In three-dimensional space, the mean vector is usually represented as μ = (μ_x, μ_y, μ_z), where μ_x, μ_y, and μ_z are the center coordinates of the Gaussian distribution in the x, y, and z directions, respectively. The covariance matrix, denoted by Σ, defines the shape and orientation of the Gaussian distribution. In three-dimensional space, the covariance matrix is a 3x3 symmetric positive definite matrix, represented as: Σ = [σ_xx σ_xy σ_xz] [σ_yx σ_yy σ_yz] [σ_zx σ_zy σ_zz]
[0072] Here, σ_xx, σ_yy, and σ_zz are the variances in the x, y, and z directions, while σ_xy, σ_xz, etc., are the covariances in different directions. The covariance matrix determines the expansion and rotation of the Gaussian distribution.
[0073] Here, the weight, denoted by w, determines the influence or importance of the Gaussian distribution. During the rendering process, different Gaussian distributions may have different weights to represent their contribution to the final image.
[0074] The amplitude, denoted by A, determines the height or intensity of the Gaussian distribution. It is a scalar value that controls the maximum value of the Gaussian distribution at its center.
[0075] Among them, the scale parameters, denoted as s_x, s_y, s_z, control the extent of the Gaussian distribution in each direction and can usually be derived from the covariance matrix.
[0076] In practical applications, these parameters together determine the shape, position, and influence of the Gaussian distribution, thus forming a complex distribution in three-dimensional space for representing and processing three-dimensional data points.
[0077] In this embodiment of the disclosure, the encoder includes: a tile embedding layer, a position encoding layer, and multiple encoder layers, which are serially connected to each other.
[0078] In some embodiments of this disclosure, the original image is input into an encoder for feature encoding to obtain key features of the original image. Inside the encoder, the original image is input into a tile embedding layer to obtain tile embeddings (also called tile embedding vectors); the tile embeddings are input into a position encoding layer to obtain position-encoded tile embeddings; and the position-encoded tile embeddings are input into the encoder layer to obtain key features of the original image. It should be noted that the encoder in these embodiments is used to extract features from the original image; this disclosure does not limit the type of encoder and can be adjusted according to actual circumstances.
[0079] In some embodiments of this disclosure, the original image is input into an encoder for feature encoding to obtain key features of the original image. This includes: within the encoder, the original image is input into a tile embedding layer, where the image is divided into tiles of fixed size, each tile is flattened into a one-dimensional vector, and the flattened tiles are mapped to the embedding space through linear projection to obtain tile embeddings; the tile embeddings are input into the position encoding layer, where learnable position encodings are added to each tile embedding to preserve position information, resulting in position-encoded tile embeddings; the position-encoded tile embeddings are input into the encoder layer, and multiple encoder layers are stacked sequentially to obtain the key features of the original image.
[0080] For example, the encoder is a vit encoder. The original image frame t is input into the tile embedding layer, which divides the image into tiles of fixed size. Each tile is flattened into a one-dimensional vector, and the flattened tiles are mapped to the embedding space through linear projection to obtain the tile embedding. The original image has a shape of H×W×C, and the tile embedding has a shape of N×D, where N is the number of tiles and D is the embedding dimension. frame t represents the image acquired at time t, and frame t-1 represents the image acquired at time t-1, i.e., the image acquired before time t. The tile embeddings are then input into the positional encoding layer, where a learnable positional code is added to each tile embedding to preserve positional information, resulting in positionally encoded tile embeddings with a shape of N×D. The positionally encoded tile embeddings are then input into the encoder layer. Multiple encoder layers are stacked sequentially to obtain the key features of the original image, which also have a shape of N×D.
[0081] Each encoder layer includes: a multi-head self-attention mechanism layer, a first residual connection and layer normalization layer, a feedforward neural network layer, and a second residual connection and layer normalization layer.
[0082] In some embodiments of this disclosure, position-encoded patches are embedded into the input encoder layer to obtain key features of the original image. One possible approach is to embed position-encoded patches into a multi-head self-attention mechanism layer of the target encoder layer to obtain self-attention output features; wherein the target encoder layer is any one of multiple encoder layers; the self-attention output features and position-encoded patches are embedded into a first residual connection and layer normalization layer of the target encoder layer to obtain first normalized features; the first normalized features are input into a feedforward neural network layer of the target encoder layer to obtain feedforward network output features; the feedforward network output features and the first normalized features are input into a second residual connection and layer normalization layer of the target encoder layer to obtain second normalized features.
[0083] For example, position-encoded image patches are embedded into the input multi-head self-attention mechanism layer to generate a query (Q), key (K), and value (V) matrix; attention scores are calculated and weighted summation is performed; multiple attention heads compute in parallel, and the results are concatenated and then linearly transformed to obtain the self-attention output features; the shape of the self-attention output features is N×D. The self-attention output features and position-encoded image patches are embedded into the first residual connection and layer normalization layer of the input target encoder layer. The self-attention output features and position-encoded image patch embeddings are added together, and a normalization operation is performed using an application layer to obtain the first normalized features; the shape of the first normalized features is N×D. The first normalized features are input into the feedforward neural network layer of the target encoder layer, two fully connected layers with an activation function in between, to obtain the feedforward network output features; the shape of the feedforward network output features is N×D. The feedforward network output features and the first normalized features are input into the second residual connection and the layer normalization layer of the target encoder layer. The feedforward network output features and the first normalized features are added together, and layer normalization is applied to obtain the second normalized features. When the target encoder layer is the last encoder layer in the encoder layer, the second normalized features are the key features of the original image.
[0084] In this embodiment, the Feature Memory (FM) network includes multiple serially connected base block modules and a head module. Each base block module consists of a traditional self-attention and cross-attention structure. The head module consists of multiple 3*3 convolutional layers. The input of the first layer's convolutional kernel is n, and the output is 4*n. The input of the first layer's convolutional kernel is 4*n, and the output is 7+3+m+1+20, where 7 represents the Gaussian parameter, 3 represents the point cloud coordinates (depth estimation can be directly calculated from the point cloud coordinates), m represents the number of object segmentation categories, 1 represents the object tracking ID, and 20 represents the dimension of the feature points.
[0085] In some embodiments of this disclosure, key features of the original image and key features of historical images are input into a feature memory network for feature fusion to obtain feature fusion information. One possible approach is to input key features of the original image and key features of historical images into multiple base block modules to obtain initial fused features; the initial fused features are then input into a head module to obtain feature fusion information. It should be noted that frame t and feature memory t-1 are input together into the FM network; where frame t features are the original image key features feature t; when t is 1, frame 1 features, i.e., feature 1, are used as feature memory t-1. Feature memory t-1 (the feature memory of the FM network at time t-1) represents a collection of key features of historical images. Through the feature memory network, feature memory t (the feature memory of the FM network at time t) can be continuously updated, helping feature t to fuse historical features and improve the representational power of the features.
[0086] It should be noted that the feature fusion information includes, but is not limited to: Gaussian parameters, point cloud information, depth estimation information, object recognition information, object segmentation information, object tracking information, and feature point extraction information.
[0087] Figure 2 is a schematic diagram of the structure of an image processing system provided by an exemplary embodiment of this disclosure. As shown in Figure 2, the original image frame t is input into the encoder to obtain the original image key features feature t; the original image key features feature t and historical image key features feature t-1...feature tn are input into a feature memory network for feature fusion to obtain Gaussian parameters, point cloud information, depth estimation information, object recognition information, object segmentation information, object tracking information, and feature point extraction information. Among them, the historical image key features are the key features of images acquired before the original image acquisition time t, and are obtained in the same way as the original image key features. For example, feature t-1 is obtained by inputting the image acquired at time t-1 before time t, i.e., frame t-1, into the encoder, and feature tn is obtained by inputting the image acquired at time tn before time t, i.e., frame tn, into the encoder.
[0088] In some embodiments of this disclosure, Gaussian parameters, point cloud information, and depth estimation information are input into an online backpropagation network for backpropagation iteration to obtain updated Gaussian parameters, updated point cloud information, and updated depth estimation information. One possible approach is to input the Gaussian parameters, point cloud information, and depth estimation information into the online backpropagation network, render the original image to obtain a rendered image corresponding to the original image; determine a loss function based on the rendered image and the original image; and perform backpropagation based on the loss function to update the Gaussian parameters, point cloud information, and depth estimation information to obtain updated Gaussian parameters, updated point cloud information, and updated depth estimation information.
[0089] For example, Figure 3 is a schematic diagram of an online backpropagation network provided by an exemplary embodiment of this disclosure. As shown in Figure 3, Gaussian parameters, point cloud information, and depth estimation information are input into the online backpropagation network. Based on the predicted Gaussian parameters, a rendered image corresponding to the original image is rendered. Based on the rendered image and the original image, a loss function is determined. The loss function can be an L2 loss rendering loss function. The L2 loss is applied to each Gaussian parameter for 10 backpropagations to obtain updated Gaussian parameters, updated point cloud information, and updated depth estimation information.
[0090] In some embodiments of this disclosure, the updated Gaussian parameters are input into the predictor corresponding to each task type to obtain prediction results. The accuracy of the updated Gaussian parameters is higher than that of the standard Gaussian parameters. The task types include: segmentation, detection, recognition, monocular estimation, mapping, localization, novel perspective synthesis, and feature point extraction. This disclosure uses backpropagation to obtain more accurate updated Gaussian parameters, making it applicable to multiple task models and improving the model's compatibility with various task scenarios.
[0091] Figure 4 is a schematic diagram of the structure of an image processing apparatus 40 provided in an exemplary embodiment of the present disclosure. As shown in Figure 4, the image processing apparatus 40 includes: an encoding module 41, a fusion module 42, a rendering module 43, and a prediction module 44.
[0092] Among them, the encoding module 41 is used to input the original image into the encoder for feature encoding to obtain the key features of the original image;
[0093] The fusion module 42 is used to input the key features of the original image and the key features of the historical image into the feature memory network for feature fusion to obtain feature fusion information, which includes: Gaussian parameters, point cloud information and depth estimation information;
[0094] The rendering module 43 is used to input Gaussian parameters, point cloud information and depth estimation information into the online backpropagation network for backpropagation to obtain updated Gaussian parameters, updated point cloud information and updated depth estimation information.
[0095] The prediction module 44 is used to input the updated Gaussian parameters into the predictor corresponding to each task type to obtain the prediction result. The accuracy of the updated Gaussian parameters is higher than that of the original Gaussian parameters.
[0096] Optionally, the feature fusion information also includes: object recognition information, object segmentation information, object tracking information, and feature point extraction information; the task types include: segmentation, detection, recognition, monocular estimation, mapping, localization, new perspective synthesis, and feature point extraction.
[0097] Optionally, the encoder includes: a tile embedding layer, a position encoding layer, and multiple encoder layers, which are serially connected; the encoding module 41, when inputting the original image into the encoder for feature encoding to obtain the key features of the original image, is used for:
[0098] Inside the encoder, the original image is input into the tile embedding layer to obtain the tile embedding;
[0099] The tile embedding is input into the position encoding layer to obtain the position encoded tile embedding;
[0100] The location-encoded patches are embedded into the input encoder layer to obtain the key features of the original image.
[0101] Optionally, each encoder layer includes: a multi-head self-attention mechanism layer, a first residual connection and layer normalization layer, a feedforward neural network layer and a second residual connection and layer normalization layer; the encoding module 41, when embedding the position-encoded patches into the input encoder layer to obtain the key features of the original image, is used for:
[0102] For the target encoder layer, the position-encoded patches are embedded into the multi-head self-attention mechanism layer of the input target encoder layer to obtain the self-attention output features; where the target encoder layer is any one of the multiple encoder layers.
[0103] The self-attention output features and the position-encoded patches are embedded into the first residual connection and layer normalization layer of the input target encoder layer to obtain the first normalized features;
[0104] The first normalized feature is input into the feedforward neural network layer of the target encoder layer to obtain the feedforward network output feature;
[0105] The output features of the feedforward network and the first normalized features are input into the second residual connection and the layer normalization layer of the target encoder layer to obtain the second normalized features.
[0106] Optionally, the feature memory network includes: multiple serially connected base block modules and a head module; the fusion module 42, when inputting key features of the original image and key features of historical images into the feature memory network for feature fusion to obtain feature fusion information, is used for:
[0107] The key features of the original image and the key features of the historical image are input into multiple basic block modules to obtain the initial fused features;
[0108] The initial fused features are input into the head module to obtain feature fusion information.
[0109] Optionally, when the rendering module 43 inputs the Gaussian parameters, point cloud information, and depth estimation information into the online backpropagation network for backpropagation iteration to obtain updated Gaussian parameters, updated point cloud information, and updated depth estimation information, it is used for:
[0110] The Gaussian parameters, point cloud information, and depth estimation information are input into the online backpropagation network to render the original image, resulting in a rendered image corresponding to the original image.
[0111] Determine the loss function based on the rendered image and the original image;
[0112] Backpropagation is performed based on the loss function, and the Gaussian parameters, point cloud information, and depth estimation information are iterated backward to obtain the updated Gaussian parameters, updated point cloud information, and updated depth estimation information.
[0113] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.
[0114] Figure 5 is a schematic diagram of the structure of an electronic device provided by an exemplary embodiment of the present disclosure. As shown in Figure 5, the electronic device includes a memory 51 and a processor 52. Additionally, the electronic device also includes a power supply component 53 and a communication component 54.
[0115] Memory 51 is used to store computer programs and can be configured to store various other data to support operation on the electronic device. Examples of this data include instructions for any application or method used to operate on the electronic device.
[0116] The memory 51 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.
[0117] Communication component 54 is used for data transmission with other devices.
[0118] The processor 52 executes computer instructions stored in the memory 51 to: input the original image into the encoder for feature encoding to obtain key features of the original image; input the key features of the original image and key features of historical images into the feature memory network for feature fusion to obtain feature fusion information, wherein the feature fusion information includes: Gaussian parameters, point cloud information and depth estimation information; input the Gaussian parameters, point cloud information and depth estimation information into the online backpropagation network for backpropagation iteration to obtain updated Gaussian parameters, updated point cloud information and updated depth estimation information; input the updated Gaussian parameters into the predictor corresponding to each task type to obtain prediction results, wherein the accuracy of the updated Gaussian parameters is higher than that of the original Gaussian parameters.
[0119] Optionally, the feature fusion information also includes: object recognition information, object segmentation information, object tracking information, and feature point extraction information; the task types include: segmentation, detection, recognition, monocular estimation, mapping, localization, new perspective synthesis, and feature point extraction.
[0120] Optionally, the encoder includes: a tile embedding layer, a position encoding layer, and multiple encoder layers, which are serially connected; when the processor 52 inputs the original image into the encoder for feature encoding to obtain the key features of the original image, it is used for:
[0121] Inside the encoder, the original image is input into the tile embedding layer to obtain the tile embedding;
[0122] The tile embedding is input into the position encoding layer to obtain the position encoded tile embedding;
[0123] The location-encoded patches are embedded into the input encoder layer to obtain the key features of the original image.
[0124] Optionally, each encoder layer includes: a multi-head self-attention mechanism layer, a first residual connection and layer normalization layer, a feedforward neural network layer and a second residual connection and layer normalization layer; the processor 52, when embedding the position-encoded patches into the input encoder layer to obtain the key features of the original image, is used for:
[0125] For the target encoder layer, the position-encoded patches are embedded into the multi-head self-attention mechanism layer of the input target encoder layer to obtain the self-attention output features; where the target encoder layer is any one of the multiple encoder layers.
[0126] The self-attention output features and the position-encoded patches are embedded into the first residual connection and layer normalization layer of the input target encoder layer to obtain the first normalized features;
[0127] The first normalized feature is input into the feedforward neural network layer of the target encoder layer to obtain the feedforward network output feature;
[0128] The output features of the feedforward network and the first normalized features are input into the second residual connection and the layer normalization layer of the target encoder layer to obtain the second normalized features.
[0129] Optionally, the feature memory network includes: multiple serially connected base block modules and a head module. When the processor 52 inputs key features of the original image and key features of historical images into the feature memory network for feature fusion to obtain feature fusion information, it is used for:
[0130] The key features of the original image and the key features of the historical image are input into multiple basic block modules to obtain the initial fused features;
[0131] The initial fused features are input into the head module to obtain feature fusion information.
[0132] Optionally, when the processor 52 inputs the Gaussian parameters, point cloud information, and depth estimation information into the online backpropagation network for backpropagation iteration to obtain updated Gaussian parameters, updated point cloud information, and updated depth estimation information, it is used to:
[0133] The Gaussian parameters, point cloud information, and depth estimation information are input into the online backpropagation network to render the original image, resulting in a rendered image corresponding to the original image.
[0134] Determine the loss function based on the rendered image and the original image;
[0135] Backpropagation is performed based on the loss function, and the Gaussian parameters, point cloud information, and depth estimation information are iterated backward to obtain the updated Gaussian parameters, updated point cloud information, and updated depth estimation information.
[0136] Accordingly, this disclosure also provides a computer-readable storage medium storing a computer program. When the computer-readable storage medium stores a computer program, and the computer program is executed by one or more processors, the one or more processors cause the one or more processors to perform the steps in the method embodiment of FIG1.
[0137] Accordingly, this disclosure also provides a computer program product, which includes a computer program / instructions, and the computer program / instructions are executed by a processor to perform the steps in the method embodiment of FIG1.
[0138] The communication component in Figure 5 above is configured to facilitate wired or wireless communication between the device containing the communication component and other devices. The device containing the communication component can access wireless networks based on communication standards, such as WiFi, 2G, 3G, 4G / LTE, 5G, or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component also includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID), Infrared Data Association (IrDA) technology, Ultra-Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.
[0139] The power supply component in Figure 5 above provides power to various components of the device in which it resides. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which it resides.
[0140] The aforementioned electronic devices also include a display screen and audio components.
[0141] The display includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touchscreen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation.
[0142] An audio component may be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the device containing the audio component is in an operating mode, such as call mode, recording mode, or voice recognition mode. The received audio signals may be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.
[0143] In the embodiments of the methods, apparatus, devices, storage media, and computer programs disclosed above, the original image is input into an encoder for feature encoding to obtain key features of the original image; the key features of the original image and key features of historical images are input into a feature memory network for feature fusion to obtain feature fusion information, wherein the feature fusion information includes: Gaussian parameters, point cloud information, and depth estimation information. This disclosure predicts Gaussian parameters through a network; inputting high-quality Gaussian parameters, point cloud information, and depth estimation information into an online backpropagation network can reduce the number of backpropagations, improve image processing efficiency, and obtain more accurate updated Gaussian parameters using the backpropagation method, which can be applied to multiple task models and improve the model's compatibility with various task scenarios.
[0144] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0145] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more flowchart illustrations and / or one or more block diagrams.
[0146] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.
[0147] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.
[0148] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0149] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.
[0150] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0151] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.
[0152] The above are merely specific embodiments of this disclosure, enabling those skilled in the art to understand or implement this disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to these embodiments, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
[0153] All embodiments disclosed herein can be executed individually or in combination with other embodiments, and are all considered to be within the scope of protection claimed by this disclosure.
Claims
1. An image processing method, comprising: The original image is input into the encoder for feature encoding to obtain the key features of the original image; The key features of the original image and the key features of the historical image are input into the feature memory network for feature fusion to obtain feature fusion information, which includes: Gaussian parameters, point cloud information and depth estimation information; The Gaussian parameters, the point cloud information, and the depth estimation information are input into an online backpropagation network for backpropagation iteration to obtain updated Gaussian parameters, updated point cloud information, and updated depth estimation information. The updated Gaussian parameters are input into the predictor corresponding to each task type to obtain the prediction result, wherein the accuracy of the updated Gaussian parameters is higher than that of the original Gaussian parameters.
2. The method of claim 1, wherein, The feature fusion information also includes: object recognition information, object segmentation information, object tracking information, and feature point extraction information; the task types include: segmentation, detection, recognition, monocular estimation, mapping, localization, new perspective synthesis, and feature point extraction.
3. The method of claim 1 or 2, wherein, The encoder includes: a tile embedding layer, a position encoding layer, and multiple encoder layers, which are serially connected; the process of inputting the original image into the encoder for feature encoding to obtain key features of the original image includes: Inside the encoder, the original image is input into the tile embedding layer to obtain the tile embedding; The image patch embedding is input into the location encoding layer to obtain the location-encoded image patch embedding; The location-encoded tiles are embedded into the encoder layer to obtain the key features of the original image.
4. The method of claim 3, wherein, Each encoder layer includes: a multi-head self-attention mechanism layer, a first residual connection and layer normalization layer, a feedforward neural network layer, and a second residual connection and layer normalization layer; the step of embedding the position-encoded image patches into the encoder layer to obtain the key features of the original image includes: For the target encoder layer, the position-encoded patches are embedded into the multi-head self-attention mechanism layer of the target encoder layer to obtain self-attention output features; wherein, the target encoder layer is any one of the multiple encoder layers; The self-attention output features and the position-encoded patches are embedded into the first residual connection and layer normalization layer of the target encoder layer to obtain the first normalized features; The first normalized feature is input into the feedforward neural network layer of the target encoder layer to obtain the feedforward network output feature. The feedforward network output features and the first normalized features are input into the second residual connection and layer normalization layer of the target encoder layer to obtain the second normalized features.
5. The method of any one of claims 1-4, wherein, The feature memory network includes: multiple serially connected basic block modules and a head module. The key features of the original image and key features of historical images are input into the feature memory network for feature fusion to obtain feature fusion information, including: The key features of the original image and the key features of the historical image are input into multiple base block modules to obtain the initial fusion features; The initial fusion features are input into the head module to obtain the feature fusion information.
6. The method of any one of claims 1-5, wherein, The step of inputting the Gaussian parameters, the point cloud information, and the depth estimation information into an online backpropagation network for backpropagation iteration to obtain updated Gaussian parameters, updated point cloud information, and updated depth estimation information includes: The Gaussian parameters, the point cloud information, and the depth estimation information are input into the online backpropagation network to render the original image, thereby obtaining the rendered image corresponding to the original image. Determine the loss function based on the rendered image and the original image; Backpropagation is performed based on the loss function to iterate back through the Gaussian parameters, the point cloud information, and the depth estimation information to obtain the updated Gaussian parameters, the updated point cloud information, and the updated depth estimation information.
7. An image processing apparatus, comprising: The encoding module is used to input the original image into the encoder for feature encoding to obtain the key features of the original image; The fusion module is used to input the key features of the original image and the key features of the historical image into the feature memory network for feature fusion to obtain feature fusion information, wherein the feature fusion information includes: Gaussian parameters, point cloud information and depth estimation information; The rendering module is used to input the Gaussian parameters, the point cloud information, and the depth estimation information into the online backpropagation network for backpropagation to obtain updated Gaussian parameters, updated point cloud information, and updated depth estimation information. The prediction module is used to input the updated Gaussian parameters into the predictor corresponding to each task type to obtain the prediction result, wherein the accuracy of the updated Gaussian parameters is higher than that of the original Gaussian parameters.
8. An electronic device, comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the steps of the method as described in any one of claims 1-6.
9. A computer readable storage medium having stored thereon a computer program, wherein, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-6.
10. A computer program product comprising computer programs / instructions, wherein, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1-6.