Image restoration method and related apparatus

Abstract

Provided in the embodiments of the present application are an image restoration method and a related apparatus, which are used for restoring images of different input modes or different video paths by means of one image restoration network, so as to reduce the area occupied by chips. The method comprises: acquiring a first image and path information, wherein the first image is an image acquired by a first image sensor, the path information is used for adjusting a network structure of a first image restoration network, so as to obtain a second image restoration network or a third image restoration network, network structures of the second image restoration network and the third image restoration network are different, the second image restoration network is used for performing image restoration on an image of a first video mode or a first video path, and the third image restoration network is used for performing image restoration on an image of a second video mode or a second video path; and performing image restoration on the first image by means of a target image restoration network, so as to obtain an image restoration result of the first image.

Description

An image restoration method and related apparatus

[0001] This application claims priority to Chinese Patent Application No. 202510127757.5, filed with the State Intellectual Property Office of China on January 27, 2025, entitled "An Image Restoration Method and Related Apparatus", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of image processing, and more particularly to an image restoration method and related apparatus. Background Technology

[0003] Currently, images acquired by image sensors (such as cameras) are unprocessed RAW images. However, limited by the performance and power consumption of image sensors, the quality of RAW images acquired by image sensors is still not high enough, suffering from low resolution, severe noise, insufficient brightness, loss of detail, and blurriness. Therefore, the quality restoration of RAW images has become a very urgent need.

[0004] Furthermore, to meet diverse user needs, terminal devices and image processing chips require support for various video shooting modes. Moreover, terminal devices typically deploy multiple cameras, each with multiple video shooting modes, including different resolutions, frame rates, exposure methods, and power consumption modes. When a terminal device deploys multiple cameras for video shooting, different video recovery networks are needed for different video paths or different video shooting modes due to potential differences in image exposure methods, power consumption requirements, or output image specifications (such as resolution and frame rate).

[0005] However, deploying multiple video recovery networks independently would consume excessive chip logic and storage space. Therefore, image recovery for different video modes or different video paths remains a pressing issue that needs to be addressed. Summary of the Invention

[0006] This application provides an image restoration method and related apparatus for restoring images from different input modes or different video paths through an image restoration network, thereby reducing the area occupied by the chip.

[0007] In a first aspect, this application provides an image restoration method, which includes: firstly, acquiring a first image to be restored and path information, wherein the first image is an image acquired by a first image sensor, and the path information is used to adjust the network structure of a first image restoration network to obtain a second image restoration network or a third image restoration network, wherein the network structures of the second image restoration network and the third image restoration network are different, the second image restoration network is used to restore images of a first video mode or a first video path, and the third image restoration network is used to restore images of a second video mode or a second video path, wherein the first video mode and the second video mode are different video modes, and the first video path and the second video path are different video paths; subsequently, image restoration of the first image can be performed through a target image restoration network to obtain an image restoration result of the first image, wherein the target image restoration network is the second image restoration network or the third image restoration network.

[0008] In this embodiment, by introducing path information to adjust the network structure of the first image restoration network, a variety of image restoration networks with different structures (such as a second image restoration network or a third image restoration network) can be obtained. These different image restoration networks can perform image restoration on images with different video modes or different video paths. Based on the video mode or video path of the input image, the network structure of the first image restoration network can be adjusted through path information to obtain the corresponding image restoration network. Only one image restoration network needs to be deployed to realize image restoration on images with different video modes or different video paths, avoiding the need to deploy an image restoration network independently for each video mode or video path, thereby reducing the chip area.

[0009] In one possible implementation, the aforementioned image restoration of the first image through the target image restoration network to obtain the image restoration result of the first image may include: processing the output features through the target image restoration network to obtain the image restoration result, wherein the output features are obtained by splicing the fused features and the shallow features of the first image, and the shallow features are obtained by feature extraction of the first image.

[0010] In one possible implementation, the method may further include: acquiring a second image and an auxiliary image, wherein the second image is any frame image from the first N frames of the first image, where N is a positive integer, the auxiliary image is an image acquired by the second image sensor, the exposure mode of the auxiliary image is different from that of the first image, the image content of the auxiliary image is the same as that of the first image, and the second image and the auxiliary image are used to obtain fusion features.

[0011] In this embodiment of the application, images with different time domains and different exposure methods can also be acquired to provide richer image content and improve the restoration quality of images or videos.

[0012] In one possible implementation, the aforementioned path information includes path exposure information, which is used to indicate the method of fusing the first image, the second image, and the auxiliary image. Before processing the output features through the target image restoration network to obtain the image restoration result, the method may further include: fusing the first image, the second image, and the auxiliary image according to the path exposure information and the attention mechanism to obtain fused features.

[0013] In this embodiment, by fusing different information such as time domain and multiple exposures, information interaction is achieved between different information sources, ensuring consistent results and providing multi-source image information for image restoration, thereby improving the quality of image or video restoration.

[0014] In one possible implementation, the aforementioned process of fusing the first image, the second image, and the auxiliary image based on path exposure information and an attention mechanism to obtain fused features may include: obtaining a query vector, a key vector, and a value vector based on the first image, the second image, and the auxiliary image, wherein the query vector is obtained by performing operations on the first image, the key vector is obtained by performing operations on the second image and the auxiliary image, and the value vector is obtained by performing operations on the second image and the auxiliary image; and performing operations on the query vector, the key vector, the value vector, and a correlation matrix mask image based on the attention mechanism to obtain fused features, wherein the correlation matrix mask image is determined based on path exposure information.

[0015] In one possible implementation, the aforementioned pathway information further includes network structure adjustment information, which is used to indicate the number of feature extraction modules in the first image restoration network. The feature extraction modules are used to extract high-frequency features of the image. Before performing image restoration on the first image through the target image restoration network to obtain the image restoration result of the first image, the method may further include: determining the number of feature extraction modules in the first image restoration network according to the network structure adjustment information to obtain a temporary image restoration network.

[0016] In this embodiment, the path information also includes network structure adjustment information. This network structure adjustment information can adjust the structure of the first image restoration network by determining the number of feature extraction modules in the first image restoration network. The feature extraction modules are used to extract high-frequency features; therefore, adjusting the network structure will not affect the low-frequency features of the image, thus maintaining consistency in brightness, noise, and artifacts, making the image restoration effect more controllable. Furthermore, by adjusting the image restoration network through path information, the consistency of the image restoration results is improved, thereby reducing the jump in restoration effect when switching between different image restoration networks for images from different video paths.

[0017] In one possible implementation, the aforementioned path information further includes power consumption information, which is used to indicate the number of convolutional layers and the number of channels in the output layer of the first image restoration network. Before performing image restoration on the first image through the target image restoration network to obtain the image restoration result of the first image, the method may further include: determining the number of convolutional layers and the number of channels in the output layer of the temporary image restoration network based on the power consumption information, and obtaining the target image restoration network.

[0018] In this embodiment, the number of convolutional layers in the output layer can be adjusted through path information to achieve dynamic convolution, adjust the output result, avoid adding extra branches, and not increase the computational load of the network.

[0019] Secondly, this application provides an image restoration apparatus, comprising: an acquisition module for acquiring a first image and path information, wherein the first image is an image acquired by a first image sensor, and the path information is used to adjust the network structure of a first image restoration network to obtain a second image restoration network or a third image restoration network, wherein the network structures of the second image restoration network and the third image restoration network are different, the second image restoration network is used to perform image restoration on an image of a first video mode or a first video path, and the third image restoration network is used to perform image restoration on an image of a second video mode or a second video path, wherein the first video mode and the second video mode are different video modes, and the first video path and the second video path are different video paths; and a restoration module for performing image restoration on the first image through a target image restoration network to obtain an image restoration result of the first image, wherein the target image restoration network is the second image restoration network or the third image restoration network.

[0020] In one possible implementation, the aforementioned restoration module is specifically used to: process the output features through a target image restoration network to obtain an image restoration result, wherein the output features are obtained by splicing the fused features with the shallow features of the first image, and the shallow features are obtained by feature extraction from the first image.

[0021] In one possible implementation, the aforementioned acquisition module is further configured to acquire a second image and an auxiliary image. The second image is any frame from the first N frames of the first image, where N is a positive integer. The auxiliary image is an image acquired by the second image sensor. The exposure mode of the auxiliary image is different from that of the first image. The image content of the auxiliary image is the same as that of the first image. The second image and the auxiliary image are used to obtain fusion features.

[0022] In one possible implementation, the aforementioned path information includes path exposure information, which is used to indicate the method of fusing the first image, the second image, and the auxiliary image. Before processing the output features through the target image restoration network to obtain the image restoration result, the device may further include: a fusion module, used to fuse the first image, the second image, and the auxiliary image according to the path exposure information and the attention mechanism to obtain fused features.

[0023] In one possible implementation, the aforementioned fusion module is specifically used to: obtain a query vector, a key vector, and a value vector based on the first image, the second image, and the auxiliary image, wherein the query vector is obtained by performing operations on the first image, the key vector is obtained by performing operations on the second image and the auxiliary image, and the value vector is obtained by performing operations on the second image and the auxiliary image; and perform operations on the query vector, the key vector, the value vector, and the correlation matrix mask image based on the attention mechanism to obtain fusion features, wherein the correlation matrix mask image is determined based on the path exposure information.

[0024] In one possible implementation, the aforementioned pathway information further includes network structure adjustment information, which is used to indicate the number of feature extraction modules in the first image restoration network. The feature extraction modules are used to extract high-frequency features of the image. Before performing image restoration on the first image through the target image restoration network to obtain the image restoration result of the first image, the device may further include: a determining module, used to determine the number of feature extraction modules in the first image restoration network according to the network structure adjustment information, and obtain a temporary image restoration network.

[0025] In one possible implementation, the aforementioned path information further includes power consumption information, which is used to indicate the number of convolutional layers and the number of channels in the output layer of the first image restoration network. Before performing image restoration on the first image through the target image restoration network to obtain the image restoration result of the first image, the aforementioned determining module is also used to determine the number of convolutional layers and the number of channels in the output layer of the temporary image restoration network based on the power consumption information, so as to obtain the target image restoration network.

[0026] Thirdly, this application provides a processor that can be connected to a memory for executing instructions stored in the memory, so that the processor performs the method as described in the first aspect or any possible implementation thereof.

[0027] Fourthly, embodiments of this application provide a chip system including a processor and input / output ports. The processor is used to implement the processing functions involved in the method described in the first aspect above, and the input / output ports are used to implement the transmission and reception functions involved in the method described in the first aspect above.

[0028] In one possible design, the chip system also includes a memory for storing program instructions and data for implementing the functions involved in the method described in the first aspect above.

[0029] This chip system can consist of chips or include chips and other discrete components.

[0030] Fifthly, embodiments of this application provide a computer-readable storage medium. This computer-readable storage medium stores computer instructions; when these computer instructions are executed on a computer, they cause the computer to perform the method as described in any of the possible implementations of the first aspect.

[0031] Sixthly, embodiments of this application provide a computer program product. This computer program product includes a computer program or instructions that, when executed on a computer, cause the computer to perform the method as described in any of the possible implementations of the first aspect.

[0032] The technical effects of the second to sixth aspects or any of their possible implementations can be found in the first aspect or the technical effects of its related possible implementations, and will not be repeated here. Attached Figure Description

[0033] Figure 1 is a schematic diagram of a system architecture provided in an embodiment of this application;

[0034] Figure 2 is a schematic diagram of an image restoration network framework proposed in this application;

[0035] Figure 3 is a schematic diagram of a single RAW video restoration network supporting different video paths;

[0036] Figure 4 is a flowchart illustrating an image restoration method provided in this application;

[0037] Figure 5 is a schematic diagram of rearranging the spatial positions of pixels in an image;

[0038] Figure 6 is a schematic diagram of the process of fusing multi-source information based on the attention mechanism;

[0039] Figure 7 is a schematic diagram of the dynamic image restoration structure based on frequency domain routing in the image restoration network;

[0040] Figure 8 is a schematic diagram of the comparison results of detail effects;

[0041] Figure 9 is a schematic diagram showing the comparison results of noise reduction effects;

[0042] Figure 10 is a schematic diagram showing the comparison results of color effects;

[0043] Figure 11 is a schematic diagram of the structure of an image restoration device provided in this application;

[0044] Figure 12 is a schematic diagram of another image restoration device provided in this application;

[0045] Figure 13 is a schematic diagram of the structure of a chip provided in this application. Detailed Implementation

[0046] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0047] To facilitate understanding, some concepts involved in this application will be introduced first.

[0048] 1. RAW images

[0049] RAW images are raw data captured by a complementary metal-oxide-semiconductor (CMOS) image sensor or charge-coupled device (CCD) and converted into digital signals.

[0050] 2. Attention Mechanism

[0051] Attention mechanisms are designed to improve processing efficiency and accuracy by allowing models to focus on key information while ignoring irrelevant details. They mimic the selective attention humans exhibit when processing information, allowing models to dynamically adjust their attention weights as they process input data, highlighting important information and ignoring less important information. Attention mechanisms determine attention weights by calculating the similarity between the query and key vectors, and then perform a weighted summation of the value vectors to obtain the final output.

[0052] 3. Pixel shuffle

[0053] Pixel refactoring is an image processing technique, specifically an upsampling method in deep learning. It involves refactoring low-resolution feature maps through convolution and multi-channel operations to obtain high-resolution feature maps. Its main function is to effectively enlarge the scaled-down feature maps, thereby generating a higher-resolution output image. Pixel unshuffle is the inverse operation of pixel shuffle, remapping high-frequency information from the spatial dimension back to the channel dimension. It is primarily used to downsample feature maps from higher spatial resolution to lower resolution.

[0054] 4. Peak Signal-to-Noise Ratio (PSNR)

[0055] Peak signal-to-noise ratio (PSNR) is the ratio between the maximum possible power of a signal and the power of destructive noise that affects the accuracy of signal representation. It is commonly used to measure the quality of reconstructed images after lossy transformations such as image compression, and serves as a standard for measuring the relationship between image signal strength and noise intensity. PSNR is usually expressed in logarithmic decibels (dB). It represents the multiple of the signal-to-noise ratio relative to a reference value, thus intuitively reflecting the image's quality level.

[0056] To make the technical solution of this application clearer and easier to understand, the system architecture of this application will be described below with reference to the accompanying drawings.

[0057] Referring to Figure 1, a system architecture provided in this application is shown. As shown in Figure 1, the artificial intelligence model training system 100 may include a computing device cluster 110, which includes at least one computing device. The computing device may be a server, such as a cloud server, a central server, an edge server, or a local server in a local data center. In some possible embodiments, the computing device may also be a mobile phone, tablet / desktop computer, laptop computer, or other terminal device.

[0058] Specifically, the image restoration network used for image restoration can be distributed in the computing unit of any computing device in the aforementioned artificial intelligence model training system 100. The computing unit can be a graphics processing unit (GPU), a neural network processing unit (NPU), etc., and there is no specific limitation here.

[0059] It is worth noting that the system architecture shown in Figure 1 is only an example and is not intended to limit its specific implementation to this example. For example, in other possible system architectures, the artificial intelligence model training system 100 may also include the client, which provides video data or image data, etc., to the computing device.

[0060] For example, the image restoration network framework proposed in this application is shown in Figure 2. The image restoration network framework includes a convolutional layer 210, a multi-source information fusion structure 220 based on a correlation matrix mask (or a fusion module), a dynamic image restoration structure 230 based on frequency domain routing (or a restoration module), a multi-level output network layer 240, and a pre-frame preprocessing module 250. The image restoration network performs image restoration on the original image frames received from different modules, generating high-quality image frames with rich details, noise suppression, and no artificial artifacts.

[0061] The multi-source information fusion structure 220 based on the correlation matrix mask can fuse the input current frame image, the previous frame image, auxiliary images, and path information to obtain fusion features. The dynamic image restoration structure 230 based on frequency domain routing and the multi-level output network layer 240 can process these fusion features to obtain multiple image restoration results, such as restoration results at different frame rates or with different models, such as outputting 4K 30 HDR video, 4K 30 FPS video, or 4K 60 FPS video. The previous frame preprocessing module 250 can perform a series of processing on the video frames output by the network, such as image registration and color changes, and use the processed image as the input for the next frame (i.e., the previously received previous frame image).

[0062] Currently, multiple different RAW domain video recovery networks need to be deployed for different video paths. These multiple RAW domain video recovery networks have different structures and are deployed independently, which consumes too much chip logic and storage area.

[0063] To address the current problem, this application proposes an image restoration method. By introducing path information, an image restoration network is used to restore images from different input modes or different video paths. By reusing the structure and parameters of the same image restoration network, the chip area and power consumption can be reduced. Figure 3 illustrates a single RAW video restoration network supporting different video modes. As shown in Figure 3, the RAW video restoration network can be adjusted using the introduced path information, allowing for image restoration of images captured by different modules with only one network deployed.

[0064] The image restoration network proposed in this application can be applied to various image restoration tasks that take RAW images as input, such as noise reduction, demosaicing, and super-resolution. The modules in the image restoration network proposed in this application can also be adapted to other tasks, such as the multi-source information fusion structure based on correlation matrix masks, which can be applied to other multi-source data fusion and restoration tasks. Furthermore, the image restoration network proposed in this application is adaptable to images acquired by different sensor models and in different imaging scenarios.

[0065] The method flow provided in this application will be described below in conjunction with the aforementioned system architecture.

[0066] Referring to Figure 4, a flowchart of an image restoration method provided in this application is shown below.

[0067] Step 401: Obtain the first image and pathway information;

[0068] The first image is either an image captured by the first image sensor in the terminal device at a first moment, or an image corresponding to the first moment in a video sequence captured by the first image sensor.

[0069] To meet diverse user needs, terminal devices typically need to support various video shooting modes. Cameras on these devices often also offer multiple video shooting modes, which may differ in input format, resolution, frame rate, exposure method, and power consumption mode. Furthermore, images from different video paths may also differ in input format, resolution, frame rate, exposure method, and power consumption mode.

[0070] Based on this, in order to enable the first image restoration network to adapt to images from different shooting modes or different video paths for image restoration, path information can be introduced. This path information is used to adjust the network structure of the first image drawing network to obtain a second image restoration network or a third image restoration structure. The second and third image restoration networks have different structures. Furthermore, the second image restoration network may include one or more image restoration networks, and each image restoration network has a different structure. Similarly, the third image restoration network may also include one or more image restoration networks, and each image restoration network has a different structure.

[0071] The second image restoration network can restore images of the first video mode or the first video path, and the third image restoration network can restore images of the second video mode or the second video path. The first video mode and the second video mode are different, and the first video path and the second video path are different, so that image restoration networks with different network structures can restore images of different video modes or different video paths.

[0072] Typically, when taking pictures using an image sensor (such as a camera), different shooting environments will affect the quality of the captured images. To obtain better image quality, different exposure modes and images in different time domains can be selected for different shooting environments.

[0073] Optionally, to improve image restoration quality, a second image and an auxiliary image can also be acquired. The second image is any frame from the previous N frames of the first image, where N is a positive integer. The auxiliary image is an image acquired by the second image sensor at a second time. The second image sensor can be the same device as the first image sensor or a different device. The second time can be the same time as the first time or a different time, and the second time can also be an adjacent time to the first time.

[0074] Specifically, the auxiliary image can be acquired at the same time (i.e., the first time and the second time are the same time) using different image sensors (the first image sensor and the second image sensor are different devices) on the same target, using an exposure method different from the first image;

[0075] Alternatively, different image sensors can be used to capture images of the same target at different times (the first time and the second time are different times, such as the second time being adjacent to the first time), using different exposure methods than the first image.

[0076] Alternatively, it can be obtained by using different image sensors on the same target at different times, and by using different exposure methods than the first image.

[0077] Acquiring auxiliary images can provide richer image content, allowing for the subsequent recovery of details from the first image based on images acquired using various exposure methods or images from different modules, thereby improving the quality of the first image. Here, "different modules" refers to a combination of multiple different cameras or multiple different shooting modes.

[0078] Optionally, the path information includes path exposure information, which is used to indicate the method of fusing the first image, the second image, and the auxiliary image. After acquiring the first image, the second image, the path information, and the auxiliary image, the first image, the second image, and the auxiliary image can be fused according to the path exposure information and the attention mechanism to obtain fused features.

[0079] Optionally, a query vector, a key vector, and a value vector can be obtained based on the first image, the second image, and the auxiliary image. The query vector is obtained by processing the first image, the key vector is obtained by processing the second image and the auxiliary image, and the value vector is obtained by processing the second image and the auxiliary image. Subsequently, an attention mechanism can be used to process the query vector, key vector, value vector, and a correlation matrix mask to obtain fused features. The correlation matrix mask is determined based on the path exposure information.

[0080] For example, the video sequence captured by the terminal device includes T frames of images. The first image is a video frame at time t. The dimensions of the first image can be [H, W, Cin], where H is the image height, W is the image width, and Cin is the number of channels of the image. RAW images correspond to Cin = 1, and RGB images correspond to Cin = 3. The dimensions of the second image are [H / 2, W / 2, Cp], where Cp is the number of channels of the second image. The dimensions of the auxiliary image are [H, W, Caux], where Caux is the number of channels of the auxiliary image. The dimensions of the path information are [Hm, Wm, Cm], where Hm and Wm are the spatial scales of the path information, and Cm is the number of channels of the first image, where Cm = Cin.

[0081] First, after acquiring the first image and the auxiliary image, the pixels of the first image and the auxiliary image can be rearranged spatially using the pixel unshuffle operation, as shown in Figure 5. After the pixel unshuffle operation, the dimensions of the first image can be transformed into [H / 2, W / 2, Cin×4], and the dimensions of the auxiliary image can be transformed into [H / 2, W / 2, Caux×4].

[0082] Subsequently, features can be extracted from the first image, the second image, and the auxiliary image through multiple 2D convolutional layers to obtain the shallow features Ft,2 of the first image with dimensions [H / 2, W / 2, Cf2], the shallow features Ft,3 of the second image with dimensions [H / 2, W / 2, Cf3], and the shallow features Ft,4 of the auxiliary image with dimensions [H / 2, W / 2, Cf4].

[0083] Next, as shown in Figure 6, which is a schematic diagram of the process of fusing multi-source information based on the attention mechanism, Ft,2 can be spatially divided into NH×NW blocks, each with dimensions [H / 2NH,W / 2NW,Cf2]. Each block can be represented as ft,2,i,j, with dimensions [hp,wp,Cf2]. Similarly, Ft,4 can be spatially divided into NH×NW blocks, each with dimensions [hp,wp,Cf4]. Here, i and j represent the spatial location of the blocks. After spatially dividing Ft,3 into NH×NW blocks, n blocks can be taken within a window centered on i,j, and these n blocks are concatenated along the channel dimension to obtain ft,3,i,j, with dimensions [hp,wp,Cf3×n].

[0084] After dividing Ft,2,Ft,3,Ft,3 into smaller blocks, we can perform operations on each block based on an attention mechanism. The process of performing operations on each block to obtain the fused feature blocks is described below:

[0085] First, the dimensions of ft,2,i,j can be transformed into [hp,wp,Cf4], and the dimensions of ft,4,i,j can also be transformed into [hp,wp,Cf4]. Then, ft,2,i,j is passed through a fully connected layer to obtain the query vector qt,i,j, with dimensions [hp×wp,Cq]. The channels of ft,3,i,j and ft,4,i,j are concatenated to obtain the feature ft,5,i,j, with dimensions [hp,wp,Cf3×n+Cf4]. Finally, ft,5,i,j is passed through a fully connected layer to obtain the key vector kt,i,j and the value vector vt,i,j, with dimensions [2×hp×wp,Ck] and vt,i,j, respectively.

[0086] Based on the attention mechanism, qt,i,j can be multiplied by the transpose of kt,i,j, and the result is normalized using the softmax function to obtain the correlation matrix rt,i,j, with dimensions [2×hp×wp,hp×wp]. Then, a correlation mask map mt,i,j, with dimensions [2×hp×wp,hp×wp], can be generated based on the path exposure information in the path information. This correlation mask map is used to control the fusion method of the first image, the second image, and the auxiliary image. Different correlation masks can be used to fuse different parts of the first image, the second image, and the auxiliary image, resulting in various fusion methods. Performing a dot product operation on rt,i,j and mt,i,j yields the correlation matrix based on the mask map. Subsequently Perform matrix multiplication with vt,i,j to obtain the fused feature blocks. When the dimensions of the correlation mask image mt,i,j are represented as [H,W,C], the dimensions H and W can be different or equal, and no specific restriction is made here.

[0087] After performing calculations on each small block, each fused feature block can be... The features are concatenated to obtain the fused feature Ft,1, which has dimensions [H / 2, W / 2, Cv].

[0088] Optionally, features can be extracted from the first image through multiple convolutional layers to obtain shallow features Ft,0 of the first image, with dimensions [H, W, Cf1]. Then, the shallow features Ft,0 and the fused features Ft,1 of the first image can be concatenated to obtain the output features.

[0089] Step 402: Perform image restoration on the first image using the target image restoration network to obtain the image restoration result of the first image.

[0090] In this embodiment, the backbone of the first image restoration network is a Unet-style structure, where each processing unit is a frequency-division dynamic module (FDB), as shown in Figure 7. As shown in Figure 7, the image restoration network includes a frequency-domain routing-based dynamic image restoration structure, which includes convolutional layers and multiple frequency-division dynamic modules (or feature extraction modules). Furthermore, the image restoration network also includes an output layer.

[0091] The target image restoration network can be a second image restoration network or a third image restoration network. The specific network structure of the target image restoration network is determined based on the video mode or video path of the first image.

[0092] Optionally, the pathway information may further include network structure adjustment information, which indicates the number of feature extraction modules in the first image restoration network, wherein the feature extraction modules are used to extract high-frequency features of the image. Based on this network structure adjustment information, the number of feature extraction modules in the first image restoration network can be determined to obtain a temporary image restoration network.

[0093] The feature extraction module extracts high-frequency features. By adjusting the number of feature extraction modules in the first image restoration network using path information, and only adjusting the number of modules extracting high-frequency features, the network structure adjustment does not affect the low-frequency features of the image. This maintains consistency in image quality, noise, and artifacts, making the image restoration effect more controllable and improving consistency. Furthermore, by improving the consistency of image restoration across different video paths, the abrupt changes in image restoration effect can be reduced when switching between different image restoration networks for different video paths. Moreover, by adjusting the network structure of the first image restoration network only through path information, multiple image restoration networks with different structures can be obtained, enabling switching between various types of image restoration networks without introducing additional control branches or increasing computational load.

[0094] In this embodiment, pathway information is introduced, which includes network structure adjustment information. This information can adjust the number of feature extraction modules in the first image restoration network. Each feature extraction module has the same structure. Therefore, image restoration networks with different network structures obtained through pathway information adjustment can reuse the same part of the network structure and parameters, thereby reducing the chip area occupied and reducing the power consumption when switching image restoration networks.

[0095] Optionally, the path information may also include power consumption information, which indicates the number of convolutional layers and channels in the output layer of the first image restoration network. Based on the power consumption information, the number of convolutional layers and channels in the output layer of the temporary image restoration network can be determined to obtain the target image restoration network.

[0096] In this embodiment, the number of convolutional layers and channels in the output layer can be adjusted through path information to dynamically adjust the output result, avoiding the need to add additional control branches and not increasing the computational load of the network.

[0097] Optionally, the output features can be analyzed using a target image restoration network to obtain the image restoration result.

[0098] Specifically, after obtaining the output features Afterwards, the output features can be extracted through a layer normalization layer, a loss-adjustable convolutional layer (Tconv layer), and a simple gate layer to obtain intermediate features. Subsequently, a discrete wavelet transform is performed on the intermediate features to obtain the high-frequency features. and low frequency characteristics Then, based on the power consumption information in the path information, the number N of feature extraction modules is determined, where the power consumption information carries the number of feature extraction modules. Each feature extraction module includes a layer normalization layer, a loss-adjustable convolutional layer (Tconv layer), and a simple gate layer. High-frequency features are enhanced through N feature extraction modules to obtain high-frequency enhanced features. Subsequently, high-frequency enhancement features can be used. and low frequency characteristics An inverse discrete wavelet transform is performed to obtain the final features. Finally, the final features are analyzed through the output layer to obtain one or more image restoration results Ot,m, where multiple image restoration results can be multiple images of different resolutions or images of different frame rates, etc.

[0099] The training and inference process of the image restoration network proposed in the embodiments of this application are described below.

[0100] During training, image data from different video modes or video paths can be used as input to the image restoration network. Constraints are applied between the network's output and the ground truth image (a manually defined RAW image) based on L1 reconstruction error (L1 loss) and perceptual error (Perceptual loss). Subsequently, the network's parameters are trained using the error between the restored image and the RAW image, resulting in the trained image restoration network.

[0101] After obtaining the trained image restoration network, it can be used to restore images or videos, resulting in restored images or videos.

[0102] For example, the image restoration network proposed in this application and the existing scheme were used to restore images for video mode 1 and video mode 2 respectively, and the peak signal-to-noise ratio (PSNR) was used as the evaluation index for RAW image restoration. The comparison results are shown in Table 1.

[0103] Table 1

[0104] In this scheme, Mode 1 has the same equivalent computational cost as the baseline algorithm of Mode 1; Mode 2 has the same equivalent computational cost as the baseline algorithm of Mode 2. As shown in Table 1, the image restoration network proposed in this embodiment yields higher quality image restoration results.

[0105] For example, the image restoration results obtained by the image restoration network proposed in this application embodiment are compared with the image restoration results obtained by existing solutions. The comparison results of detail effects are shown in Figure 8. As can be seen from Figure 8, the image restoration network proposed in this application embodiment can effectively improve the detail and detail consistency of the restored video. As can be seen from the comparison results of denoising effects shown in Figure 9, the image restoration network proposed in this application embodiment can effectively improve the denoising effect and denoising consistency of the restored video. As can be seen from the comparison results of color effects shown in Figure 10, the image restoration network proposed in this application embodiment can effectively improve the color accuracy and consistency of the restored video.

[0106] The method flow provided in this application has been described above. The apparatus provided in this application will now be described based on the aforementioned method flow.

[0107] Referring to Figure 11, a schematic diagram of the structure of an image restoration device provided in this application is shown below.

[0108] The acquisition module 1101 is used to acquire a first image and path information. The first image is an image acquired by a first image sensor. The path information is used to adjust the network structure of the first image restoration network to obtain a second image restoration network or a third image restoration network. The network structures of the second image restoration network and the third image restoration network are different. The second image restoration network is used to restore the image of the first video mode or the first video path. The third image restoration network is used to restore the image of the second video mode or the second video path. The first video mode and the second video mode are different video modes. The first video path and the second video path are different video paths.

[0109] The recovery module 1102 is used to perform image recovery on the first image through a target image recovery network to obtain the image recovery result of the first image. The target image recovery network is a second image recovery network or a third image recovery network.

[0110] In one possible implementation, the aforementioned restoration module 1102 is specifically used to: process the output features through the target image restoration network to obtain the image restoration result, wherein the output features are obtained by splicing the fused features and the shallow features of the first image, and the shallow features are obtained by feature extraction from the first image.

[0111] In one possible implementation, the aforementioned acquisition module 1101 is further configured to acquire a second image and an auxiliary image. The second image is any frame image from the first N frames of the first image, where N is a positive integer. The auxiliary image is an image acquired by the second image sensor. The exposure mode of the auxiliary image is different from that of the first image. The image content of the auxiliary image is the same as that of the first image. The second image and the auxiliary image are used to obtain fusion features.

[0112] In one possible implementation, the aforementioned path information includes path exposure information, which is used to indicate the method of fusing the first image, the second image, and the auxiliary image. Before processing the output features through the target image restoration network to obtain the image restoration result, the device may further include: a fusion module 1103, used to fuse the first image, the second image, and the auxiliary image according to the path exposure information and the attention mechanism to obtain fused features.

[0113] In one possible implementation, the aforementioned fusion module 1103 is specifically used to: obtain a query vector, a key vector, and a value vector based on the first image, the second image, and the auxiliary image, wherein the query vector is obtained by performing operations on the first image, the key vector is obtained by performing operations on the second image and the auxiliary image, and the value vector is obtained by performing operations on the second image and the auxiliary image; and perform operations on the query vector, the key vector, the value vector, and the correlation matrix mask image based on the attention mechanism to obtain fusion features, wherein the correlation matrix mask image is determined based on the path exposure information.

[0114] In one possible implementation, the aforementioned pathway information further includes network structure adjustment information, which is used to indicate the number of feature extraction modules in the first image restoration network. The feature extraction modules are used to extract high-frequency features of the image. Before performing image restoration on the first image through the target image restoration network to obtain the image restoration result of the first image, the device may further include: a determining module 1104, used to determine the number of feature extraction modules in the first image restoration network according to the network structure adjustment information, and obtain a temporary image restoration network.

[0115] In one possible implementation, the aforementioned path information also includes power consumption information, which is used to indicate the number of convolutional layers and the number of channels in the output layer of the first image restoration network. Before performing image restoration on the first image through the target image restoration network to obtain the image restoration result of the first image, the aforementioned determining module 1104 is also used to determine the number of convolutional layers and the number of channels in the output layer of the temporary image restoration network based on the power consumption information, so as to obtain the target image restoration network.

[0116] Please refer to Figure 12, a schematic diagram of another image restoration device provided in this application, as described below.

[0117] The image restoration apparatus may include a processor 1201 and a memory 1202. The processor 1201 and the memory 1202 are interconnected via a circuit. The memory 1202 stores program instructions and data.

[0118] The memory 1202 stores the program instructions and data corresponding to the steps in Figure 4 above.

[0119] The processor 1201 is used to execute the method steps performed by the image restoration apparatus shown in FIG4 above.

[0120] Optionally, the image restoration apparatus may also include a transceiver 1203 for receiving or sending data.

[0121] Optionally, the image restoration device shown in FIG12 above is a chip.

[0122] This application embodiment also provides an image restoration device, which can also be called a digital processing chip or chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface, and the program instructions are executed by the processing unit. The processing unit is used to execute the method steps of the image restoration device shown in the embodiment of FIG4 above.

[0123] This application also provides a digital processing chip. This digital processing chip integrates circuitry for implementing the functions of the processor 1201 described above, and one or more interfaces. When the digital processing chip integrates a memory, it can complete the method steps of any one or more of the foregoing embodiments. When the digital processing chip does not integrate a memory, it can be connected to an external memory via a communication interface. The digital processing chip implements the actions performed by the image restoration device in the foregoing embodiments according to the program code stored in the external memory.

[0124] The image restoration device provided in this application embodiment can be a chip, which includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip in the server to execute the method described in the embodiment shown in FIG4 above. Optionally, the storage unit can be a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0125] Specifically, the aforementioned processing unit or processor can be a central processing unit (CPU), a neural-network processing unit (NPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0126] For example, please refer to Figure 13, which is a schematic diagram of a chip structure provided in an embodiment of this application. The chip can be represented as a neural network processor (NPU) 1300. The NPU 1300 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core part of the NPU is the arithmetic circuit 1303, which is controlled by the controller 1304 to extract matrix data from the memory and perform multiplication operations.

[0127] In some implementations, the arithmetic circuit 1303 internally includes multiple process engines (PEs). In some implementations, the arithmetic circuit 1303 is a two-dimensional pulsating array. The arithmetic circuit 1303 can also be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1303 is a general-purpose matrix processor.

[0128] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 1302 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 1301 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 1308.

[0129] Unified memory 1306 is used to store input and output data. Weight data is directly transferred to weight memory 1302 via direct memory access controller (DMAC) 1305. Input data is also transferred to unified memory 1306 via DMAC.

[0130] The bus interface unit (BIU) 1310 is used for interaction between the AXI bus and the DMAC and instruction fetch buffer (IFB) 1309.

[0131] The bus interface unit 1310 (BIU) is used by the instruction fetch memory 1309 to fetch instructions from external memory, and also by the memory access controller 1305 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0132] The DMAC is mainly used to move input data from external memory DDR to unified memory 1306, or to weight data to weight memory 1302, or to input data to input memory 1301.

[0133] The vector computation unit 1307 includes multiple arithmetic processing units that further process the output of the computation circuit as needed, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is mainly used for computation in non-convolutional / fully connected layers of neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0134] In some implementations, the vector computation unit 1307 can store the processed output vector in the unified memory 1306. For example, the vector computation unit 1307 can apply linear and / or nonlinear functions to the output of the computation circuit 1303, such as performing linear interpolation on feature planes extracted by convolutional layers, or accumulating a vector of values ​​to generate activation values. In some implementations, the vector computation unit 1307 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as activation input to the computation circuit 1303, for example, for use in subsequent layers of the neural network.

[0135] The instruction fetch buffer 1309 connected to the controller 1304 is used to store the instructions used by the controller 1304;

[0136] Unified memory 1306, input memory 1301, weighted memory 1302, and instruction fetch memory 1309 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0137] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the program of the method in Figure 4 above.

[0138] This application also provides a computer-readable storage medium storing a program that, when run on a computer, causes the computer to perform the steps of the method described in the embodiment shown in FIG4 above.

[0139] This application also provides a computer program product that, when run on a computer, causes the computer to perform the method steps described in the embodiment shown in FIG4 above.

[0140] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0141] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, the specific working process of the systems, devices and units described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0142] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0143] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0144] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0145] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0146] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0147] Finally, it should be noted that the above are only specific embodiments of this application, but the protection scope of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application.

Claims

1. An image restoration method, characterized in that, include: A first image and path information are acquired. The first image is an image acquired by a first image sensor. The path information is used to adjust the network structure of the first image restoration network to obtain a second image restoration network or a third image restoration network. The network structures of the second image restoration network and the third image restoration network are different. The second image restoration network is used to restore images of a first video mode or a first video path. The third image restoration network is used to restore images of a second video mode or a second video path. The first video mode and the second video mode are different video modes. The first video path and the second video path are different video paths. The first image is restored by using a target image restoration network to obtain the image restoration result of the first image. The target image restoration network is either the second image restoration network or the third image restoration network.

2. The method according to claim 1, characterized in that, The step of performing image restoration on the first image using a target image restoration network to obtain the image restoration result of the first image includes: The output features are processed by the target image restoration network to obtain the image restoration result. The output features are obtained by concatenating the fused features with the shallow features of the first image. The shallow features are obtained by feature extraction from the first image.

3. The method according to claim 2, characterized in that, The method further includes: A second image and an auxiliary image are acquired. The second image is any frame from the first N frames of the first image, where N is a positive integer. The auxiliary image is an image acquired by a second image sensor. The exposure mode of the auxiliary image is different from that of the first image. The image content of the auxiliary image is the same as that of the first image. The second image and the auxiliary image are used to obtain the fusion feature.

4. The method according to claim 3, characterized in that, The path information includes path exposure information, which is used to indicate the method of fusing the first image, the second image, and the auxiliary image. Before processing the output features through the target image restoration network to obtain the image restoration result, the method further includes: Based on the path exposure information and attention mechanism, the first image, the second image, and the auxiliary image are fused to obtain the fused feature.

5. The method according to claim 4, characterized in that, The step of fusing the first image, the second image, and the auxiliary image based on the path exposure information and the attention mechanism to obtain the fused feature includes: Based on the first image, the second image, and the auxiliary image, a query vector, a key vector, and a value vector are obtained. The query vector is obtained by performing operations on the first image. The key vector is obtained by performing operations on the second image and the auxiliary image. The value vector is obtained by performing operations on the second image and the auxiliary image. According to the attention mechanism, the query vector, the key vector, the value vector, and the correlation matrix mask are processed to obtain the fused features, wherein the correlation matrix mask is determined based on the path exposure information.

6. The method according to any one of claims 1 to 5, characterized in that, The pathway information also includes network structure adjustment information, which indicates the number of feature extraction modules in the first image restoration network. The feature extraction modules are used to extract high-frequency features of the image. Before performing image restoration on the first image through the target image restoration network to obtain the image restoration result of the first image, the method further includes: Based on the network structure adjustment information, the number of feature extraction modules in the first image restoration network is determined to obtain a temporary image restoration network.

7. The method according to claim 6, characterized in that, The path information also includes power consumption information, which is used to indicate the number of convolutional layers and channels in the output layer of the first image restoration network. Before performing image restoration on the first image through the target image restoration network to obtain the image restoration result of the first image, the method further includes: Based on the power consumption information, the number of convolutional layers and the number of channels in the output layer of the temporary image restoration network are determined to obtain the target image restoration network.

8. An image restoration device, characterized in that, include: An acquisition module is used to acquire a first image and path information. The first image is an image acquired by a first image sensor. The path information is used to adjust the network structure of the first image restoration network to obtain a second image restoration network or a third image restoration network. The network structures of the second image restoration network and the third image restoration network are different. The second image restoration network is used to restore images of a first video mode or a first video path. The third image restoration network is used to restore images of a second video mode or a second video path. The first video mode and the second video mode are different video modes, and the first video path and the second video path are different video paths. The recovery module is used to perform image recovery on the first image through a target image recovery network to obtain the image recovery result of the first image, wherein the target image recovery network is either the second image recovery network or the third image recovery network.

9. The apparatus according to claim 8, characterized in that, The recovery module is specifically used for: The output features are processed by the target image restoration network to obtain the image restoration result. The output features are obtained by concatenating the fused features with the shallow features of the first image. The shallow features are obtained by feature extraction from the first image.

10. The apparatus according to claim 9, characterized in that, The acquisition module is further configured to acquire a second image and an auxiliary image. The second image is any frame from the first N frames of the first image, where N is a positive integer. The auxiliary image is an image acquired by a second image sensor. The exposure mode of the auxiliary image is different from that of the first image. The image content of the auxiliary image is the same as that of the first image. The second image and the auxiliary image are used to obtain the fusion feature.

11. The apparatus according to claim 10, characterized in that, The path information includes path exposure information, which is used to indicate the method of fusing the first image, the second image, and the auxiliary image. Before processing the output features through the target image restoration network to obtain the image restoration result, the device further includes: The fusion module is used to fuse the first image, the second image, and the auxiliary image based on the path exposure information and the attention mechanism to obtain the fusion feature.

12. The apparatus according to claim 11, characterized in that, The fusion module is specifically used for: Based on the first image, the second image, and the auxiliary image, a query vector, a key vector, and a value vector are obtained. The query vector is obtained by performing operations on the first image. The key vector is obtained by performing operations on the second image and the auxiliary image. The value vector is obtained by performing operations on the second image and the auxiliary image. According to the attention mechanism, the query vector, the key vector, the value vector, and the correlation matrix mask are processed to obtain the fused features, wherein the correlation matrix mask is determined based on the path exposure information.

13. The apparatus according to any one of claims 8 to 12, characterized in that, The pathway information also includes network structure adjustment information, which indicates the number of feature extraction modules in the first image restoration network. The feature extraction modules are used to extract high-frequency features of the image. Before performing image restoration on the first image through the target image restoration network to obtain the image restoration result of the first image, the device further includes: The determining module is used to determine the number of feature extraction modules in the first image restoration network based on the network structure adjustment information, so as to obtain a temporary image restoration network.

14. The apparatus according to claim 13, characterized in that, The path information also includes power consumption information, which indicates the number of convolutional layers and channels in the output layer of the first image restoration network, prior to the image restoration of the first image using the target image restoration network to obtain the image restoration result of the first image. The determining module is further configured to determine the number of convolutional layers and the number of channels in the output layer of the temporary image restoration network based on the power consumption information, thereby obtaining the target image restoration network.

15. An image restoration apparatus, characterized in that, include: A processor and a memory, wherein the processor is coupled to the memory; The memory is used to store programs; The processor is configured to execute a program in the memory, such that the method as described in any one of claims 1 to 7 is performed.

16. A computer-readable storage medium, characterized in that, Includes instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 7.

17. A computer program product containing instructions, characterized in that, When it is run on a computer, it causes the computer to perform the method as described in any one of claims 1 to 7.

Images