Image processing method, electronic device, storage medium, and program product
By extracting channel images from RAW images and combining them with RGB images to generate training data, the problem of data domain differences in image signal processing models is solved, improving training effectiveness and adaptability, and reducing the probability of artifacts and false colors.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- VERISILICON MICROELECTRONICS (SHANGHAI) CO LTD
- Filing Date
- 2024-12-16
- Publication Date
- 2026-06-25
AI Technical Summary
Existing image signal processing models suffer from data domain differences between training and testing data, resulting in poor image processing performance and problems such as false color, grid-like texture errors, and loss of detail.
The R, Gr, Gb, and B channels are extracted directly from the RAW image, merged to generate the G channel image, and combined with the R, G, and B channel images to generate an RGB image as a training reference image. The RAW image is obtained by exposure using an image sensor, and the training input image is generated through mosaicking and noise addition. The target model is trained directly using the RGB and RAW images.
It reduces the cost of constructing training data, reduces artifacts and false color defects, improves the training effect and adaptability of image signal processing models, and improves the generation quality and efficiency of training data.
Smart Images
Figure CN2024139675_25062026_PF_FP_ABST
Abstract
Description
Image processing methods, electronic devices, storage media and software products Technical Field
[0001] This application relates to the field of image processing technology, and more specifically, to an image processing method, electronic device, storage medium, and program product. Background Technology
[0002] Currently, with the development of image signal processing technology, image signal processing models such as the Demosaic model and the Joint Demosaicing and Denoising (JDD) model are widely used in low-level visual AI (Artificial Intelligence) models. Currently, related technologies generally rely on pixel offset techniques to construct training datasets when training image signal processing models. Referring to Figure 1, the pixel offset technique physically controls the camera sensor to move one pixel horizontally or vertically with each sample to capture all color information for each pixel.
[0003] Pixel offset technology relies on specialized equipment with pixel offset capabilities to acquire high-quality image data. However, the image sensors used in application scenarios may not meet the technical specifications of these specialized devices. Due to hardware differences between the specialized equipment used to capture the training dataset and the image sensors used to capture the test dataset, the image signal processing module trained using the training set has a significant difference in data domain compared to the test data. This results in poor image processing performance from the trained image signal processing model, potentially leading to issues such as false color, grid-like texture errors, large areas of red-green noise, and loss of detail. Summary of the Invention
[0004] This application provides an image processing method, electronic device, storage medium, and program product for improving the image processing effect of an image signal processing model.
[0005] In a first aspect, embodiments of this application provide an image processing method, the method comprising: extracting a first channel image from a first RAW image; wherein the first channel image includes a first R channel image, a first Gr channel image, a first Gb channel image, and a first B channel image; merging the first Gr channel image and the first Gb channel image to obtain a first G channel image; combining the first R channel image, the first G channel image, and the first B channel image to obtain a first RGB image; the first RGB image is used as a training reference image for a first training; the first training is used to represent training on a first target model for de-mosaic processing and denoising processing.
[0006] In the implementation of the above scheme, four channel images are directly extracted from the RAW image. After merging the Gr channel image and Gb channel image to obtain the G channel image, the R channel image, G channel image, and B channel image are directly combined to obtain the RGB image used as the training reference image. On the one hand, compared with the scheme in related technologies that uses the pixel offset function of special devices to construct the training dataset, the above scheme directly extracts channel images from the RAW image and constructs the RGB image used as the training reference image based on the extracted channel images, which helps to reduce the construction cost of training data. On the other hand, compared with the method in related technologies that uses interpolation to recover the RGB image based on the RAW image, the above scheme directly constructs the RGB image based on the RAW image, which can reduce the probability of image artifacts, false color and other defects, which helps to improve the quality of training data generation, and thus improve the training effect of the image signal processing model.
[0007] In one implementation of the first aspect, the method further includes: exposing the shooting scene using an image sensor to obtain multiple frames of second RAW images; merging and averaging the multiple frames of second RAW images to obtain the first RAW image.
[0008] In the implementation of the above scheme, on the one hand, compared with using special equipment with pixel shifting function, the above scheme can directly acquire RAW images by exposing the image sensor, which has a lower cost of data set construction and can avoid data domain differences between training data and test data, which is beneficial to improving the data quality of the training data generated by the above RAW image processing method; on the other hand, by merging multiple frames to obtain almost noise-free RAW images, it is easier to further process the RAW images in subsequent steps, which is beneficial to improving the training data generation efficiency of the above image processing method; furthermore, using the training data generated by the above scheme to train the first target model, such as the AI JDD (Joint Demosaicing and Denoising) model, is beneficial to improving the training effect of the first target model.
[0009] In one implementation of the first aspect, the method further includes: performing mosaic processing and noise addition processing on the first RGB image to obtain a third RAW image; the third RAW image is used as the training input image for the first training.
[0010] In the implementation of the above scheme, the third RAW image that can be used as the training input image for the first training is obtained by directly performing mosaic and noise addition processing on the almost noise-free first RGB image. The method of directly adding noise is more flexible and can select a noise model that is more in line with the real situation according to the specific application scenario. This makes the above image processing method applicable to more application scenarios and helps to improve the adaptability of the above image processing method.
[0011] In one implementation of the first aspect, the method further includes: performing the first training on the first target model using the first RGB image and the third RAW image.
[0012] In the implementation of the above scheme, the first target model can be trained directly using the first RGB image and the third RAW image. The third RAW image can be obtained by performing mosaic and noise addition processing on the first RGB image, which is beneficial to improving the flexibility of training data acquisition and thus improving the adaptability of the above image processing method.
[0013] In one implementation of the first aspect, the first training of the first target model using the first RGB image and the third RAW image includes: retraining the pre-trained first target model using the first RGB image and the third RAW image.
[0014] In the implementation of the above scheme, the first RGB image and the third RAW image can be used to retrain the first target model that has completed pre-training, which is beneficial to improving the training effect of the first target model.
[0015] In one implementation of the first aspect, the method further includes: extracting a second channel image from the second RAW image; wherein the second channel image includes a second R channel image, a second Gr channel image, a second Gb channel image, and a second B channel image; merging the second Gr channel image and the second Gb channel image to obtain a second G channel image; combining the second R channel image, the second G channel image, and the second B channel image to obtain a second RGB image; performing mosaic processing on the second RGB image to obtain a fourth RAW image; the fourth RAW image is used as the training input image for the first training.
[0016] In the implementation of the above scheme, a noisy second RGB image is obtained directly from the noisy second RAW image acquired by the image sensor, and then the noisy second RGB image is directly mosaicked to obtain the training input image for the first training. The noise contained in the second RAW image is more in line with the real situation of the image sensor, and the noise contained in the training input image obtained using the second RAW image is also more in line with the real situation, which is beneficial to improving the training data generation effect of the above image processing method, and thus improving the training effect of the image signal processing model.
[0017] In one implementation of the first aspect, the method further includes: performing the first training on the first target model using the first RGB image and the fourth RAW image.
[0018] In the implementation of the above scheme, the first target model can be trained directly using the first RGB image and the fourth RAW image. The noise contained in the fourth RAW image is real noise. Using the fourth RAW image to train the first target model is beneficial to improving the training effect of the first target model.
[0019] In one implementation of the first aspect, the first training of the first target model using the first RGB image and the fourth RAW image includes: retraining the pre-trained first target model using the first RGB image and the fourth RAW image.
[0020] In one implementation of the first aspect, the method further includes: extracting a first channel image from a first RAW image; wherein the first channel image includes a first R channel image, a first Gr channel image, a first Gb channel image, and a first B channel image; wherein the first RAW image is an image obtained by merging and averaging multiple frames of second RAW images; the second RAW image is an image obtained by exposing the shooting scene using an image sensor; merging the first Gr channel image and the first Gb channel image to obtain a first G channel image; combining the first R channel image, the first G channel image, and the first B channel image to obtain a first RGB image; the first RGB image is used as a training reference image for a second training; the second training is used to represent training on a second target model to perform de-mosaic processing; performing mosaic processing on the first RGB image to obtain a fifth RAW image; the fifth RAW image is used as a training input image for the second training.
[0021] In one implementation of the first aspect, the method further includes: performing a second training on the second target model using the first RGB image and the fifth RAW image.
[0022] In the implementation of the above scheme, the second target model can be trained directly using the first RGB image and the fifth RAW image. The first RGB image and the fifth RAW image used are almost noise-free images, which is beneficial to improving the training effect of the second target model.
[0023] In one implementation of the first aspect, the second training of the second target model using the first RGB image and the fifth RAW image includes: retraining the pre-trained second target model using the first RGB image and the fifth RAW image.
[0024] In one implementation of the first aspect, the method further includes: extracting a second channel image from a second RAW image; wherein the second channel image includes a second R channel image, a second Gr channel image, a second Gb channel image, and a second B channel image; the RAW image is an image obtained by exposing a shooting scene using an image sensor; merging the second Gr channel image and the second Gb channel image to obtain a second G channel image; combining the second R channel image, the second G channel image, and the second B channel image to obtain a second RGB image; the second RGB image is used as a training reference image for a second training; the second training is used to represent training on a second target model to perform de-mosaic processing; performing mosaic processing on the second RGB image to obtain a fourth RAW image; the fourth RAW image is used as a training input image for the second training.
[0025] In one implementation of the first aspect, the method further includes: performing a second training on the second target model using the second RGB image and the fourth RAW image.
[0026] In one implementation of the first aspect, the second training of the second target model using the second RGB image and the fourth RAW image includes: retraining the pre-trained second target model using the second RGB image and the fourth RAW image.
[0027] In one implementation of the first aspect, the step of extracting the first channel image from the first RAW image includes: sampling each channel in the first RAW image based on the Bayer array format of the first RAW image to obtain the first channel image.
[0028] In the implementation of the above scheme, sampling is performed on the four channels of R, Gr, Gb, and B. On the one hand, the collected channel data is lossless real data. Using the collected channel images to obtain training reference images and training input images is beneficial to improving the quality of training data generation, thereby improving the training effect of the image signal processing model. On the other hand, sampling can quickly extract channel images from RAW images, which is beneficial to improving the training data generation efficiency of the above image processing method.
[0029] In one implementation of the first aspect, merging the first Gr channel image and the first Gb channel image to obtain the first G channel image includes: merging and averaging the first Gr channel image and the first Gb channel image to obtain the first G channel image.
[0030] In the implementation of the above scheme, the Gr channel image and the Gb channel image are merged and averaged to obtain the G channel image. On the one hand, compared with the complex channel merging algorithm, the summation and averaging process has lower resource consumption, which is conducive to improving the training data generation efficiency of the above image processing method. On the other hand, the summation and averaging method can improve the clarity and color accuracy of the RGB image, which in turn is conducive to improving the quality of the training data generated by the above image processing method.
[0031] Secondly, embodiments of this application provide an electronic device, including: a processor, a memory, and a communication bus, wherein the processor and the memory communicate with each other through the communication bus; the memory stores computer program instructions that can be executed by the processor, and the computer program instructions are read and executed by the processor to perform the method provided in the first aspect or any possible implementation of the first aspect.
[0032] Thirdly, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when read and executed by a processor, perform the method provided in the first aspect or any possible implementation thereof.
[0033] Fourthly, embodiments of this application provide a computer program product, the computer program product including a computer program, which, when executed by a processor, implements the method provided by the first aspect or any possible implementation of the first aspect.
[0034] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing embodiments of this application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description
[0035] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0036] Figure 1 is a schematic diagram of the process of constructing a training dataset based on pixel offset technology in related technologies;
[0037] Figure 2 is a flowchart illustrating the image processing method applied to the first target model provided in an embodiment of this application;
[0038] Figure 3 is another flowchart illustrating the image processing method applied to the first target model provided in an embodiment of this application.
[0039] Figure 4 is a flowchart illustrating the first RAW image acquisition method provided in an embodiment of this application;
[0040] Figure 5 is a flowchart illustrating another image processing method applied to a second target model provided in an embodiment of this application;
[0041] Figure 6 is a flowchart illustrating the various image acquisition methods provided in the embodiments of this application;
[0042] Figure 7 is a schematic diagram of the structure of the electronic device provided in the embodiment of this application. Detailed Implementation
[0043] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.
[0044] This application provides an image processing method that directly extracts four channel images from a RAW image. After merging the Gr and Gb channel images to obtain the G channel image, it directly combines the R, G, and B channel images to obtain an RGB image used as a training reference image. On the one hand, compared to related technologies that use the pixel offset function of special devices to construct training datasets, the above method directly extracts channel images from the RAW image and constructs an RGB image used as a training reference image based on the extracted channel images, which helps reduce the cost of constructing training data. On the other hand, compared to related technologies that use interpolation to recover RGB images from RAW images, the above method directly constructs RGB images based on RAW images, which can reduce the probability of image artifacts, false colors, and other defects, which helps improve the quality of training data generation and thus improve the training effect of the image signal processing model. Furthermore, using the training data generated by the above method to train a first target model, such as an AI JDD (Joint Demosaicing and Denoising) model, helps improve the training effect of the first target model.
[0045] The image processing method described above is described below. Referring to Figures 2 and 3, this embodiment of the application provides an image processing method applied to a first target model, the method comprising:
[0046] Step S110: Extract the first channel image from the first RAW image; wherein the first channel image includes the first R channel image, the first Gr channel image, the first Gb channel image, and the first B channel image;
[0047] Step S120: Merge the first Gr channel image and the first Gb channel image to obtain the first G channel image;
[0048] Step S130: Combine the first R channel image, the first G channel image, and the first B channel image to obtain the first RGB image; the first RGB image is used as the training reference image for the first training; the first training is used to represent the training of the first target model for de-mosaic processing and denoising processing.
[0049] The aforementioned RAW images refer to unprocessed raw image files output by an image sensor. RAW images are uncompressed and unprocessed in color, preserving all the details captured by the image sensor. Each pixel in a RAW image can only record one of the three color information: R (red), G (green), and B (blue). RAW image data formats generally use the Bayer arrangement. Since the human eye is more sensitive to the green band, the green component has a larger weight in the Bayer arrangement. The ratio of R, G, and B components in the Bayer arrangement is generally R:G:B = 1:2:1, and the arrangement formats are generally GBRG, GRBG, BGGR, and RGGB.
[0050] The RGB format used in the above RGB images is an industry-standard color standard. This standard obtains various colors by varying the red (R), green (G), and blue (B) color channels and superimposing them. Unlike RAW images, where each pixel can only record one color, the color of each pixel in an RGB image can be determined by superimposing the R, G, and B colors.
[0051] Furthermore, it is understood that in step S130 above, since the R, G, and B channels are directly combined to obtain the RGB image without image restoration steps such as interpolation, the resolution of the obtained RGB image will be lower than that of the RAW image. In this embodiment, generally, a group of pixels in the RAW image corresponds to a pixel in the RGB image, and the resolution of the obtained RGB image is generally half that of the RAW image. Although the resolution of the RGB image is lower than that of the RAW image, compared with the interpolation scheme used in related technologies, the RGB image obtained in step S130 above is more realistic and also reduces the probability of image artifacts, false colors, and other defects, which is beneficial to improving the data quality of the training data generated by the above image processing method.
[0052] Understandably, the trained first target model possesses both demosaicing and denoising capabilities. This first target model can be an AI JDD (Joint Demosaicing and Denoising) model, with the input image being a RAW image and the output image being a denoised RGB image. Therefore, the almost noise-free RGB image obtained in step S130 can be used as a training reference image for the first target model.
[0053] The following describes the method for extracting the first channel image from the first RAW image in step S110:
[0054] It is understandable that, since acquiring channel information of RAW images using pixel offset technology requires high data acquisition costs and will result in data domain differences between training and test data, the embodiments of this application can extract channel images from RAW images through at least the following methods:
[0055] The first implementation method is to directly extract the four channels (R, Gr, Gb, and B) from the RAW image;
[0056] It is understandable that since the data arrangement of RAW images generally follows the Bayer array pattern and has a certain arrangement rule, the four channels R, Gr, Gb and B can be directly separated from the RAW image by using the array pattern adopted by the RAW image.
[0057] The second implementation method is to use an image processing library to extract the four channels (R, Gr, Gb, and B) from the RAW image;
[0058] Use image processing libraries such as OpenCV (Open Source Computer Vision Library) and PIL (Python Imaging Library) that provide RAW image processing capabilities to extract the four channels of R, Gr, Gb and B from RAW images.
[0059] The third implementation method is based on the Bayer array format of RAW images, sampling each channel image from the RAW image;
[0060] It is understandable that, since the data arrangement of RAW images has a certain arrangement pattern, the channel corresponding to each pixel in the image can be determined after the Bayer array format of the RAW image is determined. Taking the first RAW image shown in Figure 3 as an example, the Bayer array format of this RAW image is BGGR format. Among them, the G channel in the same row as the B channel is the Gb channel, and the G channel in the same row as the R channel is the Gr channel. The sampling method of the four channels can be: B = RAW1[0::2,0::2]; Gb = RAW1[0::2,1:2]; Gr = RAW1[1::2,0::2]; B = RAW1[1::2,1::2].
[0061] The above scheme samples the four channels of the image (R, Gr, Gb, B) through sampling. On the one hand, the collected channel data is lossless real data. Using the collected channel images to obtain training reference images and training input images helps improve the quality of training data generation, thereby improving the training effect of the image signal processing model. On the other hand, sampling can quickly extract channel images from RAW images, which helps improve the training data generation efficiency of the above image processing method.
[0062] For example, the above scheme can determine the Bayer array format of the RAW image by one of the following methods: (1) by querying the specification sheet of the image sensor that captured the RAW image; (2) by using the metadata of the RAW image to determine the Bayer array format of the RAW image.
[0063] The following describes the method for merging the Gr channel image and the Gb channel image in step S120 above to obtain the G channel image:
[0064] As an optional implementation of the above image processing method, step S120 includes: merging and averaging the first Gr channel image and the first Gb channel image to obtain the first G channel image. For example, this implementation includes:
[0065] Taking the first RAW image shown in Figure 3 as an example, for each BGGR pixel group in the RAW image The matrix form of the Gb channel image corresponding to the pixel group is as follows The matrix form of the Gr channel image is
[0066] The process of merging and averaging the Gr channel and Gb channel images of this pixel group refers to... The Gb value in The Gr values in the image are summed and averaged, and then the resulting value is assigned to the Gr values in the original image. The Gb channel and The Gr channel in the image is used to obtain two G-channel images, which can be represented as follows: and
[0067] The above scheme obtains the G channel image by merging and averaging the Gr channel image and the Gb channel image. On the one hand, compared with the complex channel merging algorithm, the summation and averaging process has lower resource consumption, which is beneficial to improving the training data generation efficiency of the above image processing method. On the other hand, the summation and averaging method can improve the clarity and color accuracy of the RGB image, thereby improving the quality of the training data generated by the above image processing method.
[0068] For example, in addition to summing and averaging, weighted averaging (the weights can be determined based on factors such as lighting conditions and image sensor characteristics) and median averaging can also be used to merge the Gr and Gb channels.
[0069] The following describes the scheme for obtaining the first RAW image according to the embodiments of this application:
[0070] As an optional implementation of the above image processing method, the image processing method further includes: exposing the shooting scene using an image sensor to acquire multiple frames of second RAW images; merging and averaging the multiple frames of second RAW images to acquire a first RAW image. An example of this implementation is:
[0071] Please refer to Figure 4. The image sensor is used to perform long exposure on the shooting scene to obtain multiple sets of second RAW images, each set containing multiple RAW images. Then, for each set of second RAW images, the multiple RAW images in the set are merged and averaged to obtain the first RAW image.
[0072] Understandably, the noise contained in RAW images acquired by image sensors generally has zero mean. Noise reduction of RAW images can be achieved by merging multiple frames of second RAW images and taking the average value, and the resulting first RAW image is almost noise-free.
[0073] On the one hand, compared to using special equipment with pixel shifting capabilities, the above scheme can directly acquire RAW images by exposing the image sensor, resulting in lower dataset construction costs and avoiding data domain differences between training and test data, which is beneficial for improving the data quality of the training data generated by the above image processing method. On the other hand, by merging multiple frames to obtain almost noise-free RAW images, it is easier to further process the RAW images in subsequent steps, which is beneficial for improving the training data generation efficiency of the above image processing method.
[0074] The following describes the method for obtaining the training input images for the first training iteration:
[0075] The aforementioned first training refers to training the first target model by performing de-mosaic and denoising processing. Therefore, the input image of the first target model is a noisy RAW image, and the output image is a noise-free RGB image. The almost noise-free first RGB noise obtained in step S130 can be used as the training reference image for the first training, while the noisy RAW image can be obtained through the following two implementation methods:
[0076] The first implementation method involves directly performing mosaicking and noise-adding processing on the first RGB image, which is almost noise-free, to obtain the training input image. This scheme is specifically as follows:
[0077] As an optional implementation of the above image processing method, the image processing method further includes: performing mosaic processing and noise addition processing on the first RGB image to obtain a third RAW image, wherein the third RAW image is used as the training input image for the first training.
[0078] It is understood that the aforementioned mosaic processing refers to converting an RGB image, where each pixel originally contains three color channels (R, G, B), into a RAW image where each pixel represents only a single color channel, using a specific algorithm or technique. Mosaic processing technology is a relatively mature technology in this field; for specific implementation schemes, please refer to relevant technologies, and this application's embodiments will not elaborate further.
[0079] In addition, it is understandable that during the process of converting the first RAW image into the first RGB image, the resolution of the first RGB image is lower than that of the first RAW image. However, the resolution of the third RAW image obtained after mosaic processing is the same as that of the first RGB image. Using the first RGB image and the third RAW image with the same resolution as the training reference image and the training input image, respectively, to train the first target model is beneficial to improving the training effect of the first target model.
[0080] For example, the above scheme can use a long-exposure RAW image as the second RAW image. Compared to the short-exposure method, the long-exposure method can provide a clearer RAW image with less noise, which is beneficial to improving the training effect of the first target model.
[0081] For example, the above scheme can use a Gaussian-Poisson noise model to add noise to the first RGB image. In addition, since different digital gain settings during the shooting process will affect the noise intensity contained in the captured image, the image sensor can be calibrated in advance to obtain a noise model that is more in line with the actual situation of the image sensor. Then, the noise model can be used to add noise to the RGB image, which is beneficial to improving the training effect of the first target model.
[0082] The above scheme directly performs mosaic and noise addition processing on the almost noise-free first RGB image to obtain a third RAW image that can be used as the training input image for the first training. The method of directly adding noise is more flexible and can select a noise model that is more in line with the real situation according to the specific application scenario. This makes the above image processing method applicable to more application scenarios and helps to improve the adaptability of the above image processing method.
[0083] As an optional implementation of the above image processing method, the image processing method further includes: performing a first training on the first target model using the first RGB image and the third RAW image.
[0084] The above scheme can directly use the first RGB image and the third RAW image to perform the first training on the first target model. The third RAW image can be obtained by performing mosaic and noise addition processing on the first RGB image, which is beneficial to improving the flexibility of training data acquisition and thus improving the adaptability of the above image processing method.
[0085] As an optional implementation of the above image processing method, the first training of the first target model using the first RGB image and the third RAW image includes: retraining the pre-trained first target model using the first RGB image and the third RAW image.
[0086] Understandably, the above retraining can be considered fine-tuning.
[0087] The above scheme can use the first RGB image and the third RAW image to retrain the pre-trained first target model, which is beneficial to improving the training effect of the first target model.
[0088] The second implementation method involves obtaining the training input image using a noisy second RAW image. Specifically, this method is as follows:
[0089] As an optional implementation of the above image processing method, the image processing method further includes: extracting a second channel image from the second RAW image; wherein the second channel image includes a second R channel image, a second Gr channel image, a second Gb channel image, and a second B channel image; merging the second Gr channel image and the second Gb channel image to obtain a second G channel image; combining the second R channel image, the second G channel image, and the second B channel image to obtain a second RGB image; performing mosaic processing on the second RGB image to obtain a fourth RAW image; the fourth RAW image is used as the training input image for the first training.
[0090] It is understood that for the specific implementation of the above-mentioned second channel image extraction step, second G channel image acquisition step, and second RGB image acquisition step, please refer to the relevant descriptions of the first channel image extraction step, first G channel acquisition step, and first RGB image acquisition step in the above description, which will not be repeated in the embodiments of this application.
[0091] The above scheme directly obtains a noisy second RGB image from a noisy second RAW image acquired by an image sensor, and then directly performs mosaic processing on the noisy second RGB image to obtain the training input image for the first training. The noise contained in the second RAW image is more in line with the real situation of the image sensor, and the noise contained in the training input image obtained using the second RAW image is also more in line with the real situation, which is beneficial to improving the training data generation effect of the above image processing method, and thus improving the training effect of the image signal processing model.
[0092] As an optional implementation of the above image processing method, the above image processing method further includes: performing a first training on the first target model using the first RGB image and the fourth RAW image.
[0093] The above scheme can directly use the first RGB image and the fourth RAW image to train the first target model. The noise contained in the fourth RAW image is real noise. Using the fourth RAW image to train the first target model is beneficial to improving the training effect of the first target model.
[0094] As an optional implementation of the above image processing method, the first training of the first target model using the first RGB image and the fourth RAW image includes: retraining the pre-trained first target model using the first RGB image and the fourth RAW image.
[0095] It is understood that when acquiring the training input image for the first training, one of the first and second implementation methods described above may be used, or both implementation methods may be used.
[0096] The above scheme describes the training dataset generation method for the first objective model. The following describes the training dataset generation method for the second objective model:
[0097] Understandably, the trained second-objective model described above possesses de-mosaic capabilities. This second-objective model can be an AIDemosaic model, with its input image being a noise-free RAW image and its output image being a noise-free RGB image. Of course, in some scenarios, the input image of the second-objective model can also be a noisy RAW image, in which case the output image can be a noisy RGB image. The following section introduces the construction schemes for the training datasets of the second-objective model in these two scenarios:
[0098] Scenario 1: The input and output images of the second target model are a noise-free RAW image and a noise-free RGB image, respectively.
[0099] As an optional implementation of the above image processing method, the image processing method further includes:
[0100] Step S210: Mosaic processing is performed on the first RGB image to obtain the fifth RAW image; the fifth RAW image is used as the training input image for the second training; the first RGB image is used as the training reference image for the second training; the second training is used to represent the training of the second target model after desasaic processing.
[0101] As an optional implementation of the above image processing method, the image processing method further includes: performing a second training on the second target model using the first RGB image and the fifth RAW image.
[0102] The above scheme can directly use the first RGB image and the fifth RAW image to perform the second training on the second target model. The first RGB image and the fifth RAW image used are almost noise-free images, which is beneficial to improving the training effect of the second target model.
[0103] As an optional implementation of the above image processing method, the second training of the second target model using the first RGB image and the fifth RAW image includes: retraining the pre-trained second target model using the first RGB image and the fifth RAW image.
[0104] The second scenario: The input and output images of the second target model are a noisy RAW image and a noisy RGB image, respectively.
[0105] Please refer to Figure 5. As an optional implementation of the above image processing method, the image processing method further includes:
[0106] Step S310: Extract the second channel image from the second RAW image; wherein, the second channel image includes the second R channel image, the second Gr channel image, the second Gb channel image, and the second B channel image; the RAW image is an image obtained by exposing the shooting scene using an image sensor;
[0107] Step S320: Merge the second Gr channel image and the second Gb channel image to obtain the second G channel image;
[0108] Step S330: Combine the second R channel image, the second G channel image, and the second B channel image to obtain the second RGB image; the second RGB image is used as the training reference image for the second training; the second training is used to represent the training of the second target model for de-mosaic processing;
[0109] Step S340: Mosaic processing is performed on the second RGB image to obtain the fourth RAW image; the fourth RAW image is used as the training input image for the second training.
[0110] Please refer to Figure 6. For ease of understanding, this application embodiment provides a focused explanation of the source and uses of the second RAW image acquired using an image sensor and the various images acquired after processing the second RAW image:
[0111] (1) Second RAW image: acquired by the image sensor after exposure, the second RAW image contains noise;
[0112] (2) First RAW image: obtained by merging multiple RAW images contained in each group of the second RAW image. The first RAW image is almost noise-free.
[0113] (3) First RGB image: The image obtained after performing channel image extraction, G channel acquisition and RGB image construction on the first RAW image. The first RGB image is almost noise-free.
[0114] (4) Third RAW image: The image obtained after mosaicking and noise addition to the first RGB image. The third RAW image contains noise.
[0115] (5) Fifth RAW image: The image obtained after mosaicking the first RGB image. The fifth RAW image is almost noise-free.
[0116] (6) Second RGB image: The image obtained after performing channel image extraction, G channel acquisition and RGB image construction on the second RAW image. The second RGB image contains noise.
[0117] (7) Fourth RAW image: The image obtained after mosaicking the second RGB image. The fourth RAW image contains noise.
[0118] The following table describes the training reference images and training input images that can be used for the first and second target models. Please refer to Table 1 for details:
[0119] Table 1. Training reference images and training input images that the target model can use.
[0120] The pre-training methods for the first and second target models in the above scheme are described below:
[0121] For example, the pre-training of the first target model and the second target model can be based on the SenseNoise-500 dataset. The SenseNoise-500 dataset is an open-source dataset widely used for training and testing AI denoising models. It is constructed by using an image sensor to take multiple RAW images of the same scene with long exposure, and obtaining the corresponding almost noise-free RAW image by adding the multiple RAW images together and taking the average.
[0122] Understandably, in the above approach, training the target model using only the training data generated by the aforementioned image processing methods yields better results in terms of texture detail and overall color cast compared to training with the PixelShift200 dataset (an open-source image dataset that provides noise-free real-world images). Furthermore, fine-tuning the pre-trained target model using the training data generated by the aforementioned image processing methods further enhances the training performance. Tests show that by combining pre-training with fine-tuning, the target model can eliminate most of the false color and mesh-like texture errors in the test images, and texture detail is also improved to some extent.
[0123] Furthermore, regarding the cost of training data construction and model training, the SenseNoise-500 dataset used in the aforementioned pre-training and retraining (fine-tuning) methods is open-source, meaning that acquiring the pre-training dataset incurs almost no cost. The RAW images required by the aforementioned image processing methods can be directly acquired using image sensors; for example, multiple sets of multi-frame RAW images under different brightness conditions can be acquired using an image sensor. After pre-training, model fine-tuning can be completed in approximately 600 iterations. Both the cost of training dataset construction and model training are superior to image signal processing model training schemes in related technologies.
[0124] Figure 7 is a schematic diagram of an electronic device provided in an embodiment of this application. Referring to Figure 7, the electronic device 400 includes a processor 410, a memory 420, and a communication interface 430. These components are interconnected and communicate with each other through a communication bus 440 and / or other forms of connection mechanisms (not shown).
[0125] The memory 420 includes one or more (only one is shown in the figure), which may be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The processor 410 and other possible components may access the memory 420 to read and / or write data therein.
[0126] Processor 410 includes one or more (only one is shown in the figure), which can be an integrated circuit chip with signal processing capabilities. The processor 410 can be a data processing core of a GPU (Graphics Processing Unit), CPU (Central Processing Unit), AI (Artificial Intelligence), NPU (Neural Network Processing Unit), ISP (Image Signal Processor), DPU (Display Processing Unit), VPU (Video Processing Unit), or DSP (Digital Signal Processor), or a processor chip applied to scenarios such as large-scale data computation. The above are merely examples and should not be construed as limiting this application.
[0127] Communication interface 430 includes one or more (only one is shown in the figure) and can be used to communicate directly or indirectly with other devices to exchange data. For example, communication interface 430 can be an Ethernet interface; it can be a mobile communication network interface, such as an interface for 3G, 4G, or 5G networks; or it can be other types of interfaces with data transmission and reception functions.
[0128] One or more computer program instructions may be stored in the memory 420. The processor 410 may read and run these computer program instructions to implement the image processing method and other desired functions or model training method and other desired functions provided in the embodiments of this application.
[0129] It is understood that the structure shown in Figure 7 is for illustrative purposes only, and the electronic device 400 may include more or fewer components than shown in Figure 7, or have a different configuration than shown in Figure 7. The components shown in Figure 7 may be implemented using hardware, software, or a combination thereof. For example, the electronic device 400 may be a single server (or other device with computing power), a combination of multiple servers, a cluster of a large number of servers, etc., and may be either a physical device or a virtual device.
[0130] This application also provides a computer-readable storage medium storing computer program instructions. These instructions are read and executed by a computer's processor to perform the image processing method and other desired functions or model training methods provided in this application. For example, the computer-readable storage medium can be implemented as the memory 420 in the electronic device 400 in FIG7.
[0131] In addition, it should be noted that the image processing method applied to the first target model, the image processing method applied to the second target model, the model training method of the first target model, and the model training method of the second target model can be deployed in the same electronic device or in different electronic devices.
[0132] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.
[0133] Furthermore, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0134] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.
Claims
1. An image processing method, characterized in that, The method includes: Extract a first channel image from the first RAW image; wherein the first channel image includes a first R channel image, a first Gr channel image, a first Gb channel image, and a first B channel image; Merge the first Gr channel image and the first Gb channel image to obtain the first G channel image; A first RGB image is obtained by combining the first R channel image, the first G channel image, and the first B channel image; the first RGB image is used as a training reference image for the first training; the first training is used to represent the training of the first target model for de-mosaic processing and denoising processing.
2. The image processing method according to claim 1, characterized in that, The method further includes: The image sensor is used to expose the shooting scene and acquire multiple frames of second RAW images; The first RAW image is obtained by merging and averaging multiple frames of the second RAW image.
3. The image processing method according to claim 2, characterized in that, The method further includes: The first RGB image is subjected to mosaic and noise reduction processing to obtain a third RAW image; the third RAW image is used as the training input image for the first training.
4. The image processing method according to claim 3, characterized in that, The method further includes: The first target model is trained using the first RGB image and the third RAW image.
5. The image processing method according to claim 4, characterized in that, The first training of the first target model using the first RGB image and the third RAW image includes: The first target model, which has completed pre-training, is retrained using the first RGB image and the third RAW image.
6. The image processing method according to claim 2, characterized in that, The method further includes: Extract a second channel image from the second RAW image; wherein the second channel image includes a second R channel image, a second Gr channel image, a second Gb channel image, and a second B channel image; Merge the second Gr channel image and the second Gb channel image to obtain the second G channel image; By combining the second R channel image, the second G channel image, and the second B channel image, a second RGB image is obtained; The second RGB image is mosaicked to obtain a fourth RAW image; the fourth RAW image is used as the training input image for the first training.
7. The image processing method according to claim 6, characterized in that, The method further includes: The first target model is trained using the first RGB image and the fourth RAW image.
8. The image processing method according to claim 7, characterized in that, The first training of the first target model using the first RGB image and the fourth RAW image includes: The first target model, which has completed pre-training, is retrained using the first RGB image and the fourth RAW image.
9. The image processing method according to claim 1, characterized in that, The method further includes: The first RGB image is mosaicked to obtain a fifth RAW image; the fifth RAW image is used as the training input image for the second training; the first RGB image is used as the training reference image for the second training; the second training is used to represent the training of the second target model after mosaicking.
10. The image processing method according to claim 9, characterized in that, The method further includes: The second target model is trained using the first RGB image and the fifth RAW image.
11. The image processing method according to claim 10, characterized in that, The second training of the second target model using the first RGB image and the fifth RAW image includes: The second target model, which has completed pre-training, is retrained using the first RGB image and the fifth RAW image.
12. The image processing method according to claim 1, characterized in that, The method further includes: Extract the second channel image from the second RAW image; wherein the second channel image includes the second R channel image, the second Gr channel image, the second Gb channel image, and the second B channel image; the RAW image is an image obtained by exposing the shooting scene using an image sensor; Merge the second Gr channel image and the second Gb channel image to obtain the second G channel image; By combining the second R-channel image, the second G-channel image, and the second B-channel image, a second RGB image is obtained; the second RGB image is used as a training reference image for the second training; the second training is used to represent the training of the second target model for de-mosaic processing; The second RGB image is mosaicked to obtain a fourth RAW image; the fourth RAW image is used as the training input image for the second training.
13. The image processing method according to claim 12, characterized in that, The method further includes: The second target model is trained a second time using the second RGB image and the fourth RAW image.
14. The image processing method according to claim 13, characterized in that, The second training of the second target model using the second RGB image and the fourth RAW image includes: The second target model, which has completed pre-training, is retrained using the second RGB image and the fourth RAW image.
15. The image processing method according to any one of claims 1 to 14, characterized in that, Extracting the first channel image from the first RAW image includes: Based on the Bayer array format of the first RAW image, each channel in the first RAW image is sampled to obtain the first channel image.
16. The image processing method according to any one of claims 1 to 14, characterized in that, The step of merging the first Gr channel image and the first Gb channel image to obtain the first G channel image includes: The first Gr channel image and the first Gb channel image are merged and averaged to obtain the first G channel image.
17. An electronic device, characterized in that, include: A processor, a memory, and a communication bus, wherein the processor and the memory communicate with each other via the communication bus; The memory stores program instructions that can be executed by the processor, and the processor can execute the method as described in any one of claims 1 to 16 by calling the program instructions.
18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a computer, cause the computer to perform the method as described in any one of claims 1 to 16.
19. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 16.