Image processing method and image processing device
By dividing images into overlapping regions based on blur information and using a machine learning model to transform blur within each region, the method addresses discontinuities at boundaries, achieving smooth and efficient blur transformation.
Patent Information
- Application Number
- PCT/JP2026/009096
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-13
- Filing Date
- 2026-03-10
- Publication Date
- 2026-11-19
AI Technical Summary
Existing image processing methods using machine learning models to transform blur in captured images often result in discontinuous blur at the boundaries between adjacent image regions due to inappropriate overlap widths, leading to unfavorable blur effects.
The method involves dividing the captured image into overlapping image regions based on blur information, setting the overlap width to be greater than the size of the blur, and using a machine learning model to transform or add blur to each region, ensuring continuous blur transitions.
This approach allows for favorable transformation and addition of blur in images while reducing processing load, resulting in smooth and accurate blur transitions across image boundaries.
Smart Images

Figure JP2026009096_19112026_PF_FP_ABST
Abstract
Description
Image Processing Method and Image Processing Apparatus
[0001] The present disclosure relates to an image processing method that performs processing related to blur on an image generated by imaging.
[0002] Patent Document 1 discloses image processing that uses a machine learning model to transform the shape of blur caused by defocus of a captured image (defocus blur).
[0003] Japanese Unexamined Patent Application Publication No. 2021-56678
[0004] In the image processing as described above, it is preferable to divide the captured image into a plurality of image regions and perform processing for each image region, but favorable blur may not be obtained at the boundaries between adjacent image regions in some cases.
[0005] An image processing method as one aspect of the present disclosure includes a dividing step of dividing a first image generated by imaging into a plurality of adjacent first image regions that overlap each other by a first width in a first direction, and a processing step of performing processing for transforming the shape of blur or adding blur on at least one of the plurality of first image regions. The dividing step is characterized in that the first width is set based on information related to blur in the above processing. Note that a program that causes a computer to execute processing according to the above image processing method also constitutes another aspect of the present disclosure.
[0006] Further, an image processing apparatus as another aspect of the present disclosure includes a dividing unit configured to divide a first image generated by imaging into a plurality of adjacent first image regions that overlap each other by a first width in a first direction, and a processing unit configured to perform processing for transforming the shape of blur or adding blur on at least one of the plurality of first image regions. The dividing unit is characterized in that it sets the first width based on information related to blur in the above processing.
[0007] According to the present disclosure, transformation of the blur shape and addition of blur to the first image can be performed favorably.
[0008] This is a diagram showing the configuration of the machine learning model in Example 1. This is a diagram explaining the problem. This is a diagram explaining the magnitude of blur. This is a block diagram showing the configuration of the image processing system in Example 1. This is an external view of the image processing system in Example 1. This is a flowchart showing the training data generation process in Example 1. This is a flowchart showing the weight learning process in Examples 1 and 2. This is a flowchart showing the estimated image generation process in Example 1. This is a flowchart showing the captured image editing process in Example 1. This is a diagram showing the user interface in Example 1. This is a diagram explaining the number of overlapping pixels in Example 1. This is a diagram explaining the image region synthesis process in Example 1. This is a diagram showing the point image intensity distribution at the defocus distance in Example 1. This is a block diagram showing the configuration of the image processing system in Example 2. This is an external view of the image processing system in Example 2. This is a flowchart showing the training data generation process in Example 2. This is a flowchart showing the estimated image generation process in Example 2. This is a diagram explaining blur in Example 2. This is a diagram showing the user interface in Example 2. This is a diagram explaining the brightness saturation map and the number of overlapping pixels in Example 2. This is a block diagram showing the configuration of the image processing system in Example 3. This is an external view of the image processing system in Example 3. This is a flowchart showing the estimated image generation process in Example 3. This is a diagram showing the user interface in Example 3.
[0009] The following describes the examples with reference to the drawings.
[0010] First, an overview of each embodiment will be described. In each embodiment, while suppressing the processing load, a second image is generated by transforming the shape of the blur contained in the captured image, which is the first image generated by imaging, or by adding blur to the captured image. The blur to be transformed or added includes blur at the in-focus plane and defocus blur, which is blur at the out-of-focus plane (blur due to defocusing).
[0011] The conversion of defocus bokeh means, for example, converting from double-line bokeh to Gaussian bokeh or spherical bokeh, and does not include processing that restores frequency components of the subject that have been reduced or lost due to the bokeh, such as bokeh sharpening. Details of the various defocus bokeh shapes will be described later. Other examples of defocus bokeh that can be converted include defocus bokeh caused by vignetting, and ring-shaped defocus bokeh caused by pupil occlusion by catadioptric lenses, etc. There are no restrictions on the shape of the defocus bokeh to be converted, nor on the shape of the defocus bokeh after conversion.
[0012] When processing captured images using a machine learning model, it is preferable to divide the captured image into multiple image regions for processing, from the perspective of processing load. This is because processing the entire captured image with a large number of pixels at once using a machine learning model requires a large amount of memory. Each image region has fewer pixels than the captured image. Two adjacent image regions are set to overlap by a predetermined overlap width in the direction in which they are adjacent (the first direction; hereinafter referred to as the adjacent direction). That is, the captured image is divided into multiple adjacent first image regions that overlap by an overlap width in the adjacent direction. The number of pixels included in the adjacent direction, which is a one-dimensional direction in terms of the overlap width, is also called the number of overlapping pixels. Details of the overlap width and the number of overlapping pixels will be described later.
[0013] Machine learning models include, for example, CNNs, GANs (Generative Adversarial Networks), and diffusion models. However, if the overlap width between adjacent image regions is not set appropriately, the blur after transformation or application may become discontinuous at the boundaries between adjacent image regions.
[0014] This discontinuity in blurring will be explained using Figures 2(A) to 2(C). In Figure 2(A), the topmost dashed line 1111 in the first row shows image region 1 extracted from the captured image, and the dashed line 1112 shows image region 2 extracted from the captured image. Image regions 1 and 2 are adjacent to each other in the adjacent direction. The black circle 1110 indicates a high-luminance subject, and the surrounding gray area 1113 shows the blurring before conversion that acted on the high-luminance subject. The dashed line 1114 in the second row shows image region 1 after the blurring has been converted, and the dashed line 1116 in the third row shows image region 2 after the blurring has been converted. The light gray area 1115 in the second row and the dark gray area 1117 in the third row show the blurring after conversion.
[0015] The fourth row shows a composite image 1119, which is obtained by combining image region 1 and image region 2 after blur conversion. Image regions 1 and 2 are combined such that parts of them overlap with each other. The dotted line 1118 indicates the position (boundary) of the combination of image regions 1 and 2. These are the same for Figures 2(B) and (C). Note that the light gray circle 1113 is also indicated in the converted image region.
[0016] Figure 2(A) shows the case where the blur after conversion is small in image regions 1 and 2, while Figures 2(B) and (C) show the case where the blur after conversion is large. Furthermore, Figures 2(A) and (B) show the case where the overlap width (number of overlapping pixels) 1121 is small, while Figure 2(C) shows the case where the overlap width 1122 is large.
[0017] The size of the bokeh will be explained using Figures 3(A) and 3(B). Figure 3(A) shows a cross-section of the Point Spread Function (PSF) of Gaussian bokeh. Figure 3(B) shows a cross-section of the PSF of spherical bokeh. The shape of each type of bokeh will be described later. The horizontal axis represents spatial coordinates (position), and the vertical axis represents intensity.
[0018] The magnitude of blur refers to the range in which the PSF has intensity. In Figure 3(A), the magnitude of blur is 1211, and in Figure 3(B), the magnitude of blur is 1212. In Figures 3(A) and (B), the range in which the PSF has intensity is defined as the range in which the PSF has intensity of 0 or greater, but the lower limit of the range in which the PSF has intensity may be set to a value other than 0. For example, the range in which the PSF has intensity may be defined as the range in which the PSF has intensity of 0.01% or greater of the peak intensity of the PSF.
[0019] As shown in Figure 2(A), even if a high-luminance subject exists in only one of the image regions 1 and 2, if the converted blur 1117 does not exceed the composite position 1118, the converted blur will not be discontinuous at the boundary between image regions 1 and 2.
[0020] In contrast, as shown in Figure 2(B), if a high-luminance subject exists in only one of the image regions 1 and 2, and the resulting blur exceeds the composite position 1118, the resulting blur may be discontinuous at the boundary between image regions 1 and 2.
[0021] In this case, as shown in Figure 2(C), the overlap width 1122 must be made larger than the overlap width 1121 in Figures 2(A) and (B) so that the blur after conversion does not exceed the composite position 1118. In other words, when compositing image regions 1 and 2 at the center of the overlap width, it is preferable to set the overlap width to be greater than or equal to the size of the blur. The compositing method for image regions 1 and 2 shown here is an example, and other compositing methods will be described later. Note that although the case where the adjacent direction is horizontal has been explained here, the same applies when the adjacent direction is vertical.
[0022] In each embodiment, the overlap width is determined (set) based on information regarding the blur in the captured image after conversion or the blur applied to the captured image. This makes it possible to generate a second image in which the shape of the blur in the captured image has been converted or the blur has been applied to the captured image, while suppressing the processing load.
[0023] In the image processing of each embodiment, a machine learning model is used, but it is not necessary to use a machine learning model. In the following, the stage of learning the weights of the machine learning model is referred to as the learning phase, and the stage of executing processing with the machine learning model using the learned weights is referred to as the estimation phase. Furthermore, the image region before the shape of the blur is transformed or blur is applied is referred to as the first image region, and the image region after the shape of the blur is transformed or blur is applied is referred to as the second image region. In addition, the first range is defined as the area in which two adjacent first image regions have a first overlap width (first width) in the adjacent direction and a height or width in a direction perpendicular to this. Similarly, the second range is defined as the area in which two adjacent second image regions have a second overlap width (second width) in the adjacent direction and a vertical height or horizontal width in a direction perpendicular to this.
[0024] In Example 1, the task performed by the machine learning model is the transformation of defocus blur on out-of-focus surfaces.
[0025] Figure 4 shows the configuration of the image processing system 100. Figure 5 shows the external appearance of the image processing system 100. The image processing system 100 includes a learning device 101, an imaging device 102, an image estimation device 103 as an image processing device, a display device 104, a recording medium 105, an output device 106, and a network 107.
[0026] The learning device 101 includes a storage unit 101a, an acquisition unit 101b, a generation unit 101c, and an update unit 101d, and learns the weights of a machine learning model used for defocus blur conversion. Details of weight learning and the defocus blur conversion process using the weights will be described later.
[0027] The imaging device 102 has an optical system 102a and an image sensor 102b, and captures an image of the subject space to generate (acquire) an image. The optical system 102a collects light incident from the subject space and forms an optical image (subject image). The image sensor 102b acquires the image by photoelectric conversion of the optical image. The image sensor 102b is, for example, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal-Oxide Semiconductor) sensor.
[0028] The image estimation device 103 includes a storage unit 103a, an acquisition unit 103b, a blur conversion unit 103c, and a generation unit 103d. The image estimation device 103 generates an estimated image (second image) by converting the defocus blur of the captured image (or a part thereof) which is a first image generated by imaging by the imaging device 102. For defocus blur conversion, a machine learning model using weights trained by the learning device 101 is used.
[0029] The learning device 101 and the image estimation device 103 are connected by a network 107, and the image estimation device 103 reads the learned weight information from the learning device 101 either during or before defocus blur conversion.
[0030] The estimated image is output to either the display device 104, the recording medium 105, or the output device 106. The display device 104 is, for example, a liquid crystal display or a projector. The user can perform editing work, etc., while checking the image in progress via the display device 104. Details of the user interface during editing work will be described later.
[0031] The recording medium 105 is, for example, a semiconductor memory, a hard disk, or a server on a network, and stores the estimated image. The output device 106 is a printer or the like.
[0032] First, the training data generation process performed by the learning device 101 will be explained using the flowchart in Figure 6. In Example 1, a CNN is used as the machine learning model. However, other models may also be used.
[0033] In step S101, the acquisition unit 101b of the learning device 101 acquires a source image. The source image may be one or more images. The source image may be a real photograph or a computer graphics (CG) image. In subsequent steps, a defocus blur is applied to the source image to create a training image and a ground truth image. For this reason, it is desirable that the source image has edges, textures, gradients, or flat areas with varying intensities and directions so that the defocus blur shape transformation can be correctly applied to various subjects.
[0034] Furthermore, it is preferable that the original image has a signal value higher than the luminance saturation value of the image sensor 102b. This is because, in reality, there are subjects that do not fall within the luminance saturation value when imaged by the imaging device 102 under specific exposure conditions. By applying defocus blur to the ground truth image and the training image, and then clipping them according to the luminance saturation value of the image sensor 102b, it is possible to reproduce subjects that do not actually fall within the luminance saturation value.
[0035] Next, in step S102, the generation unit 101c generates shape specification information and stores it in the storage unit 101a. The shape specification information specifies the size or type of defocus blur after conversion. The size of the defocus blur indicates the range in which the PSF has intensity, as described above. The type of defocus blur means blur in which the intensity distribution of the PSF is different from each other, and includes Gaussian blur, spherical blur, double-line blur, etc., as described later.
[0036] In training a machine learning model, inputting shape specification information along with the training image allows the model to learn to distinguish between multiple ground truth images with different defocus blur shapes from a single training image. In other words, even if the ground truth image contains defocus blur of various shapes, the model learns weights that perform different transformations for each defocus blur shape, rather than weights that convert to an average shape of all the defocus blurs. As a result, it can learn training data containing defocus blur of various shapes with high accuracy all at once. Consequently, it can convert to the user's preferred defocus blur while suppressing the training load and the amount of data to retain.
[0037] Figures 13(A) to (C) show the PSF for double-line bokeh, spherical bokeh, and Gaussian bokeh, respectively. In Figures 13(A) to (C), the horizontal axis represents spatial coordinates (position), and the vertical axis represents intensity.
[0038] The double-line blur shown in Figure 13(A) has a PSF with separated peak intensities. When the PSF at the defocus distance has the shape shown in Figure 13(A), the subject, which is originally a single line, appears to be double-blurred when it is defocused.
[0039] The bokeh shown in Figure 13(B) has a PSF with a flat intensity distribution. The Gaussian bokeh shown in Figure 13(C) has a PSF with a Gaussian distribution.
[0040] Specifying the size of the defocus blur is equivalent to virtually changing the F-number (aperture value) of the optical system 102a. Changing the F-number changes the size of the pupil of the optical system 102a, and therefore changes the size of the defocus blur. Through image processing that converts the defocus blur, it is also possible to change the F-number of the optical system 102a from the captured image to a value that is not physically achievable.
[0041] Specifying the type of defocus blur is equivalent to virtually changing the optical system 102a to a different lens configuration. The types of defocus blur, such as double-line blur, ball blur, and Gaussian blur, depend on the pupil function determined by the lens configuration of the optical system 102a.
[0042] In other words, specifying the size or type of defocus blur after conversion is equivalent to specifying virtual lens parameters. More specifically, specifying the f-number is equivalent to changing the spread of the pupil function. Also, specifying the type of blur, such as double-line blur or spherical blur, is equivalent to changing the amplitude or phase of the pupil function.
[0043] The shape specification information can be either a scalar or a map, but in this embodiment, a map is used. With a scalar, the shape of the defocus blur cannot be specified for each image region, but with a map, it can be specified for each image region. Therefore, a map is preferable. The map is a map with one or more channels, and it numerically represents the type and size of the defocus blur. The numerical values are normalized. For example, when specifying the type of blur with a one-channel map, 0 can be used for Gaussian blur and 1 for spherical blur. When specifying the type of blur with a multi-channel map, the area to be Gaussian blur in the first channel can be set to 1, and the area to be spherical blur in the second channel can be set to 1.
[0044] Further, when the blur magnitude is specified by a single-channel map, 0 may be set to a magnitude equivalent to F1.0, and 1 may be set to a magnitude equivalent to F2.0. On the other hand, when the blur magnitude is specified by a multi-channel map, the region to be set to a magnitude equivalent to F2.0 in the first channel may be set to 1, and the region to be set to a magnitude equivalent to F1.0 in the second channel may be set to 1.
[0045] It is desirable to apply a smoothing filter or the like to the shape specification information. This makes it possible to convert the boundary portions specifying different shapes with high precision.
[0046] Further, in the present embodiment, a plurality of pieces of shape specification information that specify various defocus blur shapes are generated. The various defocus blur shapes are shapes that can be selected via a defocus blur shape specification tab 104a as shown in FIG. 10.
[0047] Next, in step S103, the generation unit 101c generates a training image and stores it in the storage unit 101a. The training image is an image obtained by performing an imaging simulation by applying defocus blur to be converted to an original image. In order to support all types of captured images, it is preferable to apply defocus blur corresponding to various defocus amounts to the original image. The application of defocus blur can be executed by convolving the original image with a PSF, or by taking the product of the frequency characteristics of the original image and an OTF (Optical Transfer Function). Further, since it is desirable that the image on the focus plane does not change before and after the defocus blur conversion, a training image to which no defocus blur is applied and a correct image are also generated.
[0048] Next, in step S104, the generation unit 101c generates a plurality of correct images respectively corresponding to a plurality of shape designation information items for one training image, and stores the generated correct images in the storage unit 101a. The correct image is an image obtained by performing imaging simulation by applying the converted defocus blur to the original image. For the shape of the converted defocus blur, various patterns with different sizes and types corresponding to the shape designation information are prepared. Examples thereof include bokeh equivalent to F2.0, Gaussian blur equivalent to F1.0, and the like. The correct image and the training image may be an undeveloped RAW image or a developed image. Further, the generation order of the training image, the correct image, and the shape designation information may be interchanged.
[0049] Further, the numerical value of the shape designation information that designates the blur size, and the blur size applied as the correct image at that time are stored in the storage unit 101a. These pieces of information are used as information related to blur when determining a first overlap width between two adjacent first image regions, which will be described later.
[0050] Note that the blur size may depend on the direction. For example, when converting to asymmetric blur, the blur size differs depending on the direction. In this case, it is preferable to calculate the blur size in a plurality of directions and acquire the maximum value. Further, when a defocus map can be used at the time of estimation, it is preferable to hold the blur size for each defocus amount and determine the first overlap width according to the defocus amount. This makes it possible to set an appropriate first overlap width for each defocus amount. Furthermore, when the blur size applied as the correct image is changed for each image height, it is preferable to hold the maximum blur size.
[0051] Note that a real captured image obtained by the optical system 102a and the image sensor 102b may be used for generating the learning data. For example, a training image is obtained using the optical system 102a. The correct image is obtained by imaging the same subject as the training image using another optical system having a defocus blur shape different from that of the optical system 102a. In this case, a plurality of pieces of the shape designation information are generated for each of the other optical systems each having a defocus blur shape different from that of the optical system 102a, and each has a different numerical value.
[0052] Alternatively, a subregion with a predetermined number of pixels may be extracted from the training images and ground truth images generated by the two methods described above and used for training. Similarly, for shape specification information, a subregion with a predetermined number of pixels may be extracted at the same time as the training images and ground truth images, or it may be generated after the subregion with a predetermined number of pixels has been extracted from the training images and ground truth images.
[0053] Next, the weight learning process (learning phase) performed by the learning device 101 will be explained using the flowchart in Figure 7.
[0054] In step S111, the acquisition unit 101b of the learning device 101 acquires one or more sets of ground truth images and training input data from the storage unit 101a. The training input data is the input data for the CNN's learning phase. The training input data includes training images and shape specification information. From among the multiple ground truth images, a ground truth image having defocus blur corresponding to the shape specification information is selected and acquired.
[0055] Next, in step S112, the generation unit 101c inputs the training input data to the CNN and generates an output image.
[0056] Using Figure 1, the generation of the output image in Example 1 will be explained. The training input data includes a training image 201 and shape specification information 202. The training image 201 may be grayscale or may have multiple color (channel) components.
[0057] In Example 1, CNN206 has one or more convolutional or fully connected layers. During the first training, the weights of CNN206 (the values of each element of the filter and the bias) are generated by random numbers.
[0058] The concatenation layer 205 concatenates the training image 201 and the shape specification information 202 in a predetermined order in the channel direction. Other data may be concatenated between the training image 201 and the shape specification information 202. The CNN 206 takes the training input data 203, which is the concatenation of the training image 201 and the shape specification information 202, as input and generates an output image 204. If multiple sets of training input data are acquired in step S111, an output image 204 is generated for each set. Even for the same training image 201, the output image 204 will differ depending on the shape specification information 202.
[0059] There are no restrictions on the input location of the training image 201 or shape specification information 202 to the network. Alternatively, the shape specification information 202 may be converted into a feature map in a subnetwork before input. For example, the shape specification information 202 may be converted into a feature map in a subnetwork and input from the intermediate layer of CNN 206.
[0060] If the shape specification information is a scalar, the scalar is converted into a feature map using a subnetwork, and then processed in the same way as the shape specification information 202 in Figure 1.
[0061] Next, in step S113, the update unit 101d updates the CNN weights based on the error between the output image and the ground truth image. In Example 1, the Mean Squared Error (MSE) of the output image and the ground truth image is used as the error. However, other metrics may be used as long as they represent the error between the ground truth image and the output image. If multiple sets of training input data and ground truth images are obtained in step S111, the error value is calculated for each set. The weights are updated based on the calculated error values using methods such as backpropagation.
[0062] Next, in step S114, the update unit 101d determines whether weight learning is complete. Learning completion can be determined by the number of iterations of learning (weight update) reaching a predetermined number, or by the amount of change in weight during the update becoming smaller than a predetermined value. If learning is determined to be incomplete, the process returns to step S111 and acquires one or more sets of new training input data and correct images. If learning is determined to be complete, the learning process ends and the weight information is stored in the storage unit 101a.
[0063] Next, the defocus blur conversion process (estimation phase) of the captured image performed by the image estimation device 103 will be described. In Embodiment 1, the first number of overlapping pixels is determined as the first overlap width between two adjacent first image regions based on the value of the shape specification information.
[0064] The flowchart in Figure 8 shows the blur conversion process as an estimated image generation process (image processing method).
[0065] In step S201, the acquisition unit 103b of the image estimation device 103 acquires the captured image and weight information. The captured image to be acquired may be only a part of the entire captured image. The weight information is read in advance from the storage unit 101a and stored in the storage unit 103a.
[0066] Next, in step S202, the acquisition unit 103b acquires a defocus map of the captured image. The defocus map is a map that shows information about the defocus blur that acted on the subject in the captured image, and it shows the amount of defocus of the subject numerically. The defocus map can be acquired by capturing a parallax image or using DFD (Depth from Defocus), etc.
[0067] The defocus map serves two main purposes. First, it utilizes the fact that each subject has a different degree of defocus to extract areas of subjects mixed in the captured image and use them to determine the replacement area, which will be described later. Second, it is used as input to a machine learning model, enabling conversion that accurately distinguishes between in-focus subjects and defocused blur. In this embodiment, it is used to extract areas of subjects mixed in the captured image, as described in the former example. Details of the latter will be described later.
[0068] Next, in step S203, the generation unit 103d generates shape specification information corresponding to the shape that can be selected in the defocus blur shape specification tab 104a shown in Figure 10. In this embodiment, since the user specifies the replacement area, the shape specification information does not have a spatial distribution and can be the same value throughout the map.
[0069] Next, in step S204, the acquisition unit 103b acquires information regarding the blur after conversion. The information regarding the blur after conversion is read in advance from the storage unit 101a and stored in the storage unit 103a. In this embodiment, the information regarding the blur after conversion includes the numerical value of the shape specification information that specifies the size of the blur, which was saved in step S104, and the size of the blur that was used as the correct image at that time.
[0070] Next, in step S205, the blur conversion unit 103c, acting as a division means, determines a first number of overlapping pixels based on information regarding the blur after conversion. Specifically, the first number of overlapping pixels is larger the larger the amount of blur after conversion, and is set to be greater than or equal to the amount of blur after conversion.
[0071] Next, in step S206, the blur conversion unit 103c divides the captured image into a plurality of first image regions based on a first number of overlapping pixels. Two adjacent first image regions have a first range that overlaps with each other by a first number of overlapping pixels. The division process is comprised of steps S204 to S206.
[0072] The process of dividing multiple first image regions will be explained in detail using Figures 11(A) and (B). In order to prevent discontinuity in the blur after conversion at the boundary between two adjacent first image regions, it is necessary to set the first number of overlapping pixels according to the magnitude of the blur after conversion.
[0073] Figure 11(A) shows the case where the amount of blur after conversion is small, and Figure 11(B) shows the case where the amount of blur after conversion is large. 104c is the captured image, 1213 and 1215 are the first overlapping pixel counts, and 1214 and 1216 are the first ranges. The first overlapping pixel count 1215 in Figure 11(B) is greater than the first overlapping pixel count 1213 in Figure 11(A). In this way, the first overlapping pixel count is determined based on the amount of blur after conversion. Specifically, the larger the amount of blur after conversion, the larger the first overlapping pixel count should be set.
[0074] Next, in step S207 (processing step), the blur conversion unit 103c, acting as a processing means, inputs the input data to the CNN to generate a second image region. The input data includes the first image region and shape specification information. That is, the blur conversion unit 103c processes the multiple first image regions divided in step S206 to generate multiple second image regions.
[0075] The second image region is the image obtained by transforming the defocus blur of the captured image, and its shape is determined by shape specification information. As during training, the second image region is generated using the CNN shown in Figure 1. The acquired trained weights are used in the CNN.
[0076] In this embodiment, multiple estimated images with different blurs are generated for multiple shape specification information corresponding to defocus blur that can be selected in the defocus blur shape specification tab 104a shown in Figure 10. The CNN uses the same weight for the first shape specification information that specifies the size or type of defocus blur and the second shape specification information that specifies the size or type of defocus blur different from the first shape specification information. The multiple generated second image regions are stored in the storage unit 103a.
[0077] Next, in step S208 (combination step), the blur conversion unit 103c combines a plurality of second image regions to generate an estimated image. The estimated image is stored in the storage unit 103a. Two adjacent second image regions have a second range that overlaps by a second number of overlapping pixels.
[0078] In this embodiment, the first number of overlapping pixels and the second number of overlapping pixels are the same, but they do not necessarily have to be the same. Also, the sizes of the first range and the second range may or may not be the same.
[0079] Using Figure 12, the synthesis of two adjacent second image regions will be explained. 1311 and 1314 represent the second image regions, and 1310 indicates the second overlapping pixel count as a second overlapping width set for the second image regions 1311 and 1314. 1312 and 1315 indicate the second range set for the two second image regions 1311 and 1314. 1316 is the composite image generated by the synthesis of the second image regions 1311 and 1314.
[0080] The adjacent second image regions 1311 and 1314 on the left and right sides are combined at the blending position, which is the center of the second overlap width shown by the dotted line in the figure. The blended image 1316 includes the portion of the second image region 1311 to the left of the blending position and the portion of the second image region 1314 to the right of the blending position.
[0081] Alternatively, the two second image regions may be combined at a composite position offset from the center of the second overlap width. Alternatively, the second ranges of the two second image regions may be combined by weighting them according to their distance (number of pixels) from the center of the second overlap width. Furthermore, the second range with the higher brightness value may be weighted greater than the second range with the lower brightness value when combining the two second image regions. In this case, the number of overlapping pixels in the first and second regions may be more than half the size of the blur. Figure 12 illustrates the combination of two adjacent second image regions in the horizontal direction; however, two adjacent second image regions in the vertical direction can be combined in a similar manner.
[0082] Next, we will explain how users can edit captured images through the image estimation device 103, using the flowchart in Figure 9 and Figure 10.
[0083] In step S211 of Figure 9, the acquisition unit 103b of the image estimation device 103 acquires information specifying the shape of the defocused blur after conversion, which is specified by the user in the shape specification tab 104a shown in Figure 10 (this is different from the shape specification information explained in step S203, but it can also be the shape specification information if the shape of the blur can be specified).
[0084] Next, in step S212, the acquisition unit 103b acquires information on the replacement region, which indicates the area in the captured image 104c where the defocus blur is to be converted (replaced with the converted defocus blur image), selected by the user using the brush 104b shown in Figure 10. This replacement region may be determined by the user pixel by pixel, or the acquisition unit 103b may determine it from the area roughly selected by the user with the brush 104b based on the distribution of signal values in the captured image or the defocus map. By determining the replacement region based on the distribution of signal values in the captured image or the defocus map, the subject area to be converted for defocus blur can be extracted. Determining the replacement region based on the distribution of signal values in the captured image means identifying the locations where the pixel values in the captured image change discontinuously and extracting the subject area to be used as the replacement region.
[0085] Next, in step S213, the generation unit 103d replaces the replacement region selected by the user with an estimated image having the specified defocus blur shape. This reduces the number of times an estimated image is generated compared to generating an estimated image each time a replacement region is selected, enabling conversion to the defocus blur desired by the user in a short time. The estimated image used is one stored in the storage unit 103a.
[0086] With the above configuration, it is possible to generate an image in which the shape of the blur contained in the captured image has been transformed while suppressing the processing load.
[0087] Next, preferred conditions for enhancing the effects of this embodiment will be described.
[0088] The input data for CNN should also preferably include a luminance saturation map. The luminance saturation map is generated from training images during the learning phase and from captured images during the estimation phase, and has the same size as each image, indicating the luminance saturation pixel region of each image. In the luminance saturation region, there are false edges that are different from the structure of the subject due to luminance saturation, and machine learning models have difficulty distinguishing false edges from edges that have high-frequency components, such as defocus blur or focal position. The luminance saturation map allows the machine learning model to distinguish between the luminance saturation region and defocus blur or focal position with high-frequency components, enabling highly accurate conversion of defocus blur. Defocus blur with high-frequency components is likely to occur when PSF with sharp peaks, such as double-line blur, is at work.
[0089] The input data to the CNN should also preferably include a defocus map. A defocus map is a map that shows information about the defocus blur that has acted on the subject in the captured image. In the learning phase, the amount of defocus corresponding to the defocus blur that has acted on the training image is converted into a numerical value. For example, the focal plane can be set to 0, the direction away from the imaging device can be negative, and the direction towards the device can be positive. In the estimation phase, a defocus map of the captured image is obtained using disparity image acquisition or DFD (Depth from Defocus), etc. The numerical value of the defocus map at this time corresponds to the numerical value in the learning phase. The defocus map enables conversion that distinguishes between the in-focus subject and the defocus blur with high accuracy. Without a defocus map, it is not possible to distinguish between the in-focus subject and the defocus blur with high-frequency components, and the in-focus subject may become blurred.
[0090] The input data to CNN should preferably also include a lens state map. The lens state map is information that describes the state of the optical system 102a at the time of imaging, represented as (Z, F, D), where Z is the zoom position, F is the aperture value, and D is the focus distance.
[0091] The input data to CNN should preferably also include a location map. The location map is a map showing the image plane coordinates for each pixel in the image. The location map may also be in polar coordinates (corresponding to image height and azimuth).
[0092] Defocus blur varies depending on the lens state, image height, and azimuth. Since CNNs are trained to convert all defocus blurs in the training data on average, the accuracy of the conversion for each defocus blur with a different shape decreases. Therefore, by inputting a lens state map and a position map into the machine learning model, the machine learning model can identify the PSF acting on the captured image.
[0093] As a result, during the learning phase, the machine learning model learns weights that perform different transformations for each shape of defocus blur, rather than weights that perform an average transformation for all defocus blurs in the training images, even if the training images contain defocus blurs of various shapes. Consequently, in the estimation phase, highly accurate transformations are possible for each defocus blur. Therefore, it is possible to train a dataset capable of transforming defocus blurs of various shapes all at once, while suppressing a decrease in transformation accuracy.
[0094] Next, we will describe Example 2. In Example 2, the task performed by the machine learning model is to apply a filter effect. Applying a filter effect means reproducing the effect obtained by attaching a filter to a lens using image processing with a machine learning model. Examples of filters include black mist filters and white mist filters, which are called soft filters. These filters diffuse light, making it possible to obtain images with a soft depiction. By using a machine learning model, it is possible to apply the filter effect with high accuracy even around the luminance saturation region.
[0095] In this embodiment, a machine learning model is trained for each type of filter, and mode information is used as information regarding the blur to be applied.
[0096] Figure 14 shows the configuration of the image processing system 200 of Embodiment 2. Figure 15 shows the external appearance of the image processing system 200. The image processing system 200 includes a learning device 201, an imaging device 202, a lens device 203, a control device (first device) 204, an image estimation device (second device) 205, and networks 206 and 207.
[0097] The learning device 201 and the image estimation device 205 are configured, for example, by a server. The control device 204 is a device operated by the user, such as a personal computer or mobile terminal.
[0098] The learning device 201 includes a storage unit 201a, an acquisition unit 201b, a generation unit 201c, and an update unit 201d, and learns the weights of a machine learning model that applies a filter effect to captured images captured using the imaging device 202. Details regarding the learning process will be described later.
[0099] The imaging device 202 has an image sensor 202a, which converts the optical image formed by the lens device 203 into an image by photoelectric conversion to generate an image. The lens device 203 and the imaging device 202 are detachable and can be combined with multiple types of each other.
[0100] The control device 204 includes a communication unit 204a, a display unit 204b, a storage unit 204c, and an acquisition unit 204d, and controls the processing performed on captured images acquired from the imaging device 202, which is connected by wire or wireless, according to user operations. The control device 204 may also pre-store the captured images acquired from the imaging device 202 in the storage unit 204c and read them out.
[0101] The image estimation device 205 includes a communication unit 205a, an acquisition unit 205b, a storage unit 205c, and an application unit 205d. The image estimation device 205 performs a filter effect application process on an captured image at the request of a control device 204 connected via a network 206. The image estimation device 205 acquires information on learned weights from a learning device 201 connected via a network 207, either during or before the filter effect application process, and uses this information when applying the filter effect to the captured image. The estimated image after the filter effect has been applied is transmitted back to the control device 204, stored in the storage unit 204c, and displayed on the display unit 204b.
[0102] The flowchart in Figure 16 shows the training data generation process performed by the learning device 201. This will be explained using the flowchart in Figure 16.
[0103] In step S301, the acquisition unit 201b acquires the original image. Then, in the subsequent steps, a blur equivalent to a filter is applied to the original image to create a correct image.
[0104] Next, in step S302, the acquisition unit 201b of the learning device 201 acquires a training image. In Embodiment 2, the original image is used as the training image. However, the original image does not necessarily have to be used as the training image; blurring caused by aberrations and diffraction of the lens device 203 and filters such as the optical low-pass filter of the imaging device 202 may be added. Blurring due to defocusing may also be added. In this case, by adding the same blurring to the ground truth image, it becomes possible to add a filter effect while maintaining the sense of resolution.
[0105] Next, in step S303, the acquisition unit 201b acquires a blur equivalent to a filter. Figure 18 shows an example of a blur equivalent to a filter. The horizontal axis represents spatial coordinates (position), and the vertical axis represents intensity. The blur equivalent to a filter has a distribution where the central part has large values and the peripheral part has small values that spread widely, and this distribution has a soft filter effect. By adjusting the values in the peripheral part, the desired filter effect can be obtained. For example, the filter effect can be strengthened by increasing the value, and the amount of diffusion of the filter effect can be increased by increasing the spread of the distribution.
[0106] Next, in step S304, the generation unit 201c generates a correct image and stores it in the storage unit 201a. The correct image is an image obtained by performing an imaging simulation by applying a blur equivalent to a filter to the original image.
[0107] Next, we will describe the weight learning process (learning phase) performed by the learning device 201. In Example 2, the training input data consists only of training images. In addition, a machine learning model is trained for each type of filter that can be selected in the mode selection tab 204b-2, which will be described later. The rest is the same as in Example 1.
[0108] The flowchart in Figure 17 shows the filter effect application process (estimation phase) as an estimated image generation process (image processing method) executed by the control device 204 and the image estimation device 205. Figure 19 shows the user interface for the filter effect application process. In this embodiment, the user can apply a filter effect to the captured image according to the selected mode by selecting one of several pre-set modes.
[0109] As shown in Figure 19, the display unit 204b of the control device 204 displays the captured image 204b-1 and the mode selection tab 204b-2. The captured image 202b-1 is an image that is in focus throughout, with a low-luminance subject 2111 in the center and high-luminance subjects 2112 at both ends.
[0110] In step S401 of Figure 17, the acquisition unit 204d of the control device 204 acquires the mode selected by the user on the mode selection tab 204b-2. Four modes are pre-set as selectable modes, for example, as shown by the circled numbers in the figure: 1. Black mist filter (low diffusion), 2. White mist filter (low diffusion), 3. Black mist filter (high diffusion), and 4. White mist filter (high diffusion).
[0111] A large diffusion amount means that the distribution spreads widely in Figure 18. In other words, it means that the amount of blur applied is large. The types and number of selectable modes can be freely set. In Example 2, mode information indicating the mode selected by the user is used as information regarding the blur applied.
[0112] Next, in step S402, the communication unit 204a of the control device 204 transmits the captured image and mode information to the image estimation device 205, and also transmits a request to perform the process of generating an estimated image.
[0113] Next, in step S403, the communication unit 205a of the image estimation device 205 receives the captured image and mode information transmitted from the control device 204, and also receives (acquires) a request to execute a process to generate an estimated image.
[0114] Next, in step S404, the acquisition unit 205b of the image estimation device 205 acquires information regarding the brightness values of the captured image. The information regarding the brightness values of the captured image includes a brightness saturation map, the brightness values of the captured image, and information indicating the shooting scene. The information regarding brightness values may be acquired from the entire captured image or from a part of the captured image. The information indicating the shooting scene is, for example, a night view. When a night view is captured, there are many high-brightness subjects, so it is preferable to increase the first overlap width between image regions. The reason for increasing the first overlap width between image regions containing high-brightness subjects will be explained later. In this embodiment, a brightness saturation map is acquired as information regarding the brightness values of the captured image.
[0115] Figure 20(A) shows an example of a luminance saturation map for the captured image 204b-1 in Figure 19. The luminance saturation map has signal values only in the region of the high-luminance subject 2112.
[0116] Next, in step S405, the assigning unit 205d, which acts as a division means of the image estimation device 205, determines the first number of overlapping images as the first overlap width based on the acquired mode information and information regarding brightness values. The correspondence between the mode information and the first number of overlapping pixels is stored in the storage unit 205c in advance.
[0117] Specifically, the application unit 205d first determines the first number of overlapping pixels based on the mode information acquired in step S403. It is preferable to set a large first number of overlapping pixels, as a larger diffusion amount of the filter results in a larger amount of blur. Alternatively, the first number of overlapping pixels may be determined according to the selected machine learning model, rather than based on mode information.
[0118] Next, the assigning unit 205d adjusts the first number of overlapping pixels by referring to the information regarding the brightness value acquired in step S404. In this case, if the captured image includes a high-brightness subject, the first number of overlapping pixels is increased compared to when the image does not include a high-brightness subject.
[0119] The reason for increasing the first number of overlapping pixels when a high-brightness subject is that when the blur shown in Figure 18 is applied to a high-brightness subject, the small values that spread widely in the periphery are stretched and become more noticeable. As a result, the blur appears to spread more than when the same blur is applied to a low-brightness subject, so it is preferable to increase the first number of overlapping pixels. As shown in Figure 20(C), when a high-brightness subject is present, the first number of overlapping pixels is increased compared to when a high-brightness subject is not present, as shown in Figure 20(B).
[0120] Next, in step S406, the assigning unit 205d divides the captured image into a plurality of first image regions based on the first number of overlapping pixels. Steps S403 to S406 constitute the division process.
[0121] The procedure for setting the first number of overlapping pixels in step S405 is not limited to the one described above. For example, the first number of overlapping pixels may be determined by referring to information regarding the brightness value of the captured image, and then adjusted based on the mode information. Alternatively, the captured image may be divided into multiple first image regions before adjusting the first number of overlapping pixels. Furthermore, if a high-brightness subject is located in the peripheral part of the first image region, the first number of overlapping pixels may be increased compared to when it is located in the center. This is because high-brightness subjects located in the peripheral part are more prone to discontinuities in the applied blur (or the blur after conversion in Example 1) compared to high-brightness subjects located in the center.
[0122] Next, in step S407, the acquisition unit 205b acquires the learned weight information from the storage unit 205c. The weight information is previously read from the storage unit 201a and stored in the storage unit 205c.
[0123] Next, in step S408 (processing step), the input unit 205d, acting as a processing means, inputs the first image region into a machine learning model to generate a second image region. That is, by processing the multiple first image regions divided in step S406, multiple second image regions are generated.
[0124] Next, in step S409, the assigning unit 205d, acting as a synthesis means, synthesizes a plurality of second image regions to generate an estimated image. The method for synthesizing the second image regions is the same as in Example 1.
[0125] Next, in step S409, the communication unit 205a transmits the estimated image to the control device 204.
[0126] Next, in step S410, the communication unit 204a acquires the transmitted estimated image and saves it to the storage unit 204c.
[0127] In step S402, a request may be sent to generate all the captured images and estimated images for each of the pre-set selectable modes, and the estimated images for each mode generated in subsequent steps may be stored in the storage unit 205c. Once a mode is selected by the user, the estimated image corresponding to that mode is sent to the control device 204.
[0128] With the above configuration, it is possible to generate images with blur added to the captured image while suppressing the processing load.
[0129] Next, we will describe Example 3. In Example 3, instead of using a machine learning model, we generate an image with blur added to the captured image by performing a convolution process (convolution integral, sum of products) using blur.
[0130] Figure 21 shows the configuration of the image processing system 300 of Example 3. Figure 22 shows the external appearance of the image processing system 300. The image processing system 300 includes a lens device 301, an imaging device 302, an image processing device 303, and a display device 304.
[0131] The image processing device 303 includes a generation unit 303a, a storage unit 303b, an acquisition unit 303c, and a blur application unit 303d. The image processing device 303 generates an image with blur applied to the captured image (or at least a part thereof) captured by the imaging device 302.
[0132] The flowchart in Figure 23 shows the blur application process as an estimated image generation process (image processing method) executed by the image processing device 303. Figure 24 shows the user interface related to the blur application process.
[0133] In Embodiment 3, the user can specify the area to be blurred and the shape of the blur to be applied to the captured image. As shown in Figure 24, the display device 304 displays the captured image 304-1, the area division image 304-2, the blur shape specification tab 304-3, and the brush 304-4. In this embodiment, the captured image 304-1 is an image that is in focus throughout. Note that the display format is not limited to the above, for example, the area division image may be superimposed on the captured image or not displayed at all.
[0134] In step S501 of Figure 23, the acquisition unit 303c of the image processing device 303 acquires the captured image and the blur-applied region (subject region) in the captured image 304-1 selected by the user using the brush 304-4. The method for acquiring the blur-applied region is the same as in Embodiment 1. Note that the user does not necessarily have to select the blur-applied region; the acquisition unit 303c may automatically select the blur-applied region.
[0135] Next, in step S502, the acquisition unit 303c acquires blur shape information (information about blur) indicating the blur shape selected by the user using the blur shape specification tab 304-3. In Embodiment 3, the blur shape specification tab 304-3 allows selection of the blur size from five levels, and the type can be selected from Gaussian blur, spherical blur, and double-line blur. There are no restrictions on setting the blur size; the size may be set in 10 levels, or it may be set continuously. In addition, the type of blur may also be limited to Gaussian blur, spherical blur, and double-line blur.
[0136] Next, in step S503, the generation unit 303a, as a division means, determines the first overlapping pixel count as the first overlapping width based on the acquired blur shape information. The blur shape information may be changed according to the image height, or a uniform (constant) blur shape information may be used within a single image. When the blur shape information is changed according to the image height, it is preferable to determine the first overlapping pixel count based on the maximum blur size.
[0137] Next, in step S504, the generation unit 303a divides the captured image into a plurality of first image regions based on the first number of overlapping pixels. Steps S502 to S504 constitute the division process.
[0138] Next, in step S505 (processing step), the blur application unit 303d, acting as a processing means, uses the blur corresponding to the blur shape information to perform convolution processing (convolution integral, sum of products) on a plurality of first image regions to generate a plurality of second image regions. The process of obtaining the second image region g2 by convolution processing the blur d on the first image region g1 is performed by the following equation (1). In equation (1), * indicates convolution, and (x, y) indicates coordinates on the first image region.
[0139]
[0140] Since the first image region is a two-dimensional image, the blur d is also a two-dimensional filter with taps (cells) corresponding to each pixel in the first image region. In general, the more taps (cells) in the blur d, the greater the spread of the blur.
[0141] Next, in step S505, the generation unit 303a generates an estimated image by combining a plurality of second image regions. The method for combining the second image regions is the same as in Example 1.
[0142] Next, in step S506, the generation unit 303a generates a weight map based on the blur application area selected by the user. The weight map determines the proportion of each image when the captured image and the estimated image are weighted averaged, and has a continuous numerical value from 0 to 1. For example, if the value of the weight map determines the proportion of the estimated image, and the value is 0, the weighted average image will remain the same as the captured image. If the value of the weight map is 0.5, the weighted average image will be the result of adding 50% each to the pixel values of the captured image and the estimated image.
[0143] Next, in step S507 (combination step), the generation unit 303a, as a combining means, combines the captured image and the estimated image by weighting and averaging them based on the weight map. Note that it is not necessary to perform weight averaging based on the weight map; convolution processing may be performed in step S504 based on the blur-applying region.
[0144] With the above configuration, it is possible to generate images with blur added to the captured image while suppressing the processing load.
[0145] The image processing devices (103, 205, 303) of each embodiment may be incorporated into the imaging device.
[0146] (Other Embodiments) The present disclosure can also be realized by supplying a program that implements one or more of the functions of the embodiments described above to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more functions.
[0147] The embodiments described above are merely representative examples, and various modifications and changes can be made to each embodiment when implementing this disclosure.
Claims
1. An image processing method comprising: a division step of dividing a first image generated by imaging into a plurality of adjacent first image regions that overlap by a first width in a first direction; and a processing step of performing a process to transform the shape of the blur or to apply blur to at least one of the plurality of first image regions, wherein in the division step, the first width is set based on the blur information in the processing.
2. The image processing method according to claim 1, characterized in that the first direction is the horizontal or vertical direction in the first image.
3. The image processing method according to claim 1 or 2, characterized in that the blur converted in the above processing is blur due to defocusing.
4. The image processing method according to claim 1 or 2, characterized in that the blur applied in the above processing is a blur generated by a filter.
5. The image processing method according to any one of claims 1 to 4, characterized in that the information relating to blurring is information used for training and stored for the machine learning model used in the processing.
6. The image processing method according to any one of claims 1 to 4, characterized in that the information relating to blurring is information obtained through user input.
7. The image processing method according to any one of claims 1 to 4, characterized in that the first width is set as the number of pixels in the division step.
8. The image processing method according to any one of claims 1 to 7, characterized in that, in the division step, the first width is set based on the blur information and the brightness value information of the first image.
9. The image processing method according to any one of claims 1 to 8, characterized in that, in the division step, the first width is set to be larger the larger the amount of blur indicated by the blur information.
10. The image processing method according to claim 9, characterized in that the first width is set to be greater than or equal to the size of the blur.
11. The image processing method according to any one of claims 1 to 10, characterized in that the processing step is performed using a machine learning model.
12. The image processing method according to claim 11, characterized in that the machine learning model generates a second image region from input data including the first image region and information specifying the shape of the blur or blur to be applied after the processing.
13. The image processing method according to claim 12, characterized in that the information specifying the shape of the blur is information specifying at least one of the type and size of the blur.
14. The image processing method according to any one of claims 1 to 13, further comprising a synthesis step of generating a second image by combining a plurality of second image regions obtained by the above processing so that they overlap by a second width in the first direction.
15. The image processing method according to claim 14, characterized in that the second width matches the first width.
16. The image processing method according to claim 14 or 15, characterized in that, in the synthesis step, the portion overlapping by the second width is synthesized by making the weight of the portion with a larger brightness value greater than the weight of the portion with a smaller brightness value.
17. A program characterized by causing a computer to perform processing according to the image processing method described in any one of claims 1 to 16.
18. An image processing apparatus comprising: a division means for dividing a first image generated by imaging into a plurality of adjacent first image regions that overlap by a first width in a first direction; and a processing means for performing a process to transform the shape of the blur or to apply blur to the first image regions, wherein the division means sets the first width based on information regarding the blur after transformation or to be applied by the processing.
19. The image processing apparatus according to claim 18, further comprising a synthesis means for generating a second image by combining a plurality of second image regions obtained by the above-mentioned processing such that they partially overlap in the first direction.
20. A system comprising an image processing apparatus according to claim 18 or 19, and a control device that requests the image processing apparatus to divide the image into a plurality of first image regions and to perform the processing.
