Image processing apparatus, image processing method, program, generation method, imaging apparatus, semiconductor apparatus, and information processing apparatus

Abstract

The present disclosure relates to an image processing apparatus, an image processing method, a program, a generation method, an imaging apparatus, a semiconductor apparatus, and an information processing apparatus with which it is possible to more appropriately generate an image that has a wide dynamic range. Provided is an image processing apparatus comprising a processing unit that, on the basis of a single first image that has a first gradation and a first signal range, virtually generates a signal distribution comprising a second gradation that is higher than the first gradation and a second signal range that is higher than the first signal range. For example, the present disclosure can be applied to an image processing apparatus that generates a virtual HDR image from an SDR image.

Description

Image processing apparatus, image processing method, program, generation method, imaging apparatus, semiconductor device, and information processing apparatus.

[0001] This disclosure relates to an image processing apparatus, an image processing method, a program, a generation method, an imaging device, a semiconductor device, and an information processing apparatus, and more particularly to an image processing apparatus, an image processing method, a program, a generation method, an imaging device, a semiconductor device, and an information processing apparatus that can more appropriately generate images with a wide dynamic range.

[0002] Techniques for generating images with a wider dynamic range (HDR: High Dynamic Range) from images with a normal dynamic range (SDR: Standard Dynamic Range) have been known for some time (see, for example, Patent Documents 1 to 3).

[0003] Patent Document 1 discloses a technique for estimating the luminance saturation region profile from an captured image based on a mathematical formula, changing the exposure conditions based on the estimation result, and performing HDR synthesis by taking another image. Patent Document 2 discloses a technique for superimposing 3DCG and live-action footage, in which the illuminance distribution of the light source is assumed to be a Gaussian distribution and interpolated, and an illumination estimation matrix is ​​calculated using the set position information of the light source. Patent Document 3 discloses a technique for taking images while changing the exposure while fixing the field of view, detecting the saturation region, and performing HDR synthesis from multiple images.

[0004] Japanese Patent Publication No. 2017-118296, Japanese Patent Publication No. 2015-213234, Japanese Patent Publication No. 2020-529159

[0005] Conventional techniques required multiple image captures or multiple images to perform HDR synthesis. Furthermore, relying on specific mathematical models or distributions for HDR synthesis could limit its comprehensiveness and randomness. Therefore, there was a need for a technology that could generate images with a wider dynamic range more appropriately.

[0006] This disclosure is made in light of these circumstances and aims to enable the generation of images with a wider dynamic range more appropriately.

[0007] One aspect of the present disclosure is an image processing apparatus comprising a processing unit that virtually generates a signal distribution consisting of a second grayscale higher than the first grayscale and a second signal range higher than the first signal range, based on a single first image having a first grayscale and a first signal range.

[0008] One aspect of the present disclosure is an image processing method in which an image processing device virtually generates a signal distribution consisting of a second grayscale higher than the first grayscale and a second signal range higher than the first signal range, based on a single first image having a first grayscale and a first signal range.

[0009] One aspect of the present disclosure is a program that causes a computer to function as an image processing apparatus, which includes a processing unit that virtually generates a signal distribution consisting of a second grayscale higher than the first grayscale and a second signal range higher than the first signal range, based on a single first image having a first grayscale and a first signal range.

[0010] In one aspect of the present disclosure, an image processing apparatus, an image processing method, and a program are provided, based on a single first image having a first grayscale and a first signal range, a signal distribution consisting of a second grayscale higher than the first grayscale and a second signal range higher than the first signal range is virtually generated.

[0011] Furthermore, the image processing device, which is one aspect of this disclosure, may be an independent device or an internal block constituting a single device.

[0012] This figure shows an example configuration of one embodiment of an image processing system to which this disclosure is applied. This figure shows an example of a profile when imaging is performed corresponding to an ideal state and a degraded state image. This figure shows an example of a profile when development simulation is performed corresponding to an ideal state and a degraded state image. This figure explains the problems when reproducing a degraded state image from an ideal state image. This figure explains an example of reproducing a degraded state image from an ideal state image. This is a flowchart explaining the flow of virtual HDR image generation processing performed by the image processing system. This figure shows an example of luminance saturation region extraction and labeling. This is a flowchart explaining the detailed flow of virtual profile determination. This figure shows an example of distribution generation. This figure shows the luminance displacement between the edge and the top. This figure shows an example of gain values ​​assigned to each region of a segment. This figure explains the flow of light source color estimation. This figure shows an example of a virtual profile and a virtual HDR image. This figure shows an example of a change reproduction image reproduced using a virtual HDR image. This figure explains the flow of generating a change reproduction SDR image from an SDR image. This figure shows a first example of using an image generated by applying this disclosure in AI learning. This figure shows a second example of using an image generated by applying this disclosure in AI learning. This figure shows a third example of using images generated by applying this disclosure in AI learning. This figure shows a fourth example of using images generated by applying this disclosure in AI learning. This figure shows an example of using images generated by applying this disclosure in AI inference. This figure shows an example of the configuration of a learning device that performs AI learning. This figure shows an example of the configuration of a processing unit that performs AI inference. This figure shows an example of the configuration when providing image processing services via a server on the internet. This is a block diagram showing an example of the configuration of computer hardware.

[0013] <System Configuration> Figure 1 is a diagram showing an example configuration of one embodiment of an image processing system to which the present disclosure is applied.

[0014] As shown in Figure 1, the image processing system 1 consists of an imaging device 11, an external device 12, a storage device 13, an image processing device 14, and an input / output device 15.

[0015] The imaging device 11 is, for example, a device with imaging capabilities such as a camera, smartphone, or tablet terminal. The imaging device 11 captures an image of a subject and generates an SDR (Standard Dynamic Range) image. The imaging device 11 records the generated SDR image in the storage device 13. The external device 12 is a device capable of acquiring SDR images. The external device 12 records the acquired SDR image in the storage device 13. Note that only at least one of the imaging device 11 and the external device 12 needs to be provided; any device capable of acquiring SDR images may be used. Multiple imaging devices 11 and multiple external devices 12 may be provided.

[0016] The storage device 13 is composed of, for example, a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive). The storage device 13 may be configured as local storage or as cloud storage provided by a server on the Internet. The storage device 13 records SDR images (data) transmitted from the imaging device 11 and the external device 12. For example, if the imaging device 11 has a communication function, the storage device 13 receives and records SDR images transmitted from the imaging device 11 via the network. Alternatively, SDR images captured by the imaging device 11 may be recorded in the storage device 13 via a storage medium such as semiconductor memory.

[0017] The image processing device 14 is composed of a computer such as a PC (Personal Computer) or a workstation. The image processing device 14 generates a virtual HDR (High Dynamic Range) image from an SDR image acquired from the storage device 13. The image processing device 14 includes a processing unit 20 which has an image acquisition unit 21, a parameter acquisition unit 22, an image pre-processing unit 23, a virtual signal distribution generation unit 24, and an image post-processing unit 25. The processing unit 20 is realized, for example, by a processor such as a CPU (Central Processing Unit) executing a program recorded in the storage unit. The image processing device 14 also has a communication function and can exchange data with the storage device 13.

[0018] The image acquisition unit 21 acquires SDR images recorded in the storage device 13 and supplies them to the image preprocessing unit 23, the virtual signal distribution generation unit 24, and the image postprocessing unit 25. For example, the image acquisition unit 21 acquires SDR images transmitted from the storage device 13 via a network. The parameter acquisition unit 22 acquires adjustment parameters from the input / output device 15, which consists of input devices, and supplies them to the image preprocessing unit 23. The adjustment parameters are adjustment parameters for generating a virtual signal distribution. The adjustment parameters include information such as the threshold value when determining brightness saturation, functions and coefficients calculated after distance transformation for distribution generation, and the range and distribution of random numbers. The adjustment parameters can be specified by the user.

[0019] The image preprocessing unit 23 performs preprocessing on the SDR image from the image acquisition unit 21 based on the adjustment parameters from the parameter acquisition unit 22, and supplies the results of the preprocessing to the virtual signal distribution generation unit 24. As will be described in detail later, preprocessing is a process performed in advance to generate a virtual signal distribution. The virtual signal distribution generation unit 24 generates a virtual signal distribution based on the SDR image from the image acquisition unit 21 and the results of the preprocessing from the image preprocessing unit 23, and supplies it to the image postprocessing unit 25. The virtual signal distribution is a virtual signal distribution generated from a single SDR image having a first grayscale and a first signal range, and is a signal distribution (virtual profile) consisting of a second grayscale higher than the first grayscale and a second signal range higher than the first signal range.

[0020] The image post-processing unit 25 applies the virtual signal distribution from the virtual signal distribution generation unit 24 to the SDR image from the image acquisition unit 21 to generate a virtual HDR image. The virtual HDR image is an HDR image virtually generated from an existing single SDR image. The virtual HDR image is also an image having a second grayscale and a second signal range. The image post-processing unit 25 records the generated virtual HDR image in the storage device 13. For example, the virtual HDR image is transmitted over a network and recorded in the storage device 13. Note that the data recorded in the storage device 13 is not limited to the virtual HDR image, but may also include, for example, information obtained when generating the virtual signal distribution (for example, various estimation results). Note that in Figure 1, the image processing device 14 may be configured to include a storage device 13 and an input / output device 15.

[0021] <Overview of Virtual HDR Images> SDR images are images in formats such as JPEG and Bitmap. In SDR images, for example, if it is an 8-bit image, information about brightness values ​​outside the dynamic range is lost. Therefore, when the signal saturates, it is displayed as areas of crushed black or blown-out white. An 8-bit image is an image (color image) that has information for 256 shades of R (Red), 256 shades of G (Green), and 256 shades of B (Blue). HDR images are images with a wider dynamic range than SDR images. HDR images can handle floating-point numbers of 16, 32, and 64 bits (16, 32, 64-bit float), for example, and can retain range information that is lost in SDR images. In addition, HDR images can be developed into SDR with arbitrary dynamic range and tone mapping. In the following explanation, SDR images will also be referred to as SDRI (Standard Dynamic Range Image), and HDR images as HDRI (High Dynamic Range Image).

[0022] Figure 2 shows examples of profiles obtained when imaging is performed for images in an ideal state and a degraded state. In Figure 2, the upper panel shows the case when imaging is performed for an image in an ideal state, and the lower panel shows the case when imaging is performed for an image in a degraded state. As shown in the upper left panel of Figure 2, in the ideal state image A1, the light source is in the center, and the brightness at the position on the horizontal dashed line B1 passing through the center is represented by waveform C1. The upper right panel of Figure 2 shows image A2 when an imaging device 11 such as a camera performs imaging for an image in an ideal state A1, and waveform C2 which shows the relationship between the position on the dashed line B2 and the brightness. Waveform C1 is a waveform in which the position corresponding to the light source in the center is the peak. On the other hand, in image A2, the brightness outside the dynamic range D1 of the imaging device 11 is saturated and brightness information is lost, so waveform C2 is a waveform corresponding to the portion of waveform C1 in the dynamic range D1 (the dot pattern portion).

[0023] As shown in the lower left of Figure 2, in the degraded image A3, there is a light source in the center, and there are four degraded areas above, below, to the left and right of the central light source. Degradation here refers to noise and artifacts such as flare, ghosting, blurring, dynamic range, MTF (Modulation Transfer Function), and color mixing. The relationship between the position on the dashed line B3 and the brightness in the degraded image A3 is represented by waveform C3. In the lower right of Figure 2, image A4 is shown when imaging corresponding to the degraded image A3 is performed by the imaging device 11, and waveform C4 shows the relationship between the position on the dashed line B4 and the brightness. Waveform C3 is a waveform in which the position corresponding to the central light source is the peak, but the brightness at the positions corresponding to the degraded areas on the left and right is convex. On the other hand, in image A4, brightness information outside the dynamic range D3 of the imaging device 11 is lost, so waveform C4 is a waveform corresponding to the portion of the dynamic range D3 of waveform C3.

[0024] Figure 3 shows examples of profiles obtained when development simulations are performed for images in an ideal state and a degraded state. In Figure 3, the upper panel shows the case when development simulation is performed for an image in an ideal state, and the lower panel shows the case when development simulation is performed for an image in a degraded state. The upper left panel of Figure 3, similar to the upper left panel of Figure 2, shows the relationship between the position on the dashed line B1 and the brightness in the image A1 in the ideal state, represented by waveform C1. The upper right panel of Figure 3 shows image A5, obtained by simulating imaging corresponding to the image A1 in the ideal state through development simulation, and waveform C5, which represents the relationship between the position on the dashed line B5 and the brightness. In image A5, since brightness information outside the dynamic range is lost due to the development simulation, waveform C5 becomes a waveform corresponding to the dynamic range portion.

[0025] The lower left of Figure 3, similar to the lower left of Figure 2, shows the relationship between the position on the dashed line B3 and the brightness in the degraded image A3, represented by waveform C3. The lower right of Figure 3 shows image A6, which was simulated by development simulation to capture an image corresponding to the degraded image A3, and waveform C6, which represents the relationship between the position on the dashed line B6 and the brightness. In image A6, brightness information outside the dynamic range is lost due to development simulation, so waveform C6 becomes a waveform corresponding to the dynamic range portion. Here, as shown by arrow G1 in Figure 3, when reproducing the degraded image A3 from the ideal image A1, the brightness distribution of the degraded state (waveform C3) can be obtained by performing degradation calculations such as convolution using PSF (Point Spread Function) on the brightness distribution of the ideal state (waveform C1) (arrow G2 in Figure 3). In other words, by performing calculations that reproduce image quality changes (e.g., degradation) on the brightness distribution of the ideal state, it becomes possible to visualize changes in image quality.

[0026] Figure 4 illustrates the problems encountered when reconstructing a degraded image from an ideal image. As shown in the upper part of Figure 4, in image A2 captured by the imaging device 11, luminance information outside the dynamic range D1 is lost during imaging. Therefore, the relationship between position on the dashed line B2 and luminance is represented by waveform C2, and luminance information for the portion of waveform C2' corresponding to waveform C1 is missing. In other words, image A2 is an SDR image with a saturation region. For this reason, it is not possible to reconstruct the degraded image A3 from image A2 (arrow G3 in Figure 4). To put it another way, as shown by arrow G4 in Figure 4, the luminance distribution of the degraded state (waveform C3) cannot be obtained from the luminance distribution of image A2 (waveform C2). It is necessary to use the luminance distribution of the ideal state (waveform C1), which includes luminance information for the portion outside the finite dynamic range D1. These relationships are also similar for the relationship between image A5 (waveform C5) obtained from development simulation and the degraded image A3 (waveform C3).

[0027] Figure 5 illustrates an example of reproducing a degraded image from an ideal image. In Figure 5, the upper right shows image A2 (SDR image) captured by the imaging device 11 and its brightness distribution (waveform C2). In image A2, brightness information outside the dynamic range D1 is lost during imaging. However, as shown in the brightness distribution on the upper left of Figure 5, a waveform C2' corresponding to the brightness information outside the dynamic range D1 is generated for waveform C2, thereby virtually generating an ideal brightness distribution (waveform C1). That is, in the image processing device 14, the processing unit 20 generates a virtual HDR image (ideal image A1) using an existing single SDR image (image A2). By virtually generating a brightness distribution outside the dynamic range D1 in this way, a degraded image (degraded image A3) can be generated (reproduced) based on the virtual brightness distribution (waveforms C2, C2') shown on the upper left of Figure 5 (arrows G5, G6 in Figure 5). Furthermore, by applying a dynamic range D1 to this degraded image, it becomes possible to generate an SDR image (image A6).

[0028] As described above, in the processing unit 20 of the image processing apparatus 14, a virtual HDR image can be generated from an existing single SDR image. Further, using the generated virtual HDR image, the change in the image quality can be reproduced. At this time, if there is a single SDR image, a virtual HDR image can be generated, so that the labor of shooting by the user is reduced, and a virtual HDR image can be created from an existing SDR image without the user shooting. Also, since the SDR image is a low-tone image and the virtual HDR image is a high-tone image, a high-tone image can be generated from the low-tone image. For example, an image having gradations of N + 1 bits or more can be created from an image captured by the imaging apparatus 11 having an image sensor capable of generating an N-bit image. At this time, compared with the case of acquiring an SDR image or an HDR image by imaging, a high-tone image can be generated with low power consumption. Further, by using the generated virtual HDR image, changes in image quality such as flare and dynamic range can be realistically reproduced.

[0029] <Process Flow> Referring to the flowchart of FIG. 6, the flow of the virtual HDR image generation process executed by the image processing system 1 of FIG. 1 will be described. In the description of each step of the flowchart of FIG. 6, reference will be made to FIGS. 7 to 13 as appropriate.

[0030] In step S11, the image preprocessing unit 23 determines a luminance saturation determination threshold based on the adjustment parameter supplied from the parameter acquisition unit 22. In step S12, the image preprocessing unit 23 performs preprocessing on the SDR image supplied from the image acquisition unit 21. The preprocessing includes, for example, inverse gamma correction and noise removal.

[0031] In step S13, the image preprocessing unit 23 extracts a luminance saturation region from the preprocessed SDR image based on the determined luminance saturation determination threshold. A in FIG. 7 shows a specific example of the extraction of the luminance saturation region. In A of FIG. 7, luminance saturation regions B11 to B13 are extracted as saturation regions where the luminance (signal value) is saturated from the SDR image A1l.

[0032] In step S14, the image preprocessing unit 23 performs a binarization process on the SDR image from which the luminance saturation region has been extracted. Here, if the number of channels is two or more, the binarization process can be performed using information from a specific channel or multiple channels. In step S15, the image preprocessing unit 23 counts the number of luminance saturation regions extracted from the SDR image.

[0033] In step S16, the image preprocessing unit 23 performs labeling on the luminance saturation regions. Figure 7B shows an example of labeling. In Figure 7B, labels are assigned to the luminance saturation regions B11 to B13 in the binarized SDR image A11 according to the number of regions. Specifically, in Figure 7B, the number of regions N = 3, so the label "1" is assigned to the luminance saturation region B11, the label "2" is assigned to the luminance saturation region B12, and the label "3" is assigned to the luminance saturation region B13.

[0034] In step S17, the virtual signal distribution generation unit 24 performs virtual profile determination processing. In the virtual profile determination processing, the region containing the luminance saturation region to which a label has been assigned is designated as the region of interest, and the process of determining a virtual profile for each region of interest is repeatedly executed. For example, if the number of regions N = 3, the loop of step S17 is repeated 3 times. Figure 7B shows the case where the region containing the luminance saturation region B11 is designated as the region of interest E11.

[0035] Figure 8 is a flowchart illustrating the detailed flow of the virtual profile determination process corresponding to step S17. In step S21, the virtual signal distribution generation unit 24 generates a signal distribution by performing a distance transformation using Euclidean distance on the region of interest. The method for generating the signal distribution is not limited to the method using Euclidean distance; other methods such as a constant value, Manhattan distance, Chebyshev distance, or random number distribution may also be used. Furthermore, calculations may be performed on the results obtained using these methods by applying arbitrary functions or random number distributions.

[0036] In step S22, the virtual signal distribution generation unit 24 normalizes the generated signal distribution by its maximum value. For example, if distance conversion was performed using Euclidean distance, normalization can be performed using the maximum distance value. Normalization converts the signal into a format that is easy for the user to calculate. In step S23, the virtual signal distribution generation unit 24 performs a nonlinear transformation on the luminance displacement of the normalized signal distribution. Specific examples of the processing in steps S21 to S23 are shown in Figures 9 and 10.

[0037] In Figure 9, a signal distribution is generated for the binarized region of interest E11 by a distance transformation using Euclidean distance (S21), and then normalized by the maximum distance (S22) to obtain the distribution F11. In this distribution F11, when the boundary of the binarized region is the edge and the center is the top, the luminance displacement from the edge to the top is expressed by a linear relationship, as shown in Figure 10A. In Figure 10A, when the vertical axis is luminance I and the horizontal axis is position x, the relationship between I and x is a linear function (I = x / x). top This is expressed as ). Here, the fact that the brightness value is higher the closer you are to the top and lower the closer you are to the edge is represented by a linear function.

[0038] Furthermore, by nonlinearly transforming the luminance displacement of the normalized distribution (S23), the distribution F12 is obtained. In this distribution F12, the luminance displacement from the edge to the top is expressed by a nonlinear relationship, as shown in Figure 10B. In Figure 10B, the relationship between I and x is expressed as a nonlinear function (I = f(x)). The function and coefficients used in f(x) are arbitrary. Here, the nonlinear function represents that the luminance value increases as you get closer to the top, and decreases as you move away from the top and closer to the edge. f(x) can be determined for each segment corresponding to the luminance saturation region. Note that a random number distribution may be used instead of an arbitrary function.

[0039] In step S24, the virtual signal distribution generation unit 24 determines and applies a gain value. Here, a gain value (g) is set for each segment corresponding to the luminance saturation region. iThe gain value is determined. The gain value may be the same for all segments, or it may be determined randomly for each segment. If a random number distribution is used, the random number distribution may be arbitrarily adjusted to suit the learning target. Figure 11 shows an example of gain values ​​assigned to each region of a segment. In Figure 11, the horizontal axis represents the gain value, and an arbitrary gain is determined for each segment. In each region of the segment, the determined arbitrary gain value (g i ) is multiplied.

[0040] In step S25, the virtual signal distribution generation unit 24 performs light source color estimation. Figure 12 is a diagram illustrating the flow of light source color estimation. As shown in Figure 12, in light source color estimation, the peripheral region G11 for the luminance saturation region B11 is set by performing an expansion process that expands the shape of the luminance saturation region B11 contained in the region of interest E11. For example, in the expansion process, the shape of the luminance saturation region B11 contained in the region of interest E11 is expanded in two stages, large and small. In this case, in the small expansion, at least one pixel is expanded so that it is smaller than the large expansion. Also, in the large expansion, at least one pixel is expanded so that it is larger than the small expansion. Then, the peripheral region G11 can be set by performing a negative logical AND (NAND: Not AND) operation between the region of interest E11 containing the large-expanded luminance saturation region B11 and the region of interest E11 containing the small-expanded luminance saturation region B11. In this example, since the shape of the luminance saturation region B11 is circular, the shape of the peripheral region G11 is donut-shaped.

[0041] Also, when the SDR image to be processed is an RGB three-channel image (color image), for each channel image (E21-R, E21-G, E21-B), by performing a logical product (AND) operation with the target region E11 including the peripheral region G11, the peripheral regions (G21-R, G21-G, G21-B) of the SDR image can be extracted. In FIG. 12, the image E21-R is the R-channel image, the image E21-G is the G-channel image, and the image E21-B is the B-channel image. The peripheral region G21-R is the peripheral region extracted from the image E21-R, the peripheral region G21-G is the peripheral region extracted from the image E21-G, and the peripheral region G21-B is the peripheral region extracted from the image E21-B. And by calculating the average value (mean R , mean G , mean B ) within the peripheral region for each channel, the peripheral luminance information for each channel can be extracted.

[0042] In the example of FIG. 12, the average value (mean G ) of the G channel is the largest, followed by the average value (mean R ) of the R channel, and the average value (mean B ) of the B channel is the smallest. Using the average values of these channels, as shown in the following formula (1), the intensity ratio between channels is determined as the gain value (g R , g G , g B ).

[0043] g R = mean R / max(mean) g G = mean G / max(mean) g B = mean B [[ID=3�]] / max(mean) ・・・(1)

[0044] In formula (1), mean = (mean R , mean G , mean B ). In the example of FIG. 12, the average value (mean GSince ) is the largest, max(mean) = max(mean G ) And for each RGB channel, the gain value (g R , g G , g B ) are multiplied by each of these values. Thus, when a single SDR image has two or more channels, in light source color estimation, the signal balance between channels is estimated by estimating the color of the luminance saturation region using information about the peripheral region of the luminance saturation region, and a gain value (g) corresponding to the estimation result is calculated. Ri , g Gi , g Bi By multiplying this by each channel, a more realistic color can be reproduced.

[0045] In step S26, the virtual signal distribution generation unit 24 determines the virtual profile of the region of interest. That is, for the first region of interest E11 (i = 1), the processing in steps S21 to S25 described above is performed to generate a distribution and various gains (g i , g Ri , g Gi , g Bi By multiplying by ), the virtual profile of the first area of ​​interest E11 can be determined. Figure 13 shows an example of a virtual profile. As shown in Figure 13, virtual profiles A31-R corresponding to the R channel, A31-G corresponding to the G channel, and A31-B corresponding to the B channel are generated as virtual profiles of the first area of ​​interest E11.

[0046] Once the processing in steps S21 to S26 for the first area of ​​interest E11 (i = 1) is completed, the same processing in steps S21 to S26 is performed for the second area of ​​interest (the area of ​​interest including the luminance saturation area B12) and the third area of ​​interest (the area of ​​interest including the luminance saturation area B13). That is, if the number of areas N = 3, the loop in step S17 is repeated three times, and a virtual profile is determined for each area of ​​interest corresponding to the luminance saturation areas B11 to B13. Of the steps in Figure 8, steps S21, S24, and S26 are mandatory processes, while steps S22, S23, and S25 are optional processes. Once the loop in step S17 is completed, the processing proceeds from step S17 to step S18.

[0047] In step S18, the image post-processing unit 25 determines the profile for the entire region. Here, the loop from step S17 is repeated according to the number of regions, determining a virtual profile for each region of interest, and thus determining the profile for the entire region. In step S19, the image post-processing unit 25 adds the determined profile for the entire region to the unsaturated regions (regions where the brightness is not saturated) in the SDR image to be processed. For example, as shown in Figure 13, by adding the virtual profiles A31-R, A31-G, and A31-B corresponding to the region of interest E11 to the unsaturated region of the SDR image A11, a virtually generated virtual region B31 is superimposed on the portion corresponding to the brightness saturated region B11. Similarly, for the portions corresponding to the brightness saturated regions B12 and B13, by adding the virtual profiles generated for those regions of interest, virtually generated virtual regions B32 and B33 are superimposed. In this way, by adding the profile for the entire region to the unsaturated region, a virtual HDR image can be generated from a single SDR image A11. Once the process in step S19 is completed, the series of processes ends.

[0048] Of the steps in Figure 6, steps S11, S13-19 are mandatory processes, but step S12 is an optional process.

[0049] As described above, the virtual HDR image generation process can generate a wide dynamic range image (virtual HDR image) from a single image (SDR image) acquired from the imaging device 11 or external device 12, based on adjustment parameters specified by the user. Alternatively, a high-gradation image (virtual HDR image) can be generated from a low-gradation image (SDR image). In this case, since it is possible to generate a virtual profile based on any and random shape and mathematical formula, there are no limitations on comprehensiveness or randomness. Furthermore, since a virtual HDR image can be generated with just one SDR image, imaging for HDR synthesis is unnecessary, and information such as the lighting conditions at the time of imaging is also unnecessary. Therefore, it is possible to efficiently collect the image (SDR image) that will be the basis for the virtual HDR image. Thus, this disclosure makes it possible to generate a wide dynamic range image more appropriately from an existing single image.

[0050] Furthermore, in the virtual HDR image generation process, since a single SDR image as the input image has one or more channels, processing can be performed according to any combination of channels. In the virtual HDR image generation process, if a single SDR image has two or more channels, the signal balance between channels in the luminance saturation region can be estimated (color estimation of the luminance saturation region) based on information about the peripheral region of the luminance saturation region.

[0051] Furthermore, this disclosure allows for the reproduction of image quality changes using the generated virtual HDR image. For example, image quality changes can be visualized through image quality simulation. Figure 14 shows an example of a change-reproduced image using a virtual HDR image. Figure 14 shows the visualization of flare using image quality simulation as an example of visualizing image quality changes. In Figure 14, for comparison, the upper panel shows the visualization of flare using a conventional image, and the lower panel shows the visualization of flare using a virtual HDR image generated by applying this disclosure.

[0052] As shown in the upper part of Figure 14, when visualizing flare using the SDR image A41, the luminance outside the dynamic range is saturated and luminance information is lost, so the flare cannot be visualized in the change-reproduced image A42. On the other hand, as shown in the lower part of Figure 14, when visualizing flare using the virtual HDR image A43, luminance information outside the dynamic range is virtually generated, so the flare can be visualized in the change-reproduced image A44.

[0053] Here, the change-reproduced image A44 does not necessarily match reality, but it is an image that reproduces a realistic appearance, so the user can imagine how it will look. In this way, by using a virtual HDR image generated by applying this disclosure, it is possible to realistically reproduce changes in image quality and visualize those changes. In Figure 14, the visualization of flare is shown as an example, but for example, exposure time, gain changes, and changes in the illumination of the subject can also be reproduced by image quality simulation using a virtual HDR image.

[0054] The input image used in the virtual HDR image generation process is an existing single SDR image, but the image format and data array dimensions are not limited. Furthermore, it is not limited to SDR images; multiple images may be partially HDR-composited before being used as the input image. The number of channels is not limited to the three RGB channels; one or more channels are acceptable. For two or more channels, signal balance estimation between channels can be performed. Other color representation methods are also acceptable, not limited to the RGB color model, such as the CMYK color model, Lab color space, HSV color space, YCbCr color space, or multi-wavelength images. On the other hand, the output image generated by the virtual HDR image generation process is a virtual HDR image, but like the input image, the image format and data array dimensions are not limited; one or more channels are sufficient, and various color representation methods can be used. Also, the output image may have a different data format than the input image. For example, if the input image corresponds to three RGB channels, the output image may be an image corresponding to the YCbCr color space.

[0055] <Examples of application to AI> Virtual HDR images and images generated from virtual HDR images (e.g., change-reproduced SDR images) can be used for AI learning and AI inference. Since the randomness and comprehensiveness of the training data used for AI learning are important for improving inference accuracy, change-reproduced SDR images generated from virtual HDR images can be used as training data. In other words, change-reproduced SDR images do not necessarily match reality, but they reproduce realistic aspects, making them suitable for use as training data. Therefore, by using images generated by applying this disclosure in AI learning, it is possible to enhance the effectiveness of AI, such as by performing effective AI learning.

[0056] Figure 15 shows the process for generating a change-reproduced SDR image from an SDR image. As shown in Figure 15, the change-reproduced SDR image A52 is generated by performing the following steps in order on the SDR image A51: virtual HDR image generation process (S51), image change reproduction calculation process (S52), and development process with an arbitrary dynamic range (S53). For example, the SDR image A51 is an image captured by the imaging device 11. The virtual HDR image generation process is the same as the virtual HDR image generation process in Figure 6. In the image change reproduction calculation process, calculations are performed on the virtual HDR image to reproduce image changes. In the development process, the change-reproduced SDR image A52 can be generated by narrowing the dynamic range. The change-reproduced image is an image that reproduces image quality changes (for example, flare and dynamic range). Furthermore, when using SDR image A51 or change-reproduced SDR image A52 as training data, the processing in steps S51 to S53 may be performed in advance (before AI training) or on the images (SDR images) imported during training.

[0057] Figure 16 shows a first example of using an image generated by applying this disclosure in AI learning. In the virtual HDR image generation process, the same process as in the virtual HDR image generation process in Figure 6 is performed, and a virtual HDR image A62 is generated from the SDR image A61 (S61). Next, in the image change reproduction calculation process, calculations are performed on the generated virtual HDR image A62 to reproduce image changes (S62), and further development is performed with a defined dynamic range (S63). Here, by narrowing the dynamic range, a change-reproduced SDR image A63 can be generated from the virtual HDR image A62. In AI learning, a learning process is performed that uses the SDR image A61 and the change-reproduced SDR image A63 as training data to perform machine learning (S64). That is, in AI learning, by inputting a pair of noise-free SDR image A61 and change-reproduced SDR image A63 containing noise as a dataset, a trained model is generated that performs machine learning to remove noise contained in the image. As a machine learning method, for example, neural networks and deep learning can be used.

[0058] Figure 17 shows a second example of using an image generated by applying this disclosure in AI learning. In steps S71 to S73 of Figure 17, a virtual HDR image generation process (S71) is performed, similar to the process in steps S61 to S63 of Figure 16, to generate a virtual HDR image A72 from an SDR image A71. Then, an image change reproduction calculation process (S72) and a development process with a defined dynamic range (S73) are performed to generate a change-reproduced SDR image A73 from the virtual HDR image A72. Furthermore, a condition-altered SDR image A74 is generated from the virtual HDR image A72, which is an image with conditions such as exposure time and dynamic range changed. In addition, by performing an image change reproduction calculation process (S72) and a development process with a defined dynamic range (S73) on the virtual HDR image A72, a condition-altered change-reproduced SDR image A75 corresponding to the changed conditions can be generated.

[0059] In AI learning, machine learning is performed using SDR image A71, SDR image A74 under different conditions, and SDR image A73 and SDR image A75 that reproduce changes under different conditions as training data, and a trained model is generated (S74). In other words, in AI learning, by inputting pairs of noise-free SDR image A71 and SDR image A74 under different conditions, and noise-containing SDR image A73 and SDR image A75 that reproduce changes under different conditions as a dataset, a trained model is generated that has undergone machine learning to remove noise contained in the images. For example, a model to remove flare can be trained by inputting a pair of images with high exposure (SDR image A71, SDR image A73 that reproduces changes) and a pair of images with low exposure (SDR image A74 and SDR image A75 that reproduce changes under different conditions).

[0060] Figure 18 shows a third example of using an image generated by applying this disclosure in AI learning. In steps S81 to S82, a virtual HDR image generation process (S81) is performed, similar to steps S61 to S62 in Figure 16, to generate a virtual HDR image A82 from an SDR image A81, and an image change reproduction calculation process (S82) is performed to generate a change-reproduced HDR image A83 from the virtual HDR image A82. Here, the change-reproduced HDR image A83 can be generated by maintaining the dynamic range. In AI learning, machine learning is performed using the virtual HDR image A82 and the change-reproduced HDR image A83 as training data, and a trained model is generated (S83). That is, in AI learning, a pair of a noise-free virtual HDR image A82 and a change-reproduced HDR image A83 containing noise is input as a dataset, and a trained model is generated that has undergone machine learning to remove noise contained in the image.

[0061] Figure 19 shows a fourth example of using an image generated by applying this disclosure in AI learning. In steps S91 to S93 of Figure 19, similar to steps S61 to S63 of Figure 16, a virtual HDR image generation process (S91) is performed to generate a virtual HDR image A92 from an SDR image A91. Then, by performing an image change reproduction calculation process (S92) and a development process with a defined dynamic range (S93), a change-reproduced SDR image A93 is generated from the virtual HDR image A92. In addition, in the virtual HDR image generation process (S91), saturation region color estimation information A94, which is information about the color of the luminance saturation region, can be obtained. Saturation region color estimation information A94 is, for example, information about the color of the luminance saturation region estimated using the peripheral region of the luminance saturation region in the process of step S25 of Figure 8 described above.

[0062] In AI learning, machine learning is performed using the SDR image A91, saturated color estimation information A94, and change-reproduced SDR image A93 as training data, and a trained model is generated (S94). In other words, in AI learning, by inputting a pair of noise-free SDR image A91 and change-reproduced SDR image A93 containing noise as a dataset, a trained model is generated that has undergone machine learning to remove noise contained in the image. Furthermore, by simultaneously inputting supplementary information such as saturated color estimation information A94 along with the SDR image A91 to a network such as a DNN (Deep Neural Network), the inference accuracy in the inference process using the generated trained model can be improved. Note that the processing in steps S91 to S93 may be performed in advance or in batches during AI learning.

[0063] Figure 20 shows an example of using an image generated by applying this disclosure in AI inference. In Figure 20, in AI inference, an inference process is performed using a trained model generated by AI learning (S102). For example, when using the trained model generated in AI learning (S64) of Figure 16 in AI inference (S102), by performing AI inference using the trained model on an input SDR image A101 containing noise, the inference result A104 outputs an SDR image A101 with the noise removed. Similarly, when using the trained model generated in AI learning (S74) of Figure 17, noise contained in the input SDR image A101 can be removed by performing AI inference using the trained model. For example, if the SDR image A101 contains flare, the flare can be removed.

[0064] Furthermore, in Figure 20, a virtual HDR image A102 is generated by performing a virtual HDR image generation process similar to the virtual HDR image generation process in Figure 6 (S101). At this time, saturation region color estimation information A103, which is information about the color of the luminance saturation region, is acquired during the virtual HDR image generation process. For example, in AI inference (S102), if the trained model generated in AI learning (S94) in Figure 19 is used, the saturation region color estimation information A103 is input along with the SDR image A101 containing noise, and AI inference is performed using the trained model, resulting in the output of an SDR image A101 with the noise removed as the inference result A104. At this time, the inference accuracy can be improved by inputting the saturation region color estimation information A103. For example, it is possible to infer with high accuracy from real-world images.

[0065] Figure 21 shows an example of the configuration of a learning device that performs AI learning. In Figure 21, the learning device 31 consists of an acquisition unit 41, a preprocessing unit 42, a learning unit 43, and an output unit 44.

[0066] The acquisition unit 41 acquires training data such as SDR images stored in the storage device 13 and supplies it to the preprocessing unit 42. The preprocessing unit 42 performs preprocessing on the training data supplied from the acquisition unit 41. For example, the preprocessing unit 42 generates a virtual HDR image from an SDR image by performing a virtual HDR image generation process. The preprocessing unit 42 also generates change-reproduced images such as change-reproduced SDR images and change-reproduced HDR images by performing image change reproduction calculation processing and development processing with a defined dynamic range. The preprocessing unit 42 supplies images such as SDR images, virtual HDR images, and change-reproduced images to the learning unit 43 as training data. The training data may also include information obtained from the virtual HDR image generation process (for example, supplementary information such as saturation region color estimation information). The preprocessing unit 42 performs processing corresponding to the preprocessing shown in Figures 16 to 19 (S61 to S63, etc.). In the preprocessing, a large amount of varied datasets can be generated by rigorous simulations related to optics and images.

[0067] The learning unit 43 performs machine learning using training data supplied from the preprocessing unit 42 and generates a trained model. The machine learning uses techniques such as deep learning. Specifically, the learning unit 43 takes the change-reproduced image and at least one of the SDR image and virtual HDR image as input from among the SDR image, virtual HDR image, and change-reproduced image (change-reproduced SDR image or change-reproduced HDR image), and generates a trained model that has been subjected to machine learning to remove noise contained in the image. Here, the virtual HDR image is generated based on a virtual signal distribution based on the SDR image. The change-reproduced HDR image is generated by applying processing to the virtual HDR image that includes processing to reproduce image changes. In addition, supplementary information such as saturated color estimation information may be used in machine learning. For example, the learning unit 43 performs processing corresponding to the AI ​​learning (S64, etc.) in Figures 16 to 19. The learning unit 43 supplies the generated trained model to the output unit 44.

[0068] The output unit 44 outputs the trained model supplied from the learning unit 43 to the outside. For example, the output unit 44 provides the trained model to a device or chip that performs inference processing using the trained model. Alternatively, the output unit 44 may upload the trained model to a server on the network, and the server may provide the trained model to a device connected via the network. The output unit 44 may also record the trained model in the storage device 13. At least some of the functions of the learning device 31 may be included in the image processing device 14.

[0069] Figure 22 shows an example configuration of a processing unit that performs AI inference. In Figure 22, the processing unit 51 consists of an acquisition unit 61, a preprocessing unit 62, an inference unit 63, and an output unit 64. The inference unit 63 holds a trained model provided by the learning device 31 (output unit 44).

[0070] The acquisition unit 61 acquires input data such as SDR images stored in the storage device 13 and supplies it to the preprocessing unit 62. The preprocessing unit 62 supplies the input data such as SDR images supplied from the acquisition unit 61 to the inference unit 63. The preprocessing unit 62 also performs preprocessing on the input data supplied from the acquisition unit 61. For example, the preprocessing unit 62 generates a virtual HDR image from the SDR image by performing a virtual HDR image generation process. The preprocessing unit 62 performs the processing corresponding to the preprocessing (S101) in Figure 20. The preprocessing unit 62 supplies the SDR image along with supplementary information obtained from the virtual HDR image generation process (for example, saturation region color estimation information) as input data to the inference unit 63.

[0071] The inference unit 63 inputs the input data supplied from the preprocessing unit 62 into a trained model, and obtains an inference result as its output. That is, the inference unit 63 outputs an image (such as a noise-removed SDR image) obtained using an algorithm that employs a trained model that has undergone machine learning, based on the input image (such as a noise-containing SDR image). At this time, the inference accuracy can be improved by using supplementary information such as saturated region color estimation information.

[0072] Here, the trained model is generated by taking the change-reproduced image and at least one of the SDR image and virtual HDR image as input, and performing machine learning to remove noise contained in the image. The virtual HDR image is generated based on a virtual signal distribution based on the SDR image. The change-reproduced HDR image is generated by applying processing to the virtual HDR image, which includes processing to reproduce image changes. For example, the inference unit 63 performs processing corresponding to the AI ​​inference (S101) in Figure 20. The inference unit 63 supplies the inference result to the output unit 64.

[0073] The output unit 64 outputs the inference results supplied from the inference unit 63. For example, the output unit 64 can display the inference results on a display device. The output unit 64 may also record the inference results in the storage device 13. At least some of the functions of the processing device 51 may be included in the image processing device 14.

[0074] <Other Configuration Examples> Figure 23 shows an example configuration when providing an image processing service via a server on the Internet. In Figure 23, each of the electronic devices 71-1 to 71-N (N: an integer of 1 or more) and the server 72 are interconnected via the Internet 73.

[0075] The electronic device 71-1 is, for example, a camera, smartphone, or tablet terminal. The electronic device 71-1 has a semiconductor chip 81-1. The semiconductor chip 81-1 includes semiconductor devices such as an image sensor, MCU (Micro Processing Unit), microcontroller, GPU (Graphics Processing Unit), DSP (Digital Signal Processor), and FPGA (Field Programmable Gate Array). The electronic device 71-1 has an imaging function using an image sensor, etc., and captures an image of a subject and generates an image. The electronic device 71-1 also has a communication function that supports cellular communication (e.g., LTE-Advanced or 5G), wireless communication such as Wi-Fi (Local Area Network), or wired communication, and can connect to the Internet 73.

[0076] Electronic devices 71-2 to 71-N are configured in the same way as electronic device 71-1. Each of electronic devices 71-2 to 71-N also has a semiconductor chip 81. When it is not necessary to distinguish between electronic devices 71-1 to 71-N, they are referred to as electronic device 71. When it is not necessary to distinguish between semiconductor chips 81-1 to 81-N, they are referred to as semiconductor chip 81.

[0077] Server 72 can be comprised of, for example, a server installed in a data center, or a server operating in a cloud computing environment (cloud server). For example, Server 72 is an information processing device having the configuration shown in Figure 24, which will be described later. Server 72 has communication functions that support wireless or wired communication and can connect to the Internet 73. Server 72 can transmit or receive data to and from other devices via the Internet 73.

[0078] Server 72 has a processing unit 91. Processing unit 91 has functions corresponding to the processing unit 51 in Figure 22. For example, processing unit 91 takes the captured image uploaded from electronic device 71 and information obtained from preprocessing as input data, inputs it to a trained model that has undergone machine learning, performs inference processing, and provides the inference results to the transmitting electronic device 71. Specifically, in response to the captured image (SDR image including noise) received from electronic device 71, processing unit 91 provides the inference results (SDR image with noise removed) obtained using an algorithm that uses the trained model generated by the learning device 31. In other words, server 72 is a server that executes image processing algorithms (algorithms using trained models), such as a noise reduction signal processing algorithm, and can provide image processing services such as noise reduction services.

[0079] Furthermore, the electronic device 71, acting as an imaging device, may have functions corresponding to the processing device 51 in Figure 22 and execute an image processing algorithm using a trained model. Alternatively, the semiconductor chip 81, acting as a semiconductor device, may have functions corresponding to the processing device 51 in Figure 22 and execute an image processing algorithm using a trained model. By the electronic device 71 or the semiconductor chip 81 executing the image processing algorithm, the captured image (SDR image including noise) can be processed locally (local AI processing) to obtain an inference result (SDR image with noise removed). In addition, the processing unit 91 of the server 72 may have functions corresponding to the learning device 31 in Figure 21 and generate a trained model using training data. The trained model may be updated after generation based on data such as training data.

[0080] <Computer Configuration> The series of processes described above can be executed by hardware or by software. When the series of processes are executed by software, the programs that make up that software are installed on the computer. Figure 24 is a block diagram showing an example of the hardware configuration of a computer that executes the series of processes described above by program.

[0081] In a computer, the CPU (Central Processing Unit) 101, ROM (Read Only Memory) 102, and RAM (Random Access Memory) 103 are interconnected by a bus 104. An input / output interface 105 is further connected to the bus 104. An input / output interface 105 is connected to an input unit 106, an output unit 107, a storage unit 108, a communication unit 109, and a drive 110.

[0082] The input unit 106 consists of a keyboard, mouse, microphone, etc. The output unit 107 consists of a display, speaker, etc. The storage unit 108 consists of a hard disk, non-volatile memory, etc. The communication unit 109 consists of a network interface, etc. The drive 110 drives a removable recording medium 111 such as semiconductor memory, magnetic disk, optical disk, or magneto-optical disk.

[0083] In a computer configured as described above, the CPU 101 loads programs recorded in the ROM 102 and memory unit 108 into the RAM 103 via the input / output interface 105 and bus 104, and executes them, thereby performing the series of processes described above.

[0084] The program executed by the computer (CPU 101) can be provided by recording it on a removable recording medium 111, such as a packaged media. The program can also be provided via wired or wireless transmission media, such as a local area network, the internet, or digital satellite broadcasting.

[0085] In a computer, a program can be installed in the storage unit 108 via the input / output interface 105 by inserting the removable recording medium 111 into the drive 110. Alternatively, a program can be received by the communication unit 109 via a wired or wireless transmission medium and installed in the storage unit 108. Furthermore, programs can be pre-installed in the ROM 102 or the storage unit 108.

[0086] In this specification, the processes performed by a computer according to a program do not necessarily have to be performed chronologically in the order described in the flowchart. That is, the processes performed by a computer according to a program include processes that are executed in parallel or individually (e.g., parallel processing or object-based processing). Furthermore, the program may be processed by one computer (processor) or it may be processed in a distributed manner by multiple computers.

[0087] The embodiments described herein are not limited to those described above, and various modifications are possible without departing from the spirit of this disclosure. Furthermore, the effects described herein are merely illustrative and not limiting, and other effects may also occur.

[0088] Furthermore, this disclosure can take the following form.

[0089] (1) An image processing apparatus comprising a processing unit that virtually generates a signal distribution consisting of a second grayscale higher than the first grayscale and a second signal range higher than the first signal range, based on a single first image having a first grayscale and a first signal range. (2) The image processing apparatus according to (1), wherein the processing unit extracts regions where the signal value is saturated from the first image based on an arbitrary threshold, and counts the number of regions and labels the extracted saturated regions. (3) The image processing apparatus according to (2), wherein the first image has one or more channels, and the processing unit performs processing according to an arbitrary combination of channels. (4) The image processing apparatus according to (2) or (3), wherein the processing unit generates the signal distribution for each saturated region using an arbitrary method. (5) The image processing apparatus according to (4), wherein the processing unit normalizes the generation result of the signal distribution generated by distance transformation using a predetermined method using the value of the maximum distance. (6) The image processing apparatus according to (4) or (5), wherein the processing unit performs calculations on the generation result of the signal distribution using an arbitrary function or random number distribution. (7) The processing unit multiplies each of the saturated regions by an arbitrary gain value, as described in any of (2) to (6). (8) The first image has two or more channels, and the processing unit estimates the signal balance between channels in the saturated region based on information about the peripheral region of the saturated region, and performs processing based on the estimation result for each of the saturated regions, as described in any of (2) to (7). (9) The processing unit adds the signal distribution generated in all of the saturated regions to the unsaturated region, which is a region where the signal value is not saturated, to generate a second image having a second grayscale and a second signal range, as described in any of (2) to (8). (10) The second image, or a third image generated from the second image, is used in at least one of a training process for generating a trained model and an inference process using the trained model, as described in (9).(11) An image processing method comprising an image processing device virtually generating a signal distribution consisting of a second grayscale higher than the first grayscale and a second signal range higher than the first signal range, based on a single first image having a first grayscale and a first signal range. (12) A program that causes a computer to function as an image processing device equipped with a processing unit that virtually generates a signal distribution consisting of a second grayscale higher than the first grayscale and a second signal range higher than the first signal range, based on a single first image having a first grayscale and a first signal range. (13) A generation method comprising a learning device that takes the third image and at least one of the first image and the second image as input and generates a machine learning model for removing noise contained in an image, wherein the second image is a virtual signal distribution based on the first image and is generated based on the signal distribution consisting of the second gradation and the second signal range, and the third image is generated by applying a process to the second image that includes at least a process for reproducing image changes. (14) A generation method comprising: an image processing device virtually generating a signal distribution consisting of a second grayscale higher than the first grayscale and a second signal range higher than the first signal range, based on a single first image having a first grayscale and a first signal range; and generating a second image having the second grayscale and the second signal range based on the signal distribution. (15) The generation method according to (14), further comprising applying a process to the second image that includes at least a process for reproducing image changes to generate a third image.(16) An imaging device comprising a processing unit that takes an input of an image containing noise and outputs an image from which the noise has been removed using an algorithm that uses a machine learning model, wherein the model is generated by taking the third image and at least one of the first image and the second image as input and performing machine learning to remove noise contained in the image, the second image is a virtual signal distribution based on the first image and is generated based on the signal distribution consisting of the second tone and the second signal range, and the third image is generated by applying a process to the second image that includes at least a process to reproduce image changes. (17) A semiconductor device comprising a processing unit that takes an input image containing noise and outputs an image from which the noise has been removed using an algorithm that uses a machine learning model, wherein the model is generated by taking the third image and at least one of the first image and the second image as input and performing machine learning to remove noise contained in the image, the second image is a virtual signal distribution based on the first image and is generated based on the signal distribution consisting of the second tone and the second signal range, and the third image is generated by applying a process to the second image that includes at least a process to reproduce image changes.(18) Information processing device comprising: a communication unit that transmits or receives data to or from another device via the Internet; and a processing unit that outputs an image from which the noise has been removed, obtained using an algorithm with a machine learning model, in response to an input image containing noise received from the other device, wherein the communication unit transmits the image from which the noise has been removed to the other device; the model is generated by taking the third image and at least one of the first image and the second image as input and performing machine learning to remove noise contained in the image; the second image is a virtual signal distribution based on the first image and is generated based on the signal distribution consisting of the second tone and the second signal range; and the third image is generated by applying a process to the second image that includes at least a process to reproduce image changes.

[0090] 1 Image processing system, 11 Imaging device, 12 External device, 13 Storage device, 14 Image processing device, 15 Input / output device, 20 Processing unit, 21 Image acquisition unit, 22 Parameter acquisition unit, 23 Image pre-processing unit, 24 Virtual signal distribution generation unit, 25 Image post-processing unit, 31 Learning device, 41 Acquisition unit, 42 Pre-processing unit, 43 Learning unit, 44 Output unit, 51 Processing unit, 61 Acquisition unit, 62 Pre-processing unit, 63 Inference unit, 64 Output unit, 71-1 to 71-N Electronic equipment, 72 Server, 73 Internet, 81-1 to 81-N Semiconductor chip, 91 Processing unit

Claims

1. An image processing apparatus comprising a processing unit that virtually generates a signal distribution consisting of a second grayscale higher than the first grayscale and a second signal range higher than the first signal range, based on a single first image having a first grayscale and a first signal range.

2. The image processing apparatus according to claim 1, wherein the processing unit extracts regions where the signal value is saturated from the first image based on an arbitrary threshold, and counts the number of regions and labels the extracted saturated regions.

3. The image processing apparatus according to claim 2, wherein the first image has one or more channels, and the processing unit performs processing according to any combination of channels.

4. The image processing apparatus according to claim 2, wherein the processing unit generates the signal distribution for each saturation region using an arbitrary method.

5. The image processing apparatus according to claim 4, wherein the processing unit normalizes the generation result of the signal distribution generated by distance conversion using a predetermined method using the value of the maximum distance.

6. The image processing apparatus according to claim 4, wherein the processing unit performs calculations on the signal distribution generation result using an arbitrary function or random number distribution.

7. The image processing apparatus according to claim 3, wherein the processing unit multiplies each of the saturation regions by an arbitrary gain value.

8. The image processing apparatus according to claim 7, wherein the first image has two or more channels, the processing unit estimates the signal balance between channels in the saturated region based on information regarding the peripheral region of the saturated region, and performs processing based on the estimation result for each saturated region.

9. The image processing apparatus according to claim 4, wherein the processing unit adds the signal distribution generated in all of the saturated regions to the unsaturated region, which is a region where the signal value is not saturated, to generate a second image having the second grayscale and the second signal range.

10. The image processing apparatus according to claim 9, wherein the second image, or a third image generated from the second image, is used in at least one of a training process for generating a trained model and an inference process using the trained model.

11. An image processing method comprising an image processing device virtually generating a signal distribution consisting of a second grayscale higher than the first grayscale and a second signal range higher than the first signal range, based on a single first image having a first grayscale and a first signal range.

12. A program that causes a computer to function as an image processing apparatus comprising a processing unit that virtually generates a signal distribution consisting of a second grayscale higher than the first grayscale and a second signal range higher than the first signal range, based on a single first image having a first grayscale and a first signal range.

13. A generation method comprising: a learning device that takes the third image and at least one of the first image and the second image as input and generates a machine learning model for removing noise contained in an image, wherein the second image is a virtual signal distribution based on the first image and is generated based on the signal distribution consisting of the second color tone and the second signal range, and the third image is generated by applying a process to the second image that includes at least a process for reproducing image changes.

14. A generation method comprising: an image processing device virtually generating a signal distribution consisting of a second grayscale higher than the first grayscale and a second signal range higher than the first signal range, based on a single first image having a first grayscale and a first signal range; and generating a second image having the second grayscale and the second signal range based on the signal distribution.

15. The generation method according to claim 14, further comprising applying a process to the second image that includes at least a process for reproducing image changes to generate a third image.

16. An imaging device comprising a processing unit that takes an input of an image containing noise and outputs an image from which the noise has been removed using an algorithm that uses a machine learning model, wherein the model is generated by taking the third image and at least one of the first image and the second image as input and performing machine learning to remove noise contained in the image, the second image is a virtual signal distribution based on the first image and is generated based on the signal distribution consisting of the second tone and the second signal range, and the third image is generated by applying a process to the second image that includes at least a process to reproduce image changes.

17. A semiconductor device comprising a processing unit that takes an input image containing noise and outputs an image from which the noise has been removed using an algorithm that employs a machine learning model, wherein the model is generated by taking the third image and at least one of the first image and the second image as input and performing machine learning to remove noise contained in the image, the second image is a virtual signal distribution based on the first image and is generated based on the signal distribution consisting of the second tone and the second signal range, and the third image is generated by applying a process to the second image that includes at least a process to reproduce image changes.

18. Information processing device comprising: a communication unit that transmits or receives data to or from another device via the Internet; and a processing unit that outputs an image from which the noise has been removed, obtained using an algorithm with a machine learning model, in response to an input image containing noise received from the other device, wherein the communication unit transmits the image from which the noise has been removed to the other device; the model is generated by taking the third image and at least one of the first image and the second image as input and performing machine learning to remove noise contained in the image; the second image is a virtual signal distribution based on the first image and is generated based on the signal distribution consisting of the second tone and the second signal range; and the third image is generated by applying a process to the second image that includes at least a process to reproduce image changes.

Images