Light receiving device, image processing method, program, and image processing system

Abstract

A light receiving device comprising a demosaicing unit that performs demosaicing on an image, and a pre-processing unit that performs correction processing on the image by means of an anti-aliasing filter before the demosaicing by the demosaic processing unit.

Description

Light receiving device, image processing method, program, and image processing system

[0001] The present technology relates to a light receiving device, an image processing method, a program, and an image processing system.

[0002] In a light receiving device, color separation is performed using different color filters such as R, G, and B. In this process, there are cases where the sampling frequencies for respective colors are different. Colors with low sampling frequencies are more susceptible to aliasing, which refers to the folding of high-frequency components into the low-frequency domain. In an image, aliasing causes colors that do not originally exist in the subject, leading to the problem of false color generation. False color is a phenomenon in which a color that does not actually exist in the subject appears in an image.

[0003] Accordingly, a technology has been proposed that suppresses false color generation by, when performing linear matrix processing on an image signal before interpolation, calculating the average value of color signals of a plurality of pixels of the same filter color arranged around the pixel to be corrected, for the color signals of other filter colors at the position of the pixel to be corrected, and performing linear matrix calculation (Patent Document 1)

[0004] Japanese Unexamined Patent Application Publication No. 2006-129264

[0005] For false color, there is a demand for a technology that suppresses its occurrence with higher accuracy, and there is also a demand for a technology that suppresses false color that occurs in a specific subject.

[0006] The present technology has been made in view of such problems, and an object thereof is to provide a light receiving device, an image processing method, a program, and an image processing system that can suppress the generation of false colors.

[0007] In order to solve the above-mentioned problem, a first technology provides a light receiving device including: a demosaicing processing unit that performs demosaicing processing on an image; and a preprocessing unit that performs color correction processing on the image with an anti-aliasing filter before the demosaicing processing by the demosaicing processing unit.

[0008] A second technology is an image processing method in which, before performing demosaicing processing on an image, color correction processing is performed on the image with an anti-aliasing filter.

[0009] The third technique is a program that causes a computer to perform an image processing method that applies color correction to an image using an anti-aliasing filter before demosaicing the image.

[0010] The fourth technology is an image processing system comprising a light-receiving device equipped with pixels, a demosaicing processing unit that performs demosaicing on an image generated by the light-receiving device, and an image processing device that performs color correction on the image using an anti-aliasing filter before the demosaicing by the demosaicing processing unit.

[0011] This is a block diagram showing the configuration of camera 10. This is a block diagram showing the configuration of light receiving device 100. This is a block diagram showing the configuration of image processing device 200 in the first embodiment. This is a block diagram showing the configuration of image processing system 1000. This is a diagram showing an RGB Bayer array type color filter. This is a diagram showing a basic anti-aliasing filter. This is a diagram showing the effect of the basic anti-aliasing filter. This is a diagram showing a second anti-aliasing filter. This is a diagram showing a reduction matrix for the second anti-aliasing filter. This is a diagram showing a reduction matrix for the basic anti-aliasing filter. This is a block diagram showing the configuration of image processing device 200 in the second embodiment. This is a diagram showing an adjustment matrix. This is a diagram showing a second anti-aliasing filter adjusted with the adjustment matrix. This is a diagram showing a basic anti-aliasing filter adjusted with the adjustment matrix. This is a diagram showing the effect of this technology. This is a diagram showing a modified example of the second anti-aliasing filter.

[0012] The embodiments of this technology will be described below with reference to the drawings. The description will be in the following order. <First Embodiment> [Configuration of Camera 10] [Configuration of Light Receiving Device 100 and Image Processing Device 200] [Processing in Pre-processing Unit 202 and Post-processing Unit 205] <Second Embodiment> [Configuration of Image Processing Device 200] [Processing in Filter Adjustment Unit 207 and Pre-processing Unit 202] <Modified Examples>

[0013] <First Embodiment> [Configuration of Camera 10] The configuration of the camera 10 and the light receiving device 100 will be described with reference to Figure 1.

[0014] The camera 10 is comprised of a light receiving device 100, a control unit 11, a storage unit 12, and a communication unit 13, and captures R, G, B (Red, Green, Blue) images and video consisting of multiple consecutive frame images.

[0015] The light-receiving device 100 is also called an image sensor, and examples of light-receiving devices 100 include CMOS (Complementary Metal Oxide Semiconductor) image sensors, CCD (Charge Coupled Device) image sensors, and intelligent vision sensors. An intelligent vision sensor is a sensor that uses a stacked structure in which a pixel chip and a logic chip are stacked, and the logic chip is equipped with image processing functions such as AI.

[0016] The control unit 11 consists of a CPU (Central Processing Unit), ROM (Read Only Memory), and RAM (Random Access Memory). The CPU functions as an arithmetic processing unit that performs various processing tasks and controls the camera 10 as a whole and its individual parts. The CPU executes various processes according to the program stored in the ROM or the program loaded into the RAM from the storage unit 12. The RAM stores data and other information necessary for the CPU to perform various processes as appropriate.

[0017] The storage unit 12 is a storage medium such as a hard disk, SSD (Solid State Drive), or flash memory. Various applications, image data, and other data are stored in the storage unit 12. However, if all images captured and output data from the camera 10 are saved to external storage, the camera 10 does not need to have a storage unit 12.

[0018] The communication unit 13 connects the camera 10 to an external device in a way that enables communication. Communication methods include SerDes-based data transfer methods such as GMSL (Gigabit Multimedia Serial Link) and FPD-link (registered trademark), as well as methods such as MIPI (Mobile Industry Processor Interface) and I2C (Inter-Integrated Circuit). Alternatively, communication methods such as HDMI (High-Definition Multimedia Interface), USB (Universal Serial Bus), Wi-Fi, Bluetooth (registered trademark), Wireless LAN (Local Area Network), NFC (Near Field Communication), and Ethernet (registered trademark) may be used.

[0019] The camera 10 may also include an input unit for user operation, a display unit for displaying images and GUI (Graphical User Interface), and drives and connection ports for connecting external storage media, etc.

[0020] Camera 10 may be equipped with known functions (such as subject detection, face detection, and scene detection) that can detect various types of information from images generated during shooting. These information detection functions can be implemented using machine learning or deep learning methods, template matching methods, matching methods based on the brightness distribution information of the subject, artificial intelligence methods, instance segmentation, and the like.

[0021] In this embodiment, the camera 10 is described as an in-vehicle camera mounted on an automobile. However, the camera 10 equipped with the light-receiving device 100 of this technology is not limited to in-vehicle use and can be used for any application.

[0022] [Configuration of the light receiving device 100 and the image processing device 200] Next, the configuration of the light receiving device 100 and the image processing device 200 will be described with reference to Figures 2 and 3.

[0023] The light receiving device 100 is composed of a pixel chip 110, a readout circuit 120, and a logic chip 130.

[0024] The pixel chip 110 has multiple pixels arranged in a two-dimensional array, each performing photoelectric conversion to output a pixel signal.

[0025] The readout circuit 120 reads out pixel signals from each pixel of the pixel chip 110 and supplies them to the logic chip 130.

[0026] The logic chip 130 is equipped with signal processing circuits and a DSP (Digital Signal Processor) memory, and is capable of performing ISP (Image Signal Processor) processing and AI (Artificial Intelligence) processing. In this technology, the logic chip 130 functions as an image processing device 200.

[0027] As shown in Figure 3, the image processing device 200 includes a white balance processing unit 201, a pre-processing unit 202, a tone mapping unit 203, a demosaicing processing unit 204, a post-processing unit 205, and a gamma correction unit 206.

[0028] The image processing device 200 receives a RAW image as image data. This RAW image is, for example, a RAW image obtained by reading the values ​​of each pixel in the pixel chip 110 in raster order, that is, a RAW image that maintains the pixel arrangement according to the Bayer array.

[0029] The white balance processing unit 201 performs white balance processing to balance the RGB levels of the image data based on a predetermined white balance setting.

[0030] The preprocessing unit 202 performs color correction processing on the image data before the demosaicing unit 204 performs demosaicing, by applying an anti-aliasing low-pass filter (hereinafter referred to as an anti-aliasing filter) to suppress false colors caused by moiré patterns. Details of the processing in the preprocessing unit 202 will be described later.

[0031] The tone mapping unit 203 performs tone mapping processing to compress the gradation of the image data.

[0032] The demosaicing processing unit 204 performs demosaicing on the image data and generates a color image containing R, G, and B signals for each pixel by referring to the R, G, and B pixel information adjacent to each pixel according to the arrangement of the Bayer array color filters and performing color interpolation.

[0033] The post-processing unit 205 performs color correction on the image data that has been demosaiced by the demosaicing unit 204. Details of the processing in the post-processing unit 205 will be described later.

[0034] The gamma correction unit 206 applies gamma correction to the image data, which corrects the gradation according to the gradation characteristics.

[0035] In this technology, since the pre-processing unit 202 only needs to perform processing before the demosaicing unit 204, the pre-processing unit 202 may also perform processing after the tone mapping unit 203.

[0036] The image processing device 200 performs the above processing on the image data. The image data processed by the image processing device 200 is stored in the memory unit 12 of the camera 10, or output from the camera 10 to an external device via the communication unit 13. If the camera 10 is an in-vehicle camera mounted on a car, the image data is output to the ECU (Electronic Control Unit) or SOC (Security Operation Center) that controls the car, and is used for purposes such as autonomous driving control and parking surveillance. In addition to the processing described above, the image processing device 200 may also perform gain correction, compression, and other processing.

[0037] The image processing device 200 is configured as described above. In this embodiment, the light receiving device 100 functions as the image processing device 200, and the light receiving device 100 performs the image processing according to this technology. However, as shown in Figure 4, the light receiving device 100 and the image processing device 200 can constitute an image processing system 1000. In this case, the light receiving device 100 does not function as the image processing device 200, and electronic devices such as the camera 10, personal computer, smartphone, server, or control units such as ECUs and SOCs that control automobiles have the function of the image processing device 200 and perform the image processing according to this technology. Alternatively, the image processing method in this technology may be executed by the light receiving device 100, electronic devices, or control units executing a program. The program may be pre-installed on electronic devices or control units, or it may be distributed via download or storage media for users to install.

[0038] [Processing in pre-processing unit 202 and post-processing unit 205] Next, the processing in pre-processing unit 202 and post-processing unit 205 will be explained. First, the processing in pre-processing unit 202 will be explained. In this embodiment, as an example, the light receiving device 100 will be described as performing color separation using an RGB Bayer array type color filter as shown in Figure 5.

[0039] The preprocessing unit 202 is characterized by performing color correction processing by applying an anti-aliasing filter to the image data before mosaic processing by the demosaicing unit 204. A typical anti-aliasing filter is the 11 filter. The 11 filter is shown in Figure 6A, and when it is represented as a two-dimensional array, it is shown in Figure 6B. In the following explanation, the outputs will be represented as R', Gr', Gb', B' for inputs R, Gr, Gb, B. This typical 11 filter is called the basic anti-aliasing filter.

[0040] In the basic anti-aliasing filter shown in Fig. 6B, the influence of aliasing is suppressed, and the occurrence of moire and false colors is suppressed by attenuating input R by multiplying it by 0.5, multiplying input Gr and input Gb by 0.25, and mixing G into output R'. In addition, in the basic anti-aliasing filter, the influence of aliasing is suppressed, and the occurrence of moire and false colors is suppressed by attenuating input B by multiplying it by 0.5, multiplying input Gr and input Gb by 0.25, and mixing G into output B'.

[0041] Since the sampling frequency (Nyquist frequency) of R and B is half that of G, R and B are more susceptible to aliasing caused by folded signal values. The fact that R and B are susceptible to aliasing means that moire caused by aliasing is more likely to occur, and false colors are generated when the balance of R, G, and B is disrupted. For example, in the case of yellow composed of R and G, a phenomenon occurs where R is emphasized and the color appears reddish. Therefore, by applying the basic anti-aliasing filter to attenuate signal values around the Nyquist frequency of G, which has a large influence, moire can be suppressed.

[0042] Fig. 7 shows the effect of the basic anti-aliasing filter. Fig. 7A shows an example of actual signal values of image data, where the solid line represents ideal signal values (signal values actually possessed by the subject), and the broken line represents folded signal values (signal values not actually possessed by the subject) in frequency components exceeding the Nyquist frequency. Fig. 7B is an example of signal values (acquired signal values) of image data acquired through imaging. As indicated by the hatched portion, the acquired signal values include signal values not present in the ideal signal values due to folding.

[0043] By applying a 1x1 filter as the basic anti-aliasing filter to the acquired signal values, as shown in Fig. 7C, the acquired signal values near the Nyquist frequency can be attenuated. This allows R, G, and B to approach their original ratios, and can suppress the occurrence of false colors caused by moire.

[0044] In the present technology, the anti-aliasing filter is adjusted and used in accordance with the color of the subject that is the target of color correction.

[0045] When the camera 10 is used as an in-vehicle camera, among various subjects captured by the in-vehicle camera, there is a problem that false colors are likely to occur in LED (Light Emitting Diode) traffic lights used in specific countries or regions. In particular, among red, blue, and yellow signal lights, there is a problem that red mixes into yellow where the R and G signal levels are equal, causing the yellow signal light to appear red. This problem arises due to the frequency characteristics of the signal lights of LED traffic lights.

[0046] Therefore, the present technology will be described by taking as an example the case of suppressing false colors that occur in image data when capturing an LED traffic light, particularly the mixing of red into yellow signal lights. Accordingly, the preprocessing unit 202 performs color correction processing using the basic anti-aliasing filter shown in FIG. 6 adjusted for the purpose of suppressing red mixed into yellow signal lights of LED traffic lights. Hereinafter, the basic anti-aliasing filter adjusted for the purpose of suppressing red mixed into yellow signal lights of LED traffic lights will be referred to as a second anti-aliasing filter.

[0047] The second anti-aliasing filter shown in FIG. 8 is adjusted to correct only red, in order to suppress red in the yellow signal light. Therefore, to suppress the output R', the input R is multiplied by 0.5, and the input Gr and input Gb that are adjacent to R and constitute yellow are multiplied by 0.25, which is the same as in the basic anti-aliasing filter.

[0048] On the other hand, since the second anti-aliasing filter is intended to suppress red in the yellow signal light, the output Gr', output Gb', and output B' are adjusted not to be corrected. Accordingly, the output Gr' is the same as the input Gr, the output Gb' is the same as the input Gb, and further the output B' is adjusted to be the same as the input B. In this way, the second anti-aliasing filter is adjusted so as not to correct colors other than red, which is the target of correction. This makes it possible to suppress the occurrence of false colors without affecting colors other than red of the target yellow signal light in the image data.

[0049] The preprocessing unit 202 applies a second anti-aliasing filter to the image data and performs color correction processing before the demosaicing process is carried out by the demosaicing unit 204.

[0050] Note that the color of the LED traffic light used as the target of processing is merely an example, and the basic anti-aliasing filter can be used to adjust the color of other subjects. Furthermore, the basic anti-aliasing filter can be adjusted according to the type of subject and the color to be corrected, and then processed in the pre-processing unit 202.

[0051] Next, the processing in the post-processing unit 205 will be described.

[0052] Generally, image development processes include color correction using linear matrices or color correction matrices (referred to as color correction matrices) to adjust the R, G, and B signal values ​​to match the characteristics of the human eye. The post-processing unit 205 performs color correction using this color correction matrix after the demosaicing process performed by the demosaicing unit 204.

[0053] When the preprocessing unit 202 applies a second anti-aliasing filter to the image data and performs color correction processing, the R, G, and B signal values ​​are blended, affecting colors other than the target color for processing, resulting in a problem of reduced color reproduction.

[0054] Therefore, in this technology, a new color correction matrix is ​​created by superimposing a table on the color correction matrix that is generally used in the color correction process performed after demosaicing, in which the preprocessing unit 202 applies a second anti-aliasing filter to the image data to reduce the blended colors (or, to put it another way, mitigate the influence of the blended colors). Hereafter, this new color correction matrix will be referred to as the second color correction matrix.

[0055] Then, after the color correction processing by the pre-processing unit 202 and the demosaicing processing by the demosaicing unit 204, the post-processing unit 205 performs color correction processing using the second color correction matrix. This restores the color reproduction of the subject, which has been reduced due to color blending by the pre-processing unit 202 applying the second anti-aliasing filter to the image data.

[0056] In this embodiment, we will describe as an example the case in which the preprocessor 202 performs color correction processing using the second anti-aliasing filter for LED traffic lights shown in Figure 8. Specifically, the second color correction matrix restores the color reproduction of the subject by reducing the G blended into R by applying the second anti-aliasing filter. In this way, the postprocessor 205 performs color correction processing on the image using a second color correction matrix that superimposes a table for reducing the blended colors by the color correction processing of the preprocessor 202.

[0057] If we define the color correction matrix used in the color correction process generally performed after demosaicing as the first color correction matrix M1, the reduction matrix CM as the table for reducing the blended colors by the processing in the preprocessing unit 202, and the second color correction matrix used in the postprocessing unit 205 as M2, then the second color correction matrix M2 can be calculated using the following formula 1.

[0058] [Formula 1] M2=CM×M1

[0059] A table that reduces the G component from R and B is superimposed on the first color correction matrix M1 as a reduction matrix CM to form the second color correction matrix M2. Then, after processing by the pre-processing unit 202 and the demosaicing unit 204, the post-processing unit 205 performs color correction processing on the second color correction matrix M2, thereby restoring the color reproduction that has been reduced due to color mixing by the pre-processing unit 202 applying a second anti-aliasing filter to the image data.

[0060] Figure 9 shows the reduction matrix CM when the preprocessing unit 202 applies a second anti-aliasing filter to the image data to perform color correction. Due to the second anti-aliasing filter, the input R is multiplied by 0.5 in output R', so the coefficient of input R in output R' is doubled in the reduction matrix CM. Furthermore, to reduce the G component that increased due to doubling the coefficient of input R, the coefficient of G in output R' is multiplied by -1 in the reduction matrix CM. This restores the original color of the subject and subtracts other colors. Note that in the second anti-aliasing filter, output Gr', output Gb', and output B' are the same as input Gr, input Gb, and input B, respectively, so there is no need to correct the input R, G, B in output G' and the input R, G, B in output B' in the reduction matrix. In this way, the reduction matrix CM is determined according to the coefficient of the second anti-aliasing filter applied by the preprocessing unit 202.

[0061] Furthermore, when the preprocessing unit 202 applies the basic anti-aliasing filter shown in Figure 6 to the image data and performs color correction processing, the reduction matrix is ​​as shown in Figure 10. In this case, the coefficients of output R' and output G' for the reduction matrix are the same as the reduction matrix when the preprocessing unit 202 uses the second anti-aliasing filter. Also, because the input B is multiplied by 0.5 in output B' due to the basic anti-aliasing filter, the coefficient of R in output B' is doubled in the reduction matrix. Furthermore, to reduce the G component that increased due to doubling the coefficient of input B, the coefficient of G in output B' is multiplied by -1. This restores the original color of the subject and subtracts other colors.

[0062] The post-processing unit 205 performs color correction processing on the image data using the second color correction matrix M2 after the demosaicing processing performed by the demosaicing processing unit 204.

[0063] The processing of the image processing device 200 in the first embodiment is performed as described above.

[0064] <Second Embodiment> [Configuration of Image Processing Device 200] Next, a second embodiment of the present technology will be described. The configuration of the camera 10 and the light receiving device 100 is the same as in the first embodiment. In the second embodiment as well, the light receiving device 100 has the functions of an image processing device 200, and the light receiving device 100 is mounted on the camera 10. However, in the second embodiment as well, the image processing system 1000 may be configured with the light receiving device 100 and the image processing device 200, similar to the first embodiment.

[0065] As shown in Figure 11, the image processing apparatus 200 includes a white balance processing unit 201, a pre-processing unit 202, a tone mapping unit 203, a demosaicing processing unit 204, a post-processing unit 205, and a gamma correction unit 206, similar to the first embodiment. However, in the second embodiment, the pre-processing unit 202 performs color correction processing on the image data before the white balance processing unit 201. The fact that the pre-processing unit 202 performs color correction processing on the image data before the demosaicing processing unit 204 is the same as in the first embodiment.

[0066] Furthermore, in the second embodiment, the image processing device 200 includes a filter adjustment unit 207. In the second embodiment, the processing of the preprocessing unit 202 is performed before the white balance processing. Therefore, since the white balance processing is performed after the processing of the preprocessing unit 202, the anti-aliasing filter needs to take the effect of white balance into consideration, and the optimal coefficient changes for each scene. Therefore, the filter adjustment unit 207 adjusts the anti-aliasing filter taking white balance into consideration.

[0067] [Processing in the filter adjustment unit 207 and pre-processing unit 202] Next, the processing in the filter adjustment unit 207 and pre-processing unit 202 will be described. Note that the processing in the white balance processing unit 201, tone mapping unit 203, demosaicing processing unit 204, post-processing unit 205, and gamma correction unit 206 is the same as in the first embodiment.

[0068] Similar to the first embodiment, the second embodiment suppresses false colors that occur in LED traffic lights used in specific countries or regions, particularly the mixing of red with yellow traffic lights, among the various subjects captured by the in-vehicle camera. Therefore, the preprocessor 202 performs color correction processing using a second anti-aliasing filter.

[0069] In the second embodiment, white balance processing is performed after processing by the preprocessing unit 202. Therefore, it is necessary to consider that the white balance processing performed after processing by the preprocessing unit 202 applies white balance gain to the inputs Gr and Gb included in the output R'. The white balance gain consists of an R gain that adjusts the redness of the image and a B gain that adjusts the blueness of the image.

[0070] First, the filter adjustment unit 207 calculates the white balance gain based on the input image data, which is a RAW image. The filter adjustment unit 207 may also obtain the white balance gain from shooting control information in the camera 10, the characteristics of the camera 10, the user's white balance setting in the camera 10, scene information obtained from the camera 10, or a white balance gain calculation device or white balance gain calculation software. In addition, the filter adjustment unit 207 may pre-set the white balance gain for a specific subject. In this embodiment, as an example, the white balance gain is set to R gain = 2.19 and B gain = 1.78.

[0071] The filter adjustment unit 207 adjusts the second anti-aliasing filter according to the white balance gain using the adjustment matrix shown in Figure 12. Specifically, the filter adjustment unit 207 multiplies the coefficients of input Gr and input Gb used to calculate the output R' in the second anti-aliasing filter by the reciprocal of the R gain, which is the white balance gain. This is done to cancel the white balance gain applied to input Gr and input Gb included in the output R' by the processing in the white balance processing unit 201 after the processing in the pre-processing unit 202, by multiplying by the reciprocal of the R gain in advance.

[0072] Furthermore, the filter adjustment unit 207 uses the adjustment matrix shown in Figure 12 to multiply the coefficients of input Gr and input Gb, which are used to calculate the output B' in the second anti-aliasing filter, by the reciprocal of the white balance gain, B gain. This process takes into account that after processing by the pre-processing unit 202, the white balance gain is applied to Gr and Gb included in the output B' by processing in the white balance processing unit 201, and multiplies by the reciprocal of the B gain in advance to cancel that white balance gain.

[0073] Furthermore, the filter adjustment unit 207 uses the adjustment matrix shown in Figure 12 to multiply the coefficient of input R for calculating output Gr' by the R gain in the white balance gain, and multiply the coefficient of input B for calculating output Gr' by the B gain in the white balance gain. In addition, the coefficient of input R for calculating output Gb' is also multiplied by the R gain in the white balance gain, and the coefficient of input B for calculating output Gb' is also multiplied by the B gain in the white balance gain.

[0074] As described above, if the white balance gain is R gain = 2.19 and B gain = 1.78, the filter adjustment unit 207 adjusts the second anti-aliasing filter using the adjustment matrix, resulting in the image shown in Figure 13.

[0075] Furthermore, when the basic anti-aliasing filter is adjusted using the adjustment matrix shown in Figure 12, the basic anti-aliasing filter becomes as shown in Figure 14. When the preprocessing unit 202 applies the basic anti-aliasing filter to the image data and performs color correction processing, the filter adjustment unit 207 adjusts the basic anti-aliasing filter in this manner.

[0076] The preprocessing unit 202 performs color correction processing on the image data before demosaicing using the second anti-aliasing filter adjusted by the filter adjustment unit 207. Then, in the second embodiment, after the color correction processing by the preprocessing unit 202, the white balance processing unit 201 performs white balance processing on the image data.

[0077] Processing is performed using this technology as described above. This technology suppresses false colors caused by aliasing in image data by applying an anti-aliasing filter. Meanwhile, color reproduction can be maintained for subjects where false colors do not occur. Furthermore, by adjusting the anti-aliasing filter according to a specific subject, false colors occurring in that subject can be suppressed.

[0078] In particular, when color correction is performed using the second anti-aliasing filter in the above-described embodiment, it is possible to suppress the mixing of red with the yellow signal light of the LED traffic light, as shown in Figures 15A to 15C.

[0079] Furthermore, since demosaicing is a process that generates a color image containing R, G, and B signals for each pixel by performing color interpolation by referring to the R, G, and B pixel information adjacent to each pixel, the occurrence of moiré and false colors can be suppressed more effectively by performing processing in the preprocessing unit 202 before demosaicing to reduce the effects of aliasing.

[0080] In both the first and second embodiments, the pre-processing unit 202 may have multiple types of anti-aliasing filters in advance and perform color correction processing using an anti-aliasing filter determined by automatic selection based on scene detection results, subject detection results, etc., or by user selection. Furthermore, the post-processing unit 205 may have multiple second color correction matrices in advance, each of which a reduction matrix corresponding to one of the multiple types of anti-aliasing filters in the pre-processing unit 202 is superimposed, and may select a second color correction matrix according to the anti-aliasing filter used by the pre-processing unit 202 and perform processing.

[0081] <Modifications> Although embodiments of this technology have been described in detail above, this technology is not limited to the embodiments described above, and various modifications are possible based on the technical concept of this technology.

[0082] In this embodiment, filter 11 is used as the anti-aliasing filter, but in this technology, any filter that can be used as an anti-aliasing filter may be used.

[0083] In the embodiment, the example of suppressing the mixing of red with yellow signal lights in LED traffic lights was used, but this technology is not limited to that, and color correction can be performed by adjusting the anti-aliasing filter according to the subject and color to be color corrected. For example, in order to suppress the mixing of blue with yellow, color correction is performed using an anti-aliasing filter as shown in Figure 16.

[0084] In this embodiment, the camera 10 equipped with the light-receiving device 100 is mounted on an automobile. However, the camera 10 equipped with the light-receiving device 100 of this technology is not limited to automobiles, but can be mounted on various mobile devices such as motorcycles, bicycles, drones, airplanes, personal mobility devices, ships, and robots. Furthermore, the camera 10 equipped with the light-receiving device 100 of this technology is not limited to mobile devices, but may be mounted on other devices or equipment, or used as a standalone camera.

[0085] The technology can also take the following configurations: (1) A light receiving device comprising: a demosaicing processing unit that performs demosaicing on an image; and a preprocessing unit that performs color correction processing on the image using an anti-aliasing filter before the demosaicing processing by the demosaicing processing unit. (2) The light receiving device according to (1), wherein the anti-aliasing filter is adjusted according to the color of the subject to be corrected. (3) The light receiving device according to (1) or (2), wherein the anti-aliasing filter is an 11 filter. (4) The light receiving device according to (3), wherein the 11 filter as the anti-aliasing filter is adjusted so as not to correct any color other than R (red) and B (blue) which are to be corrected. (5) The light receiving device according to any one of (1) to (4), further comprising a postprocessing unit that performs color correction processing on the image using matrix processing after the demosaicing processing by the demosaicing processing unit. (6) The light receiving device according to (5), wherein the postprocessing unit performs color correction processing using a color correction matrix adjusted according to the processing of the preprocessing unit. (7) The light receiving device according to (6), wherein the color correction matrix is ​​adjusted to reduce the colors blended in the color correction processing in the preprocessing unit. (8) The light receiving device according to any one of (1) to (7), further comprising a white balance processing unit that performs white balance processing on the image after the color correction processing by the preprocessing unit. (9) The light receiving device according to (8), further comprising a filter adjustment unit that adjusts the anti-aliasing filter according to the white balance gain. (10) The light receiving device according to any one of (1) to (9), mounted on an in-vehicle camera. (11) The light receiving device according to (2), wherein the anti-aliasing filter is adjusted to suppress the red color in the yellow signal light of an LED traffic light. (12) An image processing method that performs color correction processing on an image using an anti-aliasing filter before demosaicing the image. (13) A program that causes a computer to execute the image processing method that performs color correction processing on an image using an anti-aliasing filter before demosaicing the image.(14) An image processing system comprising: a light receiving device having pixels; a demosaicing processing unit that performs demosaicing on an image generated by the light receiving device; and an image processing device that performs color correction on the image using an anti-aliasing filter before the demosaicing by the demosaicing processing unit.

[0086] 10...Camera 100...Light receiving device 200...Image processing device 201...White balance processing unit 202...Pre-processing unit 204...Demosaicing processing unit 205...Post-processing unit 207...Filter adjustment unit 1000...Image processing system

Claims

1. A light receiving device comprising: a demosaicing processing unit that performs demosaicing on an image; and a preprocessing unit that performs color correction processing on the image using an anti-aliasing filter before the demosaicing processing by the demosaicing processing unit.

2. The light receiving device according to claim 1, wherein the anti-aliasing filter is adjusted according to the color of the subject to be corrected.

3. The light receiving device according to claim 1, wherein the anti-aliasing filter is an 11 filter.

4. The light receiving device according to claim 3, wherein the 11 filter, which serves as the anti-aliasing filter, is adjusted so as not to correct any element other than either R (red) or B (blue) which are the elements to be corrected.

5. The light receiving device according to claim 1, further comprising a post-processing unit that performs color correction processing on the image by matrix processing after demosaicing by the demosaicing unit.

6. The light receiving device according to claim 5, wherein the post-processing unit performs color correction processing using a color correction matrix adjusted according to the processing of the pre-processing unit.

7. The light receiving device according to claim 6, wherein the color correction matrix is ​​adjusted to reduce the blended colors in the color correction processing in the preprocessing unit.

8. The light receiving device according to claim 1, further comprising a white balance processing unit that performs white balance processing on the image after the color correction processing by the preprocessing unit.

9. The light receiving device according to claim 8, further comprising a filter adjustment unit for adjusting the anti-aliasing filter according to the white balance gain.

10. The light receiving device according to claim 1, which is mounted on an in-vehicle camera.

11. The light receiving device according to claim 2, wherein the anti-aliasing filter is adjusted to suppress the red color in the yellow signal light of an LED traffic light.

12. An image processing method that performs color correction on an image using an anti-aliasing filter before demosaicing the image.

13. A program that causes a computer to perform an image processing method that applies color correction to an image using an anti-aliasing filter before demosaicing the image.

14. An image processing system comprising: a light receiving device having pixels; a demosaicing processing unit that performs demosaicing on an image generated by the light receiving device; and an image processing device that performs color correction on the image using an anti-aliasing filter before the demosaicing by the demosaicing processing unit.

Images