Image processing apparatus and image processing method

Abstract

Provided are an image processing apparatus and an image processing method that are capable of suppressing interruption of imaging due to protection of a sensor by a shutter. An image processing apparatus (100) is provided with: a saturation detection unit that detects a saturated pixel region in thermal image data; a distance information acquisition unit that acquires distance information corresponding to the saturated pixel region; a determination unit that, on the basis of the distance information, makes a determination for controlling opening and closing of a shutter that blocks light from entering a sensor; and a shutter control unit that controls the shutter on the basis of the determination result of the determination unit.

Description

Image Processing Apparatus and Image Processing Method

[0001] The present disclosure relates to an image processing apparatus and an image processing method.

[0002] Patent Document 1 discloses an image processing apparatus that processes image data captured by a far-infrared sensor. This apparatus detects a saturated region where pixels are saturated. The image processing apparatus controls a dynamic range in accordance with the detection result of the saturated region. Then, the opening and closing of a shutter is controlled based on the detection result of the saturated region and the setting of the dynamic range.

[0003] Japanese Unexamined Patent Publication No. 2021-110552

[0004] In infrared sensors such as microbolometers used in infrared imaging apparatuses, there is a problem that sensor elements deteriorate when high-intensity light is incident thereon. For example, when sunlight is directly incident on an infrared sensor, pixels are burned. In this case, the sensitivity of the burned pixels changes, resulting in deterioration of image quality.

[0005] In Patent Document 1, the sensor element is protected by closing the shutter. However, there is a problem that imaging cannot be performed during the period when the shutter is closed.

[0006] The present disclosure has been made in view of the above points, and provides an image processing apparatus and an image processing method capable of suppressing interruption of imaging caused by protecting a sensor with a shutter.

[0007] The image processing apparatus according to the present embodiment includes: a thermal image data acquisition unit that acquires thermal image data captured by a sensor; a saturation detection unit that detects a saturated pixel region in the thermal image data; a distance information acquisition unit that acquires distance information corresponding to the saturated pixel region; a determination unit that, when the saturated pixel region is detected, performs determination for controlling opening and closing of a shutter that blocks incidence of light to the sensor based on the distance information corresponding to the saturated pixel region; and a shutter control unit that controls opening and closing of the shutter based on a determination result of the determination unit.

[0008] The image processing apparatus according to this embodiment includes: a thermal image data acquisition unit that acquires thermal image data captured by a sensor; a saturation detection unit that detects a saturated pixel region in the thermal image data; a recognition unit that recognizes an object captured in the thermal image data; a determination unit that, when the saturated pixel region is detected, makes a determination to control the opening and closing of a shutter that blocks the incidence of light to the sensor based on the overlap between the recognition range in which the object is recognized and the saturated pixel region; and a step of controlling the opening and closing of the shutter based on the result of the determination.

[0009] The image processing method according to this embodiment includes the steps of: acquiring thermal image data captured by a sensor; detecting saturated pixel regions in the thermal image data; acquiring distance information corresponding to the saturated pixel regions; if the saturated pixel regions are detected, making a determination based on the distance information corresponding to the saturated pixel regions to control the opening and closing of a shutter that blocks the incidence of light to the sensor; and a shutter control unit that controls the opening and closing of the shutter based on the result of the determination.

[0010] The image processing method according to this embodiment includes the steps of: acquiring thermal image data captured by a sensor; detecting a saturated pixel region in the thermal image data; recognizing an object captured in the thermal image data; if the saturated pixel region is detected, making a determination to control the opening and closing of a shutter that blocks the incidence of light to the sensor based on the overlap between the recognized range in which the object is recognized and the saturated pixel region; and controlling the opening and closing of the shutter based on the result of the determination.

[0011] The present disclosure aims to provide an image processing device and an image processing method that can suppress interruptions in imaging caused by protecting a sensor with a shutter.

[0012] This is a control block diagram showing the configuration of the imaging system. This is a control block diagram showing the configuration of the infrared camera. This is a control block diagram showing the configuration of the image processing unit. This is a diagram showing thermal image data when direct sunlight is incident. This is a diagram showing thermal image data when reflected sunlight is incident. This is a flowchart showing the image processing method according to Embodiment 1. This is a control block diagram showing the configuration of the imaging system according to Embodiment 2. This is a flowchart showing the image processing method according to Embodiment 2. This is a control block diagram showing the configuration of the image processing unit according to Embodiment 3. This is a control block diagram showing the configuration of the image processing unit according to Embodiment 4. This is a flowchart showing the image processing method according to Embodiment 4.

[0013] Specific embodiments applying this disclosure will be described in detail below with reference to the drawings. However, this disclosure is not limited to the following embodiments. Also, for clarity, the following descriptions and drawings have been simplified as appropriate.

[0014] Embodiment 1 An image processing device and imaging system according to this embodiment will be described with reference to Figure 1. Figure 1 is a block diagram showing the configuration of the imaging system 1. The imaging system 1 comprises an image processing device 100, an infrared camera 200, and a display device 300. The imaging system 1 is, for example, an imaging system for vehicles used in vehicles to acquire thermal images of the surroundings, or an imaging system used for surveillance purposes.

[0015] Here, the imaging system 1 is described as an in-vehicle system mounted on a vehicle. By mounting an infrared camera 200 on the vehicle, safe driving support can be provided by detecting heat-generating objects such as people. The infrared camera 200 is mainly used to support visual safety checks, such as at night. The infrared camera 200 is also effective when the ambient light level is low, such as around sunrise and sunset or inside tunnels, even when it is not nighttime.

[0016] The infrared camera 200 captures thermal images (infrared images). The infrared camera 200 captures thermal images at, for example, 15 to 30 frames per second. For example, the infrared camera 200 captures thermal images by detecting far-infrared radiation from a subject. The infrared camera 200 is a far-infrared (FIR) camera capable of visualizing heat (far-infrared radiation) emitted from an object. The thermal image data captured by the infrared camera 200 is also called thermal image data (or infrared image data). The thermal image data shows the temperature distribution within the imaging range of the infrared camera 200.

[0017] The infrared camera 200 captures images of the area in front of or around the vehicle. The infrared camera 200 outputs thermal image data to the image processing device 100. The thermal image data input from the infrared camera 200 to the image processing device 100 is also called input thermal image data. Thermal image data refers to thermal image data captured by the infrared camera. The configuration of the infrared camera 200 will be described later.

[0018] The image processing device 100 generates display image data or image data for object recognition by performing image processing on the input thermal image data. For example, the image processing device 100 performs various processes such as correcting pixel variations in the infrared camera 200 and removing noise caused by changes in characteristics over time. Furthermore, the image processing device 100 functions as a control device for controlling the infrared camera 200. For example, the image processing device 100 controls the shutter of the infrared camera 200. Specifically, it closes the shutter to protect the infrared camera 200 when sunlight enters the infrared camera 200. The image processing device 100 also closes the shutter when the sensor 230, described later, performs calibration by imaging the shutter.

[0019] The image processing device 100 outputs the processed display image data to the display device 300. The display image data that has been processed by the image processing device 100 and output to the display device 300 is also called the output thermal image data. In other words, the input thermal image data is the data before processing by the image processing device 100, and the output thermal image data is the data after processing by the image processing device 100.

[0020] The display device 300 is equipped with a liquid crystal display or the like for displaying display image data. The display device 300 displays display image data that has been processed by the image processing device 100. This allows the display device 300 to display a more accurate temperature distribution.

[0021] The image processing device 100 performs object recognition processing on the image data that has undergone image processing. The image processing device 100 uses a learning model of a person, a vehicle, etc., to recognize objects contained in the image data and outputs the recognition result to a display device 300 or the like. The recognition result may be output on the display device 300 so as to be superimposed on the output thermal image.

[0022] The image processing device 100 includes a control IF (interface) 110, a ROM (Read Only Memory) 120, a RAM (Random Access Memory) 130, a system control unit 140, a thermal image data acquisition unit 160, an image processing unit 170, a display image data output unit 180, and a distance information acquisition unit 190.

[0023] The thermal image data acquisition unit 160 acquires thermal image data captured by the infrared camera 200. For example, it is equipped with an interface (IF) for inputting thermal image data from the infrared camera 200. The interface between the thermal image data acquisition unit 160 and the infrared camera 200 may be a wired connection or a wireless connection.

[0024] ROM 120 stores control programs and various parameters for controlling the image processing device 100. For example, ROM 120 stores the image processing program executed by the image processing unit 170. ROM 120 also stores the system control program executed by the system control unit 140.

[0025] The RAM 130 stores various programs and parameters used for their execution. Furthermore, it stores calculation data for the image processing unit 170.

[0026] The system control unit 140 controls the entire imaging system 1. The system control unit 140 is equipped with a processor such as a CPU (Central Processing Unit). The processor of the system control unit 140 can control the entire system by reading control programs for controlling each functional unit (control IF 110, shutter control unit 141, thermal image data acquisition unit 160, image processing unit 170, display image data output unit 180, and distance information acquisition unit 190) from ROM into RAM and executing them.

[0027] The system control unit 140 includes a shutter control unit 141. The shutter control unit 141 controls the opening and closing of the shutter of the infrared camera 200. The shutter control unit 141 outputs an instruction to close the shutter of the infrared camera 200. In other words, the shutter control unit 141 outputs a control signal to open and close the shutter.

[0028] The control IF 110 is an interface for controlling the infrared camera 200. For example, the shutter control unit 141 outputs a control signal via the control IF 110 to open and close the shutter of the infrared camera 200. The image processing device 100 can also transmit various signals and data to the infrared camera 200 via the control IF 110. Furthermore, the control IF 110 can receive various signals and data from the infrared camera 200. For example, the control IF 110 acquires the ambient temperature measured by the temperature sensor of the infrared camera 200.

[0029] The image processing unit 170 performs predetermined image processing on the thermal image data acquired by the thermal image data acquisition unit 160. The image processing unit 170 may include a processor for executing an image processing program. The processor of the image processing unit 170 may be a general-purpose processor or a dedicated processor. The processor of the image processing unit 170 and the processor of the system control unit 140 may be the same. The processing in the image processing unit 170 will be described later.

[0030] The distance information acquisition unit 190 acquires distance information indicating the distance from the infrared camera 200 to the object, which is included in the thermal image data. The distance information acquisition unit 190 acquires distance information by estimating the distance based on the depth information derived from the thermal image data. Here, the distance (depth) from the infrared camera 200 to the object is defined as the object distance. The distance information corresponds to the object distance. The distance information acquisition unit 190 outputs the distance information to the image processing unit 170. The distance information acquisition unit 190 acquires the object distance from the infrared camera 200 to the object, which is included in the thermal image data. The image processing unit 170 performs image processing on the input thermal image data to generate display image data or image data for object recognition.

[0031] Here, the distance information acquisition unit 190 takes thermal image data as input and uses a depth estimation model 191 to acquire distance information indicating the distance from the infrared camera 200 to an object contained in the thermal image data. The depth estimation model 191 is, for example, a deep learning model such as Depth Anything. The depth estimation model 191 takes thermal image data as input and estimates the depth for each pixel. In other words, when thermal image data is input to the depth estimation model 191, the depth estimation model 191 outputs a value (depth) indicating the object distance for each pixel. The depth estimation model 191 outputs a value indicating the relative distance.

[0032] For example, the output value of the depth estimation model 191 can be multi-level, such as 8 bits. The output value of the depth estimation model 191 decreases as the object distance to the pixel increases. Of course, the output value may also increase as the object distance to the pixel increases. The distance information acquisition unit 190 estimates the object distance to the object being imaged at each pixel. The depth estimation model 191 may also estimate the object distance for each region containing multiple pixels, or for each object.

[0033] Such a depth estimation model 191 is constructed from training data that includes thermal image data and distance information. For example, the distance measurement results of a distance measuring sensor such as LiDAR (Light Detection and Ranging) are used as distance information. The training data used for deep learning includes distance information as the correct label. Distance information is associated with the data of each pixel in the thermal image data. When the depth estimation model 191 receives thermal image data as input, it outputs distance information for each pixel. Note that since thermal image data has color information similar to that of a visible light image, the input image data may be a visible light image. The image processing unit 170 may generate display image data from the thermal image data using the distance information. For example, the image processing unit 170 generates display image data by superimposing a value indicating the object distance onto the thermal image data. The depth estimation model 191 estimates the distance based on the depth information derived from the thermal image data. The distance information acquisition unit 190 then acquires the distance information based on the depth information estimated from the thermal image data. Hereinafter, in this embodiment, distance information means depth information.

[0034] The display image data output unit 180 is an interface that outputs the display image data to the display device 300. This allows the user, such as the vehicle driver, to view the display image data. Therefore, even in dark conditions such as at night, people and other objects around the vehicle can be recognized. Furthermore, by outputting the object recognition results along with the display image data to the display device 300, the user can view the recognition results of people and other objects around the vehicle.

[0035] The detailed configuration of the infrared camera 200 will be explained using Figure 2. Figure 2 is a schematic block diagram showing the configuration of the infrared camera 200. The infrared camera 200 includes a lens 210, a shutter 220, a sensor 230, a transmission device 240, and a temperature sensor 250.

[0036] The lens 210 forms an image of far-infrared light from the subject onto the light-receiving surface of the sensor 230. The lens 210 has at least one lens element. For example, the lens 210 may have multiple lenses, such as a zoom lens and a focus lens.

[0037] The sensor 230 has multiple pixels. Each pixel of the sensor 230 receives far-infrared light from the subject. This allows for the capture of a thermal image of the subject. For example, the sensor 230 is a microbolometer for detecting far-infrared radiation. The sensor 230 has multiple pixels arranged in a two-dimensional array. The detected value (detection signal) of each pixel forms a thermal image of the subject.

[0038] The transmission device 240 serves as an interface for transmitting various signals and data to the image processing device 100. The transmission device 240 transmits thermal image data captured by the sensor 230 to the image processing device 100. The transmission device 240 also receives control signals from the image processing device 100. Furthermore, the transmission device 240 may transmit lens information related to the zoom and focus of the lens 210.

[0039] The temperature sensor 250 measures the temperature in the operating environment of the infrared camera. The transmission device 240 transmits the ambient temperature measured by the temperature sensor 250 to the image processing device 100. The ambient temperature measured by the temperature sensor 250 is used for shutterless correction, which involves calibrating the sensor 230 without closing the shutter.

[0040] The shutter 220 is positioned on the front side of the sensor 230. The shutter 220 blocks far-infrared light that enters the sensor 230 from the outside. The shutter 220 has an opening and closing mechanism. The shutter 220 is opened and closed by a control signal from the shutter control unit 141. When the shutter 220 is closed (hereinafter also simply referred to as the closed state), external light from the lens 210 does not enter the sensor 230. In other words, in the closed state, far-infrared light from the shutter 220 enters the sensor 230. When the shutter 220 is open (hereinafter also simply referred to as the open state), far-infrared light from the lens 210 enters the sensor 230. The shutter 220 can make the imaging area of ​​the sensor 230 a thermally uniform surface. For example, the shutter 220 may be a barrier or cover positioned on the front side of the lens 210, or it may be positioned between the lens 210 and the sensor 230. Closing the shutter prevents the sensor 230 from burning out.

[0041] The detailed configuration of the image processing unit 170 will be explained using Figure 3. Figure 3 is a block diagram showing the configuration of the image processing unit 170. The image processing unit 170 includes a defective pixel correction unit 171, a Non-Uniformity Correction (NUC) unit 172, a saturation detection unit 174, a determination unit 175, and a tone mapping unit 176. The image processing unit 170 in Figure 3 may also include a recognition unit (not shown) for image data output from the NUC unit 172. For example, the recognition unit recognizes and labels objects contained in the image data. The image processing unit 170 works in cooperation with the system control unit 140 and the RAM 130 to execute predetermined processing.

[0042] The defective pixel correction unit 171 interpolates the pixel values ​​of defective pixels of the sensor 230, which are pre-stored in the RAM 130, using the pixel values ​​of pixels surrounding the defective pixel. The defective pixel correction unit 171 performs the above interpolation process on the input thermal image data acquired by the thermal image data acquisition unit 160, and outputs the interpolated image data to the NUC unit 172 and the saturation detection unit 174.

[0043] The NUC unit 172 performs processing for correcting output variation between pixels. For example, the NUC unit 172 performs shutterless correction based on the environmental temperature detected by the temperature sensor 250. Since a known method can be used for the shutterless correction in the NUC unit 172, description thereof is omitted.

[0044] The tone mapping unit 176 performs tone mapping on the thermal image data subjected to NUC correction. For example, the tone mapping unit 176 adjusts the overall luminance of the thermal image data using a histogram of the thermal image data. For example, when the dynamic range of the display device 300 is narrower than the dynamic range of the infrared camera 200, the tone mapping unit 176 may perform dynamic range compression or the like, compress the dynamic range which is a ratio of the brightest luminance to the darkest luminance of the thermal image data, and adjust the dynamic range to fall within a range displayable by the display device 300. Further, the tone mapping unit 176 may use Contrast Limited Adaptive Histogram Equalization (CLAHE), which can suppress overexposure and underexposure through local processing and obtain thermal image data with high contrast. This makes it possible to adjust the dynamic range of the thermal image data to a dynamic range displayable by the display device 300. In addition, overexposure and the like can be suppressed, and thermal image data with high contrast can be obtained. Since a known method can be used for the tone mapping in the tone mapping unit 176, description thereof is omitted.

[0045] The image processing unit 170 outputs the tone-mapped thermal image data as output thermal image data to the display image data output unit 180. Note that the image processing unit 170 may perform AGC (Auto Gain Control) processing or the like instead of tone mapping.

[0046] The saturation detection unit 174 detects pixel saturation in the thermal image data. The saturation detection unit 174 then detects saturated pixel regions in the thermal image data where pixel saturation has occurred. The saturation detection unit 174 identifies the pixel addresses of the saturated pixels. A saturated pixel is a pixel whose pixel output value (referred to as pixel data) has reached its upper limit. For example, if the brightness level of each pixel in the imaging data is represented by 14 bits ranging from zero to 16383, a pixel at a coordinate where the pixel output value is 16383 is called a saturated pixel. Of course, the pixel output value of a saturated pixel is not limited to the upper limit; it may also be a value indicating a predetermined level or higher. This allows the image processing unit 170 to identify saturated pixel regions in the thermal image data where pixels are saturated. A saturated pixel region is a region where saturated pixels or substantially saturated pixels exist consecutively.

[0047] The determination unit 175 makes a determination to control the opening and closing of the shutter when a saturated pixel region is detected. Specifically, the determination unit 175 determines whether the pixel saturation is due to the incidence of direct sunlight or reflected sunlight. Direct sunlight refers to sunlight from the sun that directly enters the infrared camera 200. Reflected sunlight refers to sunlight from the sun that is reflected by structures such as building walls or glass and then enters the infrared camera 200. The determination unit 175 receives distance information from the distance information acquisition unit 190. Here, the distance information acquisition unit 190 acquires distance information indicating the object distance to the object imaged with the saturated pixel. The determination unit 175 makes a determination based on the distance information corresponding to the saturated pixel region. The determination process of the determination unit 175 will be described later.

[0048] When direct sunlight is incident on the infrared camera 200, the amount of light incident on the infrared camera 200 becomes extremely high. Accordingly, burn-in occurs on the pixels (microbolometers) of the infrared camera 200. When sunlight is reflected by a building wall or window glass and is incident on the infrared camera 200, the amount of light incident on the infrared camera 200 is lower than that of the direct light. Therefore, when pixel saturation occurs due to reflected sunlight, burn-in of the pixels of the infrared camera 200 does not occur. Then, the determination unit 175 determines whether the pixel saturation in the thermal image data is caused by incidence of direct light or incidence of reflected light, based on distance information associated with saturated pixels in the thermal image data acquired from the distance information acquisition unit 190. The image processing unit 170 outputs a determination signal indicating the determination result to the system control unit 140.

[0049] The shutter control unit 141 controls opening and closing of the shutter 220 in accordance with the determination signal indicating the determination result. When the determination unit 175 determines that saturation is caused by incidence of direct light, the shutter control unit 141 outputs a control signal for closing the shutter 220. Accordingly, incidence of light to the infrared camera 200 can be blocked. When the determination unit 175 determines that saturation is caused by incidence of reflected light, the shutter control unit 141 outputs a control signal for opening the shutter. Accordingly, infrared imaging of the surroundings of the vehicle can be continued.

[0050] A method for determining whether pixel saturation in a saturated pixel region is caused by incidence of direct light or incidence of reflected light will be described. Fig. 4 is a diagram showing thermal image data when direct sunlight is incident (a schematic diagram showing an image P1 in which the sun is included in the angle of view of the infrared camera 200), and Fig. 5 is a diagram showing thermal image data when reflected sunlight is incident (a schematic diagram showing an image P2 in which the sun is not included in the angle of view of the infrared camera 200).

[0051] In Figure 4, the sun S is included in the field of view of the infrared camera 200, so the sun S and the surrounding area become the saturated pixel region R1. In Figure 5, the sun S is not included in the field of view of the infrared camera 200, but sunlight is reflected by the window glass of building O and incident on the infrared camera 200. Therefore, the area corresponding to the window glass of building O becomes the saturated pixel region R2.

[0052] The distance information indicates the object distance to the object captured by the pixels in the saturated pixel regions R1 and R2. In other words, the distance information indicates the distance from the infrared camera 200 to the subject. Here, the subject in saturated pixel region R1 is the sun S. The distance information associated with the pixels in saturated pixel region R1 of image P1 indicates the distance from the infrared camera 200 to the sun S. The subject in saturated pixel region R2 is a building O. The distance information associated with the pixels in saturated pixel region R2 of image P2 indicates the distance from the infrared camera 200 to the building O.

[0053] The distance to the sun S is practically infinite. Therefore, the distance to the sun S is greater than the distance to the building O. If the object distance to the object captured by the pixels in the saturated pixel region is greater than or equal to the threshold distance, the determination unit 175 determines that the saturation is due to the incidence of direct light. In other words, if the object in the saturated pixel region is farther than the threshold distance, the determination unit 175 determines that the saturation is due to the incidence of direct light. If the object distance to the object captured by the pixels in the saturated pixel region is less than the threshold, the determination unit 175 determines that the saturation is due to the incidence of reflected light. In other words, if the object in the saturated pixel region is closer than the threshold distance, the determination unit 175 determines that the saturation is due to the incidence of reflected light. In this way, the determination unit 175 makes a determination according to the object distance to the subject captured by the pixels included in the saturated pixel regions R1 and R2.

[0054] In this way, the determination unit 175 can appropriately determine whether the saturation is due to the incidence of direct sunlight or reflected light. Therefore, the shutter 220 can be closed when direct light is incident. Since the infrared camera 200 can be protected by the shutter 220, pixel burn-in can be prevented. When direct light is not incident, the shutter 220 can be kept open. Therefore, even when the infrared camera 200 is protected by the shutter 220, thermal image data can be acquired appropriately. In other words, the period during which thermal image data cannot be acquired can be shortened.

[0055] Next, the image processing method according to this embodiment will be described using Figure 6. Figure 6 is a flowchart of the image processing method according to this embodiment 1. First, the thermal image data acquisition unit 160 acquires thermal image data from the infrared camera 200 (step S101). The distance information acquisition unit 190 acquires distance information (S102). Here, the thermal image data is input to the depth estimation model 191, and the depth estimation model 191 outputs a value indicating the object distance (depth information) for each pixel, thereby acquiring depth information as distance information.

[0056] The saturation detection unit 174 detects the saturated pixel region of the thermal image data (step S103). If there is no saturated pixel region (NO in S104), the shutter 220 opens (step S105). In other words, the shutter control unit 141 opens the shutter 220.

[0057] If there is a saturated pixel region (YES in S104), the determination unit 175 determines whether the object distance indicated in the distance information is greater than or equal to the threshold distance (S106). If the object distance is less than the threshold distance (NO in S106), the shutter 220 opens (step S105). In other words, the determination unit 175 determines that the saturation is due to the incidence of reflected light, so the shutter control unit 141 opens the shutter 220.

[0058] If the object distance is greater than or equal to the threshold distance (YES in S106), the shutter 220 closes (step S107). In other words, the determination unit 175 determines that saturation has occurred due to the incidence of direct light, and the shutter control unit 141 closes the shutter 220. The above process is repeated for each frame or at regular intervals. In this way, the determination unit 175 makes a determination in the pixels of the saturated pixel region based on the object distance indicated by the distance information. Even when the infrared camera 200 is protected by the shutter 220, thermal image data can be acquired appropriately.

[0059] Even if the shutter 220 is closed (step S107), the acquisition of thermal image data may continue. After the shutter 220 is closed, the determination unit 175 outputs an instruction to the shutter control unit 141 to open the shutter 220 after a predetermined period of time, such as 5 to 10 seconds, and returns to step S102. If the shutter remains open (step S105), the process returns to step S102.

[0060] In this way, the determination unit 175 makes a determination using distance information indicating the object distance to the subject (object) in the saturated pixel region. If the object distance is indicated as a relative distance, a threshold is set for that relative distance. For example, if the relative distance is represented by 8 bits, any value in the range of 0 to 255 (for example, 10) can be set as the threshold. Here, the output value of the depth estimation model 191 may be set so that the larger the relative distance, the smaller the output value of the depth estimation model 191. In this case, a threshold may be set for the output value of the depth estimation model 191. If the output value indicating the relative distance estimated by the depth estimation model 191 is less than or equal to the threshold, the object distance is determined to be greater than or equal to the threshold distance.

[0061] Although the object distance indicated by the distance information was a relative distance, it may also be an absolute distance. In this case, the distance information acquisition unit 190 or the image processing unit 170, etc., may perform calibration to convert the relative distance to an absolute distance. The distance information acquisition unit 190 can perform calibration using information such as the camera field of view, sensor cell pitch, and absolute distance reference point. When the object distance is indicated as an absolute distance, an absolute distance such as 100m becomes the threshold distance.

[0062] Furthermore, if the depth estimation model 191 is unable to estimate the object distance, the determination unit 175 may determine that it is saturation due to the incidence of direct light. In this case, if the depth estimation model 191 is unable to estimate the object distance, it may output a flag or the like to indicate that it is unable to estimate. Alternatively, if the depth estimation model 191 is unable to estimate the object distance, it may output the value that gives the maximum possible object distance. Also, if the depth estimation model 191 is below a predetermined position, it may determine that it is impossible to estimate. In this way, the determination unit 175 can determine that it is saturation due to the incidence of direct light when the object distance is greater than or equal to a threshold distance, or when it is impossible to estimate. This allows the shutter 220 to be closed quickly when it is not possible to appropriately determine whether the saturation is due to the incidence of direct light or the incidence of reflected light.

[0063] In this embodiment, the depth estimation model 191 estimates the object distance based on thermal image data. Therefore, there is no need to add a distance measuring sensor or a distance image sensor, which simplifies the device configuration.

[0064] Embodiment 2 Embodiment 2 will be described with reference to Figure 7. Figure 7 is a control block diagram showing the configuration of the imaging system 10 according to Embodiment 2. Embodiment 2 differs from Embodiment 1 in the method of acquiring distance information. Specifically, the imaging system 1 is further equipped with a distance image sensor 400. Another difference is that the distance information acquisition unit 190 does not use a depth estimation model 191 to acquire distance information. Details common to Embodiment 1 will be omitted as appropriate. For example, the image processing unit 170 has the same configuration as in Figure 3.

[0065] The distance image sensor 400 measures the distance to an object by irradiating each pixel of the sensor with infrared light and using the time it takes for the light to reflect back from the object's surface. It then generates and outputs distance image data corresponding to the object distance for each pixel of the sensor. The distance image sensor 400 is, for example, a Time of Flight (ToF) camera. Alternatively, the distance image sensor 400 may be a stereo camera or a LiDAR. The distance image data captured by the distance image sensor 400 is two-dimensional image data. The value of each pixel in the distance image data indicates the object distance to the subject. The distance image sensor 400 outputs the distance image data to the distance information acquisition unit 190.

[0066] For example, the distance image sensor 400 performs imaging in synchronization with the infrared camera 200. The distance image sensor 400 is installed so as to face the same direction as the infrared camera 200. The field of view of the distance image sensor 400 overlaps with the field of view of the infrared camera 200. Pixels are associated between the distance image data captured by the distance image sensor 400 and the thermal image data captured by the infrared camera 200. Specifically, the distance information acquisition unit 190 has a conversion table that converts the pixel addresses of the thermal image data to the pixel addresses of the distance image data. Based on the distance image data, the distance information acquisition unit 190 can determine the object distance to the object captured by the infrared camera 200.

[0067] The distance information acquisition unit 190 acquires distance information indicating the object distance to an object in the saturated pixel region based on the distance image data. For example, the distance information acquisition unit 190 converts the pixel address of a saturated pixel to the pixel address of the distance image data. The distance information acquisition unit 190 can then use the value indicated by that pixel address in the distance image data as the object distance. The distance information acquisition unit 190 then calculates distance information indicating the object distance to an object in the saturated pixel region. Hereinafter, in this embodiment, distance information means distance information based on distance image data.

[0068] Similar to Embodiment 1, the determination unit 175 determines, based on distance information, whether the pixel saturation is due to the incidence of direct light or reflected light. The determination unit 175 makes a determination in the pixels of the saturated pixel region based on the object distance indicated by the distance information. The determination unit 175 compares the object distance with the threshold distance and makes a determination based on the comparison result. This allows the determination unit 175 to appropriately determine whether the saturation is due to the incidence of direct light or reflected light. Even when the infrared camera 200 is protected by the shutter 220, thermal image data can be appropriately acquired.

[0069] Here, the object distance is an absolute distance. Therefore, the threshold distance is an absolute distance such as 100m. The threshold distance may also be set according to the sensing range of the distance image sensor 400. If the maximum sensing distance of the distance image sensor 400 is 200m, the threshold distance can be set to the maximum sensing distance of 200m.

[0070] Figure 8 is a flowchart of the image processing method according to this second embodiment. Figure 8 has basically the same steps as the flowchart in Figure 6, with the addition of step S202. Therefore, steps S101 to S107 in Figure 6 correspond to steps S201 and S203 to S208, respectively. Accordingly, explanations of content that overlaps with the first embodiment will be omitted as appropriate.

[0071] When the thermal image data acquisition unit 160 acquires thermal image data captured by the infrared camera 200 (S201), the distance information acquisition unit 190 acquires distance image data captured by the distance image sensor 400 (S202). As described above, the distance image sensor 400 and the thermal image data have overlapping fields of view and are captured in synchronous motion. Based on the distance image data, the distance information acquisition unit 190 acquires distance information indicating the distance to an object (S203).

[0072] The saturation detection unit 174 detects the saturated pixel region of the thermal image data (step S204). If there is no saturated pixel region (NO in S205), the shutter 220 opens (step S206). In other words, the shutter control unit 141 opens the shutter 220.

[0073] If there is a saturated pixel region (YES in S205), the determination unit 175 determines whether the object distance indicated in the distance information is greater than or equal to the threshold distance (S207). If the object distance is less than the threshold distance (NO in S207), the shutter 220 opens (step S206). In other words, the determination unit 175 determines that the saturation is due to the incidence of reflected light, so the shutter control unit 141 opens the shutter 220.

[0074] If the object distance is greater than or equal to the threshold distance (YES in S207), the shutter 220 closes (step S208). In other words, the determination unit 175 determines that saturation has occurred due to the incidence of direct light, and the shutter control unit 141 closes the shutter 220. The above process is repeated for each frame or at regular intervals. In this way, the determination unit 175 makes a determination in the pixels of the saturated pixel region based on the distance to the object indicated by the distance information. Even when the infrared camera 200 is protected by the shutter 220, thermal image data can be acquired appropriately.

[0075] Even if the shutter 220 is closed (step S208), the acquisition of thermal image data may continue. After the shutter 220 is closed, the determination unit 175 outputs an instruction to the shutter control unit 141 to open the shutter 220 after a predetermined period of time, such as 5 to 10 seconds, and returns to step S201. If the shutter remains open (step S206), the process returns to step S201.

[0076] Furthermore, if the distance image sensor 400 cannot measure the object distance, that is, if the emitted infrared light is not reflected back and the object distance cannot be measured, the determination unit 175 may determine that the saturation is due to the incidence of direct light. For example, if an object captured by a saturated pixel is farther away than the maximum sensing distance, the distance image sensor cannot measure the object distance. In this case, since the object distance cannot be measured, the determination unit 175 determines that the saturation is due to the incidence of direct light. Thus, if the object distance is greater than or equal to the threshold distance, or if it is impossible to measure, the determination unit 175 determines that the saturation is due to the incidence of direct light. This allows the shutter 220 to be closed quickly when it is not possible to appropriately determine whether the saturation is due to the incidence of direct light or the incidence of reflected light.

[0077] Furthermore, in Embodiment 2, since distance image data captured by the distance image sensor 400 is used, the object distance can be measured with high accuracy. This improves the judgment accuracy, allowing the shutter 220 to be controlled more appropriately.

[0078] Embodiment 3 An image processing apparatus according to Embodiment 3 will be described with reference to Figure 9. Figure 9 is a block diagram showing the configuration of the image processing unit 170 in Embodiment 3. In this embodiment, the image processing unit 170 includes a recognition unit 178. The configuration and processing other than the recognition unit 178 are the same as in Embodiments 1 and 2, so their explanation will be omitted as appropriate.

[0079] The tone mapping unit 176 outputs tone-mapped thermal image data to the recognition unit 178. The recognition unit 178 recognizes objects (subjects) contained in the thermal image data. For example, the recognition unit 178 can recognize objects by performing image processing such as contour extraction or pattern matching. Since known image processing methods can be used for image analysis processing in the recognition unit 178, a detailed explanation is omitted. The recognition unit 178 may perform object recognition using segmentation algorithms or deep learning models. The recognition unit 178 may label each pixel of the thermal image data. The recognition unit 178 assigns a different label to each subject (object) contained in the thermal image data.

[0080] The determination unit 175 makes a determination based on the recognition result from the recognition unit 178. If the object in the saturated pixel region is a building, the determination unit 175 determines that the saturation is due to the incidence of reflected light. If the recognition unit 178 cannot recognize an object in the saturated pixel, the determination unit 175 determines that the saturation is due to the incidence of direct light. Here, a building is a structure O such as a skyscraper, as shown in Figure 5. In this way, even if sunlight is reflected by a distant building, it is possible to prevent misjudging the saturation as being due to the incidence of direct light. The shutter can be controlled appropriately.

[0081] For example, if sunlight is reflected by the window glass of a distant building, the object distance to that building may exceed the threshold distance. In this embodiment, the determination accuracy can be improved by having the determination unit 175 make a determination based on the object recognition result. For example, if the recognition unit 178 recognizes an object in the saturated pixel region as a building, bridge, or other structure, the determination unit 175 determines that the saturation is due to the incidence of reflected light. In this way, by having the determination unit 175 make a determination based on the object recognition result, the determination of saturation due to the incidence of reflected light can be made with greater accuracy.

[0082] Furthermore, objects that are determined to be saturated due to reflected light are not limited to buildings, but may also be mobile objects such as automobiles, buses, trains, vehicles, drones, airplanes, helicopters, flying objects, and autonomous mobile robots. For example, pixel saturation may occur due to reflection from the windows of a vehicle. Alternatively, if a mobile object has a mirrored or reflective surface, pixel saturation may occur due to reflection from that surface. The recognition unit 178 can make accurate determinations by recognizing artificial objects such as buildings and mobile objects.

[0083] The configuration according to this embodiment can be combined with embodiments 1 and 2. Even if the object distance indicated by the distance information is greater than or equal to the threshold distance, if the object in the saturated pixel region is a building or a moving object, the determination unit 175 determines that it is a reflection due to the incidence of reflected light. In other words, if the object distance is greater than or equal to the threshold distance and no object is recognized in the saturated pixel region, the determination unit 175 determines that it is saturation due to the incidence of direct light. Alternatively, if the object distance is greater than or equal to the threshold distance and the sun is recognized in the saturated pixel region, the determination unit 175 determines that it is saturation due to the incidence of direct light.

[0084] In the above description, the recognition unit 178 recognizes an object based on the tone-mapped thermal image data, but it may also recognize an object based on the thermal image data before tone mapping. For example, the recognition unit 178 may recognize an object based on the thermal image data output by the NUC unit 172 or the thermal image data output by the defective pixel correction unit 171.

[0085] Furthermore, the recognition unit 178 may recognize an object based on image data other than thermal image data. For example, the recognition unit 178 may recognize an object based on distance image data captured by the distance image sensor 400. Alternatively, the recognition unit 178 may recognize an object using image data captured by another camera, such as a visible light camera. Furthermore, it may recognize an object using image data captured by two or more types of cameras or sensors.

[0086] Two or more embodiments from Embodiments 1 to 3 can be combined as appropriate. For example, in the configuration of Embodiment 1, the determination unit 175 may perform a determination based on the object recognition result as in Embodiment 3. Alternatively, in the configuration of Embodiment 2, the determination unit 175 may perform a determination based on the object recognition result as in Embodiment 3.

[0087] Embodiment 4 The image processing of the image processing device 100 in the imaging system according to Embodiment 4 will be described with reference to Figure 10. Figure 10 is a block diagram showing the configuration of the image processing device 100. This embodiment differs from Embodiments 1 to 3 in that the image processing device 100 is not provided with a distance information acquisition unit 190, and the image processing unit 170 is equipped with a recognition unit 178. Specifically, the determination unit 175 makes a determination based on the recognition result of the recognition unit 178. Details that are the same as those in the above embodiments will be omitted as appropriate. For example, the recognition unit 178 performs object recognition based on thermal image data or other camera image data, as shown in Embodiment 3. This makes it possible to recognize artificial objects such as buildings and moving objects. Here, it will be explained that the recognition unit 178 recognizes objects based on thermal image data.

[0088] When input thermal image data is received by the image processing unit 170, the defective pixel correction unit 171 corrects the defective pixels. The saturation detection unit 174 detects saturated pixels in the thermal image data after the defective pixels have been corrected. The saturation detection unit 174 then detects saturated pixel regions in the thermal image data where pixel saturation has occurred. The saturation detection unit 174 outputs the detection result of the saturated pixel region to the determination unit 175. The detection result of the saturated pixel region includes the pixel address and other information.

[0089] Furthermore, the NUC unit 172 and the tone mapping unit 176 perform processing on the thermal image data from which defective pixels have been corrected. The recognition unit 178 performs image recognition on the tone-mapped thermal image data. These processes are the same as described above, so their explanation is omitted. The recognition unit 178 outputs the recognition result to the determination unit 175. The recognition result associates the recognized object with its pixel address. The recognition unit 178 can recognize whether the object is a building or a moving object. If the recognition unit 178 recognizes a building or a moving object, it then identifies the recognition range.

[0090] When a saturated pixel area is detected, the determination unit 175 makes a determination based on the overlap between the recognition range in which a building or moving object is recognized as an object and the saturated pixel area. For example, if the overlap is greater than or equal to a certain percentage of the saturated pixel area, the determination unit 175 determines that the saturation is due to reflected light. In this case, the determination unit 175 outputs a determination signal to open the shutter. If the overlap is less than a certain percentage of the saturated pixel area, or if there is no overlap, the determination unit 175 determines that the saturation is due to direct light. In this case, the determination unit 175 outputs a determination signal to close the shutter. In this way, the shutter can be controlled appropriately even without using distance information.

[0091] The image processing method according to this embodiment will be explained using Figure 11. Figure 11 includes some steps that are common with Figures 6 and 8. Therefore, explanations of content that overlaps with the flow charts in Figures 6 and 8 will be omitted as appropriate.

[0092] The thermal image data acquisition unit 160 acquires thermal image data captured by the infrared camera 200 (S301). The recognition unit 178 recognizes objects captured in the thermal image data (S302). The recognition unit 178 can identify or classify the type of object. Therefore, the recognition unit 178 can recognize that there are buildings or moving objects. When the recognition unit 178 detects buildings or moving objects, it identifies the recognition range. Of course, the recognition unit 178 may also perform object recognition based on image data other than thermal image data.

[0093] The saturation detection unit 174 detects the saturated pixel region of the thermal image data (step S303). If there is no saturated pixel region (NO in S304), the shutter 220 opens (step S306). In other words, the shutter control unit 141 keeps the shutter 220 open.

[0094] If there is a saturated pixel region (YES in S304), the determination unit 175 determines whether the overlap between the saturated pixel region and the recognition range is greater than or equal to a certain level (step S305). For example, the determination unit 175 identifies the overlapping portion between the saturated pixel region and the recognition range in the thermal image data. The determination unit 175 determines the ratio of the overlapping portion to the saturated pixel region or the recognition portion. Then, the determination unit 175 determines whether the ratio of the area of ​​the overlapping portion is greater than or equal to a certain ratio.

[0095] Then, if the overlap is greater than a certain level (YES in S305), the shutter 220 opens (step S306). In other words, the determination unit 175 determines that saturation is due to the incidence of reflected light from the object, so the shutter control unit 141 opens the shutter 220. If the overlap is not greater than a certain level (NO in S305), the shutter 220 closes (step S307). In other words, the determination unit 175 determines that saturation is due to the incidence of direct light, so the shutter control unit 141 opens the shutter 220. For example, if there is no overlap at all, or if buildings or moving objects are not recognized as objects, there is a high possibility that saturation is occurring due to the incidence of direct light. Alternatively, if the saturated pixel area and the recognized area are significantly misaligned and only a part of them overlap, there is a high possibility that saturation is occurring due to the incidence of direct light. Thus, if the proportion of the overlap is less than a certain level, the shutter control unit 141 closes the shutter.

[0096] The above process is repeated for each frame or at regular intervals. In this way, the determination unit 175 makes a determination based on the overlap between the saturated pixel area and the recognition area. Even when the infrared camera 200 is protected by the shutter 220, thermal image data can be acquired appropriately. Furthermore, since the process of calculating the distance to the object is not required, the processing load can be reduced.

[0097] Furthermore, some or all of the processing in the image processing apparatus 100 described above can be implemented as a computer program. Such a program can be stored using various types of non-temporary computer-readable media and supplied to a computer. Non-temporary computer-readable media include various types of tangible recording media. Examples of non-temporary computer-readable media include semiconductor memory (e.g., mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (Random Access Memory)). Programs may also be supplied to a computer using various types of temporary computer-readable media. Examples of temporary computer-readable media include electrical signals, optical signals, and electromagnetic waves. Temporary computer-readable media can be supplied to a computer via wired communication channels such as electric wires and optical fibers, or via wireless communication channels.

[0098] The present invention has been described in detail above based on embodiments, but it goes without saying that the present invention is not limited to the above embodiments and can be modified in various ways without departing from its essence. It is also possible to combine two or more of the above embodiments as appropriate.

[0099] This application claims priority based on Japanese Patent Application No. 2025-051980, filed on 26 March 2025, and Japanese Patent Application No. 2025-156925, filed on 22 September 2025, and incorporates all of their disclosures herein.

[0100] 1.10 Imaging System 100 Image Processing Unit 110 Control IF 120 ROM 130 RAM 140 System Control Unit 141 Shutter Control Unit 160 Thermal Image Data Acquisition Unit 170 Image Processing Unit 171 Defective Pixel Correction Unit 172 NUC Unit 174 Saturation Detection Unit 175 Judgment Unit 176 Tone Mapping Unit 178 Recognition Unit 180 Display Image Data Output Unit 190 Distance Information Acquisition Unit 191 Depth Estimation Model 200 Infrared Camera 210 Lens 220 Shutter 230 Sensor 240 Transmission Device 250 Temperature Sensor 300 Display Device 400 Distance Image Sensor

Claims

1. An image processing apparatus comprising: a thermal image data acquisition unit that acquires thermal image data captured by a sensor; a saturation detection unit that detects saturated pixel regions in the thermal image data; a distance information acquisition unit that acquires distance information corresponding to the saturated pixel regions; a determination unit that, when a saturated pixel region is detected, makes a determination to control the opening and closing of a shutter that blocks the incidence of light to the sensor based on the distance information corresponding to the saturated pixel region; and a shutter control unit that controls the opening and closing of the shutter based on the determination result of the determination unit.

2. The image processing apparatus according to claim 1, wherein the determination unit determines whether to close the shutter if the distance information corresponding to the saturated pixel region is greater than or equal to a threshold distance or if the distance cannot be estimated.

3. The image processing apparatus according to claim 1, wherein the distance information acquisition unit acquires distance information corresponding to the saturated pixel region based on depth information based on the thermal image data.

4. The image processing apparatus according to claim 3, wherein the distance information acquisition unit acquires distance information corresponding to the saturated pixel region based on depth information estimated using a depth estimation model that takes the thermal image data as input.

5. The image processing apparatus according to claim 1, wherein the distance information acquisition unit acquires distance information corresponding to the saturated pixel region based on distance image data captured by the distance image sensor.

6. The image processing apparatus according to claim 1, wherein the determination unit determines, based on distance information corresponding to the saturated pixel region, whether the saturated pixel region is saturated due to the incidence of direct sunlight or due to the incidence of reflected sunlight.

7. The image processing apparatus according to claim 6, comprising: a recognition unit that recognizes an object captured in the thermal image data; and, if the recognition range in which the object is recognized overlaps with the saturated pixel region, the apparatus determines that the saturation is due to the incidence of reflected light.

8. An image processing apparatus comprising: a thermal image data acquisition unit that acquires thermal image data captured by a sensor; a saturation detection unit that detects saturated pixel regions in the thermal image data; a recognition unit that recognizes an object captured in the thermal image data; a determination unit that, when a saturated pixel region is detected, makes a determination to control the opening and closing of a shutter that blocks the incidence of light to the sensor based on the overlap between the recognition range in which the object is recognized and the saturated pixel region; and a shutter control unit that controls the opening and closing of the shutter based on the result of the determination.

9. The image processing apparatus according to claim 8, wherein the determination unit determines whether to close the shutter if the recognition range and the saturated pixel area do not overlap, or if the overlap is less than a certain percentage.

10. An image processing method comprising: acquiring thermal image data captured by a sensor; detecting saturated pixel regions in the thermal image data; acquiring distance information corresponding to the saturated pixel regions; if the saturated pixel regions are detected, making a determination to control the opening and closing of a shutter that blocks the incidence of light to the sensor based on the distance information corresponding to the saturated pixel regions; and controlling the opening and closing of the shutter based on the result of the determination.

11. An image processing method comprising: acquiring thermal image data captured by a sensor; detecting saturated pixel regions in the thermal image data; recognizing an object captured in the thermal image data;, if the saturated pixel region is detected, making a determination to control the opening and closing of a shutter that blocks the incidence of light to the sensor based on the overlap between the recognized range in which the object is recognized and the saturated pixel region; and controlling the opening and closing of the shutter based on the result of the determination.

Images