Distance image generation device, distance image generation method, and program

Abstract

A distance image generation device according to the present disclosure comprises: an acquisition unit that acquires a distance image and an infrared image obtained by receiving light reflected by a subject which reflects emitted light; a maximum value detection unit that divides the infrared image into a plurality of blocks, and acquires a maximum value of an infrared luminance value from pixels of a to-be-processed block and pixels of blocks around the to-be-processed block; a saturation determination unit that determines whether the maximum value of the infrared luminance value acquired by the maximum value detection unit is a saturation threshold value; and a correction unit that, on the basis of the infrared luminance value of pixels of the infrared image and the determined result of the saturation determination unit, corrects at least one among the infrared luminance value of the pixels of the infrared image and a distance measurement value of pixels of the distance image.

Description

Distance image generation apparatus, distance image generation method, and program

[0001] The present disclosure relates to a distance image generation apparatus, a distance image generation method, and a program.

[0002] An indirect ranging camera is known that performs distance measurement by irradiating light with light emission from a laser diode and receiving the reflected light thereof. It is also known as a publicly known technique that highly accurate distance measurement can be performed by using a multi-spot laser capable of performing multi-point irradiation and extracting a distance measurement value of the centroid position of each dot thereof.

[0003] For example, the following Patent Document 1 discloses a method of acquiring non-saturated dot irradiation in a saturated region by using dot irradiation and surface irradiation, calculating whether or not the number of saturated pixels is equal to or greater than a threshold value at the time of previously captured surface irradiation, and reducing the light amount of dot irradiation to be captured next when the number of saturated pixels is equal to or greater than the threshold value.

[0004] Japanese Unexamined Patent Publication No. 2024-105147

[0005] However, in the invention described in the above Patent Document 1, when a subject is a retroreflective plate or the like and the intensity of the reflected light thereof is large, if the light amount of dot irradiation is reduced to a light amount that does not cause saturation, there is a fear that dots having a small reflected light amount like a distant subject cannot be acquired due to insufficient light amount. Thus, there has been a problem that the accuracy of the distance image deteriorates due to the influence of optical diffuse reflection.

[0006] In view of the above problems, an object of the present disclosure is to provide a distance image generation apparatus, a distance image generation method, and a program capable of obtaining an accurate distance image even when there is optical diffuse reflection.

[0007] To solve the above-mentioned problems and achieve the objective, the distance image generation apparatus according to this disclosure includes: an acquisition unit that acquires a distance image and an infrared image obtained by receiving reflected light reflected by a subject from emitted light; a maximum value detection unit that divides the infrared image into a plurality of blocks and acquires the maximum value of the infrared brightness from the pixels of the block to be processed and the pixels of the blocks surrounding the block to be processed; a saturation determination unit that determines whether the maximum value of the infrared brightness obtained by the maximum value detection unit is a saturation threshold; and a correction unit that corrects at least one of the infrared brightness values ​​of the pixels of the infrared image and the distance measurement values ​​of the pixels of the distance image based on the infrared brightness values ​​of the pixels of the infrared image and the determination result of the saturation determination unit.

[0008] To solve the above-mentioned problems and achieve the objective, the distance image generation method according to the present disclosure includes the steps of: acquiring a distance image and an infrared image obtained by receiving reflected light reflected by a subject from emitted light; dividing the infrared image into a plurality of blocks and acquiring the maximum value of the infrared brightness from the pixels of the block to be processed and the pixels of the blocks surrounding the block to be processed; determining whether the maximum value of the infrared brightness obtained in the step of acquiring the maximum value is a saturation threshold; and correcting at least one of the infrared brightness value of the pixels of the infrared image and the distance measurement value of the pixels of the distance image based on the infrared brightness value of the pixels of the infrared image and the determination result of the step of determining whether it is a saturation threshold.

[0009] To solve the above-mentioned problems and achieve the objective, the program according to this disclosure causes a computer to perform the following steps: acquire a distance image and an infrared image obtained by receiving reflected light reflected by an object from emitted light; divide the infrared image into a plurality of blocks and acquire the maximum value of the infrared brightness from the pixels of the block to be processed and the pixels of the blocks surrounding the block to be processed; determine whether the maximum value of the infrared brightness obtained in the step of acquiring the maximum value is a saturation threshold; and correct at least one of the infrared brightness values ​​of the pixels of the infrared image and the distance measurement values ​​of the pixels of the distance image based on the infrared brightness values ​​of the pixels of the infrared image and the determination result of the step of determining whether it is a saturation threshold.

[0010] According to this disclosure, it is possible to provide a distance image generation device, a distance image generation method, and a program that can obtain accurate distance images even when there is optical diffuse reflection.

[0011] Figure 1 is a schematic diagram illustrating the outline of the distance image generation device according to this disclosure. Figure 2 is a diagram showing an example of the configuration of the distance image generation device according to this disclosure. Figure 3 is a diagram showing an example of information stored in the infrared image storage unit of the distance image generation device according to this disclosure. Figure 4 is a diagram showing an example of information stored in the distance image storage unit of the distance image generation device according to this disclosure. Figure 5 is a diagram showing an example of the distribution in the X direction of infrared brightness values ​​of an infrared image acquired by the acquisition unit of the distance image generation device according to this disclosure. Figure 6 is a diagram showing an example of a method for dividing an infrared image into multiple blocks by the maximum value detection unit of the distance image generation device according to this disclosure. Figure 7 is a diagram showing an example of a method for acquiring the maximum value between adjacent blocks of an infrared image by the maximum value detection unit of the distance image generation device according to this disclosure. Figure 8 is a diagram showing an example of a filter used in the processing of the filter processing unit of the distance image generation device according to this disclosure. Figure 9 is a diagram showing an example of the distribution in the X direction of infrared brightness values ​​and dynamic thresholds of an infrared image after filtering by the filter processing unit of the distance image generation device according to this disclosure. Figure 10 is a diagram showing an example of the distribution in the X direction of infrared brightness values ​​of an infrared image after correction processing by the correction unit of the distance image generation device according to this disclosure. Figure 11 shows both the infrared image acquired by the distance image generation device according to this disclosure and the processed infrared image. Figure 12 shows both the distance image acquired by the distance image generation device according to this disclosure and the processed distance image. Figure 13 is a flowchart of the first flow of the distance image generation method according to this disclosure. Figure 14 is a flowchart of the second flow of the distance image generation method according to this disclosure. Figure 15 shows an example of the configuration of the second embodiment of the distance image generation device according to this disclosure. Figure 16 shows an example of the number of dots, output power, and method of dividing the distance image blocks in the long-range distance measurement mode of the light-emitting unit of the second embodiment of the distance image generation device according to this disclosure. Figure 17 shows an example of the number of dots, output power, and method of dividing the distance image blocks in the short-range distance measurement mode of the light-emitting unit of the second embodiment of the distance image generation device according to this disclosure.

[0012] Embodiments of this disclosure will be described in detail below with reference to the drawings. However, the embodiments described below will not limit this disclosure.

[0013] (Overview of the Distance Image Generating Device) First, an overview of the distance image generating device 100 according to this disclosure will be described using Figure 1. Figure 1 is a schematic diagram illustrating the overview of the distance image generating device 100 according to this disclosure. As shown in Figure 1, the distance image generating device 100 includes a prism 10, a lens 20, a control unit 130, a light-emitting unit 140, and light-receiving units 150A and 150B. The distance image generating device 100 also includes other components, but these components will be briefly explained here, and then other components and detailed explanations of these components will be given later.

[0014] As shown in Figure 1, the distance image generation device 100 for measuring the distance to a subject emits near-infrared light LD1 from the light-emitting unit 140, collects the reflected near-infrared light LD2 reflected from the subject A to be measured by the lens 20, and receives it with the light-receiving unit 150B to obtain a distance image (also called a depth image or DEPTH image). The light-emitting unit 140 may emit light continuously over time, or for example, may emit light in a plurality of individual pulses.

[0015] The control unit 130 is a controller that manages and controls the distance image generation device 100. The control unit 130 is implemented by a CPU (Central Processing Unit) or an MPU (Micro Processing Unit), which executes various programs stored in the memory unit 120 using RAM as the working area. Alternatively, the control unit 130 may be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0016] The light-emitting unit 140 emits light toward a subject or the like. The light-emitting unit 140 may emit light in any multiple directions, such as horizontally or vertically.

[0017] The light-emitting unit 140 may be equipped with a dot-type VCSEL (VCSEL: Vertical Cavity Surface Emitting Laser). The VCSEL uses a highly reflective distributed Bragg reflector formed by a semiconductor or dielectric laminated structure in the reflector, thereby resonating the light perpendicular to the substrate surface and emitting the light perpendicular to the substrate surface.

[0018] The light-emitting unit 140 may emit multi-spot laser light from a dot-projection type VCSEL. In the case of multi-spot light, it is less affected by the diffuse reflection of light called scattering. Therefore, distance can be measured with high accuracy. In the case of a dot-projection type VCSEL, the distance can be measured with high accuracy by separating and extracting the distance measurement values ​​of the centroid position of each dot of the reflected light received by the light-receiving unit 150 using a threshold.

[0019] The light-receiving unit 150 emits light at the light-emitting unit 140, captures reflected light from the subject with a lens, and receives the focused reflected light. The light-receiving unit 150 may be a Time of Flight (ToF) sensor. The ToF sensor comprises an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) image sensor equipped with a photodiode, and a distance calculation unit that calculates the distance to the subject based on the reception of reflected light from the image sensor. For example, in the case of the iToF distance measurement method, the distance calculation unit calculates the distance to the subject using the following equation (1) based on the near-infrared light received by the image sensor and the timing of the shutter pulse, and outputs a depth image. The light-receiving unit 150 also outputs an infrared image (IR image) obtained at the same time. Here, D represents the distance to the subject, C represents the speed of light, Tp represents the pulse width of infrared light, and α represents the ratio due to the timing of the sensor's exposure.

[0020]

[0021] If the received light is strong and the photodiode's light reception capacity is saturated, the light receiving unit 150 outputs the infrared luminance value (IR value) at its maximum value (for example, 4095 in the case of 12 bits) and the distance value (DEPTH value) at its maximum value (for example, 65535 in the case of 16 bits). In other cases, if there are many subjects and infrared light cannot reach the pixels, making distance measurement impossible, the light receiving unit 150 outputs the infrared luminance value (IR value) as 0 and the distance value (DEPTH value) at its maximum value (for example, 65535 in the case of 16 bits).

[0022] The input unit 160 receives various types of information from the user. For example, the input unit 160 may be implemented using various switches, a keyboard, a mouse, etc., and receive various types of information from the user. Alternatively, the input unit 160 may be implemented using a touch panel, and receive various types of information from the user via the display surface.

[0023] The display unit 170 displays various information to the user. For example, the display unit 170 displays an infrared image and a distance image. The display unit 170 may also display, for example, a GUI (Graphical User Interface) for receiving various information from the user, or the results of various processes. The display unit 170 may be implemented by, for example, a liquid crystal display, an organic EL (Electroluminescence) display, a microLED (Light Emitting Diode) display, etc. The display unit 170 may also be a touch panel of various types, such as a capacitive touch panel.

[0024] The distance image generation device 100 may also obtain an RGB image in addition to a distance image. In that case, as shown in Figure 1, the prism 10 separates the reflected light into near-infrared light and visible light. The near-infrared light is received by the light receiving unit 150B to capture a distance image, and the visible light is received by the light receiving unit 150A to obtain an RGB image.

[0025] As described above, the distance image generation device 100 has industrial applicability for purposes such as recognizing the surrounding environment in autonomous vehicles, autonomous taxis, and autonomous trucks, which are expected to be realized as next-generation mobility. It may also be used in industrial applications such as controlling the amount of movement of actuators in machines equipped with actuators, counting the number of people in a given space, and identifying the type of product, measuring dimensions, and volume in factories that produce various products.

[0026] (First Embodiment) (Configuration of Distance Image Generating Device) Next, the configuration of the distance image generating device 100 according to the present disclosure will be described with reference to Figure 2. Figure 2 is a diagram showing an example of the configuration of the distance image generating device 100 according to the present disclosure. As shown in Figure 2, the distance image generating device 100 according to the present disclosure comprises a communication unit 110, a storage unit 120, a control unit 130, a light-emitting unit 140, a light-receiving unit 150, an input unit 160, and a display unit 170. These configurations will be described in order below.

[0027] The communication unit 110 connects the inside and outside of the distance image generation device 100 so that they can communicate with each other, and transmits and receives information between the inside and outside of the distance image generation device 100. The communication unit 110 may be implemented, for example, by a wireless LAN (Local Area Network) card, a Bluetooth® module, a Wi-Fi® module, an antenna, etc., when wireless communication is used. Alternatively, the communication unit 110 may be implemented by, for example, an Ethernet® interface device as defined in IEEE 802.3, or a serial communication interface device, etc., when wired communication is used.

[0028] The memory unit 120 is a storage device that stores various types of information. The memory unit 120 comprises a main memory and an auxiliary storage device. The main memory may be implemented using semiconductor memory elements such as RAM (Random Access Memory), ROM (Read Only Memory), or flash memory. The auxiliary storage device may be implemented using a hard disk or an SSD (Solid State Drive), for example.

[0029] As shown in Figure 2, the storage unit 120 includes an infrared image storage unit 121 and a distance image storage unit 122. These configurations will be described below.

[0030] The infrared image storage unit 121 stores information related to infrared images (also known as IR images). Here, an example of the information stored in the infrared image storage unit 121 will be explained using Figure 3. Figure 3 is a diagram showing an example of information stored in the infrared image storage unit 121 of the distance image generation device 100 according to this disclosure.

[0031] The infrared image storage unit 121 stores information related to items such as "infrared image ID," "acquisition date and time," "infrared image coordinates," and "infrared brightness value," linked together, as shown in Figure 3.

[0032] The "Infrared Image ID" is an identifier that identifies an infrared image and is represented by a string of characters or a number. The "Acquisition Date and Time" is information indicating the date and time the infrared image identified by the "Infrared Image ID" was acquired. The "Infrared Image Coordinates" is information indicating the coordinates of the infrared image identified by the "Infrared Image ID" and may store, for example, the coordinates in the X direction and the coordinates in the Y direction side by side. The "Infrared Brightness Value" is information indicating the infrared brightness at the "Infrared Image Coordinates" of the infrared image identified by the "Infrared Image ID" and may be, for example, a code value expressed in units of [CV].

[0033] In other words, Figure 3 shows an example where the acquisition date and time of the infrared image identified by the infrared image ID "IRIMG#1" is "TIME#1", and the infrared brightness value at the infrared image coordinates "(X#1-1, Y#1-1)" of the said infrared image is stored as "IR#1-1".

[0034] Furthermore, the information stored in the infrared image storage unit 121 is not limited to information relating to the items "infrared image ID," "acquisition date and time," "infrared image coordinates," and "infrared brightness value," but may also store any other information related to infrared images.

[0035] The distance image storage unit 122 stores information related to the distance image. Here, an example of the information stored in the distance image storage unit 122 will be explained using Figure 4. Figure 4 is a diagram showing an example of the information stored in the distance image storage unit 122 of the distance image generation device 100 according to this disclosure.

[0036] The distance image storage unit 122 stores information related to items such as "distance image ID," "acquisition date and time," "distance image coordinates," and "distance value," linked together, as shown in Figure 4, for example.

[0037] The "Distance Image ID" is an identifier that identifies the distance image, and can be represented by a string of characters or a number, for example. The "Acquisition Date and Time" is information indicating the date and time the distance image identified by the "Distance Image ID" was acquired. The "Distance Image Coordinates" is information indicating the coordinates of the distance image identified by the "Distance Image ID," and can be stored side by side, for example, the coordinates in the X direction and the coordinates in the Y direction. The "Distance Value" is information indicating the distance to the subject at the "Distance Image Coordinates" of the distance image identified by the "Distance Image ID," and can be a numerical value expressed in units of [m], for example. In other words, the "Distance Value" is the "Distance Measurement Value."

[0038] In other words, Figure 4 shows an example where the acquisition date and time of the distance image identified by the distance image ID "DPIMG#1" is "TIME#1", and the distance value at the distance image coordinates "(X#1-1, Y#1-1)" of the said distance image is stored as "DP#1-1".

[0039] Furthermore, the information stored in the distance image storage unit 122 is not limited to information relating to the items "distance image ID," "acquisition date and time," "distance image coordinates," and "distance value," but may also store any other information related to the distance image.

[0040] As shown in Figure 2, the control unit 130 includes an acquisition unit 131, a maximum value detection unit 1321, a filter processing unit 1322, a saturation determination unit 133, a dynamic threshold calculation unit 134, a centroid extraction unit 135, a correction unit 136, and an output control unit 137. The control unit 130 implements these functions by reading and executing programs (software) from the storage unit 120 to implement the functions of each processing unit (acquisition unit 131, maximum value detection unit 1321, filter processing unit 1322, saturation determination unit 133, dynamic threshold calculation unit 134, centroid extraction unit 135, correction unit 136, and output control unit 137) that the control unit 130 is equipped with. Note that these functions of the control unit 130 may be implemented by electronic circuits. Furthermore, the control unit 130 may execute these processes with a single CPU, or it may be equipped with multiple CPUs and execute these processes in parallel with multiple CPUs.

[0041] Here, the processing flow in the control unit 130 will be explained. As shown in Figure 2, after the acquisition unit 131 has processed, the processing of the maximum value detection unit 1321 and the filter processing unit 1322 may be performed simultaneously in parallel. After the processing of the maximum value detection unit 1321, the saturation determination unit 133 is processed. In the saturation determination unit 133, if a predetermined condition is met, the dynamic threshold calculation unit 134 is processed; if the predetermined condition is not met, the centroid extraction unit 135 is processed. On the other hand, after the processing of the filter processing unit 1322, the correction unit 136 is processed. Then, in the output control unit 137, the infrared image and distance image after processing by the centroid extraction unit 135 are combined with the infrared image and distance image after processing by the correction unit 136 to obtain the processed infrared image and distance image. The processed infrared image and distance image are then output by the output control unit 137 to the display unit 170 or to external devices and equipment. The processing of these configurations will be explained in detail below.

[0042] The acquiring unit 131 acquires various types of information. The acquiring unit 131 acquires various types of information from external devices or equipment of the distance image generating apparatus 100, or internal devices or equipment of the distance image generating apparatus 100, the input unit 160, and the like. For example, the acquiring unit 131 acquires an infrared image captured by a light receiving unit 150, which will be described later, and a distance image. After acquiring the infrared image and the distance image, the acquiring unit 131 stores the acquired infrared image and the distance image in the infrared image storage unit 121 and the distance image storage unit 122, respectively. In addition, the acquiring unit 131 acquires information related to an operation mode input by a user from the input unit 160.

[0043] Here, the distribution in the X direction of infrared luminance values of the infrared image acquired by the acquiring unit 131 will be described with reference to FIG. 5. FIG. 5 is a diagram illustrating an example of the distribution in the X direction of infrared luminance values of an infrared image acquired by the acquiring unit 131 of the distance image generating apparatus 100 according to the present disclosure. In FIG. 5, the horizontal axis represents the position of a pixel in the X direction of the infrared image, and the vertical axis represents the infrared luminance value at a certain Y coordinate of the infrared image. In the infrared image shown in FIG. 5, since there are a plurality of portions where the infrared luminance value is a saturation value, it can be seen that there are a plurality of locations where the amount of received light is saturated. The processing described below is performed on such an infrared image.

[0044] The maximum value detecting unit 1321 divides the infrared image acquired by the acquiring unit 131 described above into a plurality of blocks, and acquires the maximum value of infrared luminance values from pixels of a block to be processed and pixels of blocks surrounding the block to be processed. Specifically, the maximum value detecting unit 1321 reads an infrared image from the infrared image storage unit 121, divides the infrared image into a plurality of blocks, acquires the maximum value of infrared luminance values for each block, further performs comparison between adjacent blocks, and acquires the maximum value of infrared luminance values including blocks neighboring the block to be processed.

[0045] Here, a method of dividing an infrared image into a plurality of blocks by the maximum value detection unit 1321 will be described with reference to FIG. 6. FIG. 6 is a diagram illustrating an example of a method of dividing an infrared image into a plurality of blocks by the maximum value detection unit 1321 of the distance image generating apparatus 100 according to the present disclosure. FIG. 6 shows, as an example, an example in which a 640×480 pixel infrared image is divided into a plurality of blocks by unit blocks each having a block size of 6×6. In FIG. 6, it is assumed that the horizontal direction corresponds to the X direction and the vertical direction corresponds to the Y direction.

[0046] As described above, the maximum value detection unit 1321 may divide, for example, a 640×480 pixel infrared image into a plurality of blocks by unit blocks each having a block size of 6×6. The method of dividing an infrared image into a plurality of blocks by the maximum value detection unit 1321 is not limited to such a division method, and may be divided by any division method. Further, when the infrared image cannot be divided exactly by the block size, for example, a small block may be set for division so that no portion is left unprocessed as shown at the right end of FIG. 6.

[0047] Here, when a 640×480 pixel infrared image is divided into a plurality of blocks with a block size of 6×6, the number of blocks in the X direction BLOCK_NUM_X and the number of blocks in the Y direction BLOCK_NUM_Y are respectively the numbers represented by the following formulas (2) and (3).

[0048]

[0049] In the example shown in FIG. 6, an example in which a small block is arranged at the right end as an example where blocks cannot be divided exactly in the X direction has been described; however, the small block may be arranged at any position such as the left end or the center, for example. Further, although an example in which a small block is used when division is not exact has been described, a division method in which only the last block is set to 10 pixels and the number of blocks is set to 106 may also be employed. Then, the maximum value detection unit 1321 acquires the maximum value of infrared luminance values for each of the blocks obtained by dividing the infrared image.

[0050] Next, Figure 7 illustrates the method for obtaining the maximum value between adjacent blocks in the maximum value detection unit 1321. Figure 7 shows an example of a method for obtaining the maximum value between adjacent blocks of an infrared image by the maximum value detection unit 1321 of the distance image generation device 100 according to this disclosure. In Figure 7, the horizontal direction corresponds to the X direction, and the vertical direction corresponds to the Y direction. As shown in Figure 7, the maximum value detection unit 1321 compares the maximum value of the infrared brightness of the block of interest (the block to be processed) with the maximum values ​​of the eight surrounding blocks in the up, down, left, right, and diagonal directions. In Figure 7, the X and Y coordinates are (1,1), and the block of interest is (0,0), (0,1), (0,2), (1,0), (1,2), (2,0), (2,1), and (2,2) surrounding it. In all of these blocks, the maximum value of the infrared brightness detected by the maximum value detection unit 1321 is taken as the maximum value of the infrared brightness of the block of interest (the block to be processed).

[0051] If the block of interest (the block to be processed) is at the edge of the infrared image, it will be compared with adjacent blocks. For example, if the block of interest (the block to be processed) is (0,0), it will be compared with the three blocks (0,1), (1,0), and (1,1). Also, if the block of interest (the block to be processed) is (0,1), it will be compared with the five blocks (0,0), (0,2), (1,0), (1,1), and (1,2).

[0052] As explained above, if the maximum value is detected only once for each block of the infrared image, the maximum value of the dot can be correctly obtained when the reflected light dot is within the block, and therefore the output can be correct even when the light received by the photodiode of the light receiving unit 150 is saturated. However, the area where the light received by the photodiode of the light receiving unit 150 is saturated tends to occur near the center of the reflected light dot, but when the reflected light dot crosses between blocks, the maximum value is taken for each of the dots divided in each block, so the maximum value is obtained separately for each block, and in some blocks, an unsaturated part may be considered as the maximum value. As a result, there may be blocks where a dot that is actually saturated is treated as an unsaturated dot.

[0053] Therefore, in such cases, the maximum value detection unit 1321 detects the maximum value from an even more adjacent block. In other words, if the reflected light dots traverse between blocks, the maximum value detection unit 1321 detects the maximum value from an adjacent block to the block of interest. As a result, if there is a dot within a block where the photodiode's light reception is saturated, the maximum value of the dot can be correctly detected in the same way, and thus the output can be correct even when the photodiode's light reception is saturated.

[0054] Furthermore, the maximum value detection unit 1321 also acquires the maximum value of adjacent blocks even when the reflected light dots traverse between blocks. This makes it possible to correctly extract saturated dots even when the reflected light dots traverse between blocks.

[0055] Meanwhile, the filter processing unit 1322 performs filtering on the infrared image acquired by the acquisition unit 131. The kernel size and kernel coefficients of the filter used in the filter processing unit 1322 may be any values. As an example, Figure 8 shows the kernel coefficients of a 5x5 filter used for filtering in the filter processing unit 1322. Figure 8 is a diagram showing an example of a filter used for processing in the filter processing unit 1322 of the distance image generation device 100 according to this disclosure. As shown in Figure 8, a kernel coefficient of 0.1 may be placed in the central part of the filter, and kernel coefficients such as 0, -0.1, and -0.125 may be placed in the peripheral part of the filter. The kernel size and kernel coefficients of the filter used for filtering in the filter processing unit 1322 are not limited to those described in Figure 8, and may be any values.

[0056] The filter processing unit 1322 performs filtering on the infrared image using the filter described above. Specifically, the filter processing unit 1322 performs a sum-of-products operation on all pixels of the infrared image, that is, it multiplies the kernel coefficient by the infrared brightness value of the pixels in the infrared image and sums the calculated value for the pixel of interest and its surrounding pixels, and uses this sum as the infrared brightness value of the filtered infrared image. This makes the infrared brightness value of the infrared image at the centroid position of the reflected light dot stand out compared to the surrounding pixels.

[0057] The saturation determination unit 133 determines whether each block is saturated based on the maximum value detected by the maximum value detection unit 1321. Specifically, the saturation determination unit 133 determines that a block is saturated if it satisfies the following equation (4), where BX is the position of the block in the X direction of the infrared image, BY is the position of the block in the Y direction of the infrared image, and IR_BLK(BY, BX) is the result of detecting the maximum value of the block at the BY, BX positions.

[0058]

[0059] As mentioned earlier, equation (4) above assumes that the infrared image is 12 bits and that when saturated, it reaches the maximum value of 4095. In cases other than this assumption, the maximum value of the infrared brightness used for saturation determination can be arbitrarily set. The saturation determination unit 133 considers that a block is not saturated if it does not satisfy equation (4) above.

[0060] The dynamic threshold calculation unit 134 calculates a dynamic threshold, which is a threshold set for each pixel position in the infrared image. For example, the dynamic threshold calculation unit 134 calculates the dynamic threshold DTHR(y,x) using the following formula (5) based on the infrared brightness value IMG_IR(y,x) of the infrared image recorded in the infrared image storage unit 121 and the saturation threshold SAT_THR.

[0061]

[0062] Here, IMG_IR(y,x) is the infrared brightness value of the infrared image recorded in the infrared image storage unit 121. The saturation threshold SAT_THR is the infrared brightness value at saturation, and may be 4095, for example, if the infrared image is 12 bits.

[0063] The dynamic threshold calculation unit 134 is not limited to calculating the dynamic threshold using the method described above with formula (5). It may also calculate a dynamic threshold set for each pixel position in the infrared image based on the pixel value IMG_IR(y,x) of the infrared image and the saturation threshold SAT_THR using any other method.

[0064] Next, the infrared luminance values ​​after filtering and the distribution of the dynamic threshold in the X direction of the distance image generation device 100 according to this disclosure will be explained with reference to Figure 9. Figure 9 is a diagram showing an example of the distribution of the infrared luminance values ​​and the dynamic threshold in the X direction of the infrared image after filtering by the filter processing unit 1322 of the distance image generation device 100 according to this disclosure. As shown in Figure 9, the dynamic threshold calculated by the dynamic threshold calculation unit 134 using the infrared image shown in Figure 5 as input changes according to the position of the pixels in the X direction of the infrared image. Also, as shown in Figure 9, the infrared luminance values ​​of the infrared image after filtering show that the peaks of the reflected light dots are prominent. By using such a dynamic threshold, when the amount of light received by the photodiode is saturated, it is possible to extract only the vicinity of the dot saturation.

[0065] The centroid extraction unit 135 extracts the centroid position of the dots in the infrared image. Specifically, if the saturation determination unit 133 described above determines that the infrared brightness value of the pixels in the block to be processed is not saturated, the centroid extraction unit 135 extracts the centroid position of the dots using the infrared brightness value of the pixels in the block determined to be unsaturated and the distance measurement value of the pixels in the distance image corresponding to the pixels in the block determined to be unsaturated. For example, the centroid extraction unit 135 uses a centroid extraction threshold THR to extract the centroid position of the dots, considering the infrared brightness value exceeding the centroid extraction threshold THR as the centroid position of the dot. That is, the centroid extraction unit 135 extracts only the distance measurement value of the distance image corresponding to the position of the centroid position of each dot in the infrared image, and sets the distance measurement values ​​of the distance image other than the centroid position of the dots to invalid values, for example, (65535).

[0066] Furthermore, the centroid extraction unit 135 sets the infrared brightness value of the infrared image outside the centroid position of the dot to an invalid value of 0. The reason for extracting only the centroid portion is that, in the dot-illuminated image, the accuracy of distance measurement is good at the centroid of the dot, but the accuracy of distance measurement decreases as you move away from the centroid of the dot. This makes it possible to retain only the distance measurement values ​​with good measurement accuracy in the distance image.

[0067] As explained above, if the saturation determination unit 133 determines that a block is not saturated, the centroid extraction unit 135 extracts only the infrared brightness value of the infrared image and the distance measurement value of the depth image at the centroid position of the dot. If the block is saturated, the correction unit 136, which will be explained later, extracts only the infrared brightness value of the infrared image and the distance measurement value of the peripheral portion of the saturated dot. In all other cases, both the infrared brightness value of the infrared image and the distance measurement value of the depth image are treated as invalid values.

[0068] The correction unit 136 corrects the infrared brightness values ​​of the pixels in the infrared image to be processed and the distance measurements of the pixels in the depth image corresponding to the pixels in the infrared image to be processed, based on the infrared brightness values ​​of the infrared image after filtering and the dynamic threshold calculated by the dynamic threshold calculation unit 134. For example, the correction unit 136 compares the infrared brightness values ​​of the filtered infrared image using the following equations (6) and (7) to determine whether the infrared brightness value F(y,x) of the filtered infrared image is greater than or equal to the dynamic threshold DTHR(y,x) and whether the infrared brightness value IMG_IR(y,x) of the infrared image is at a saturation value.

[0069]

[0070] If the above conditions are met, the correction unit 136 determines that the pixel is a pixel in the peripheral area of ​​a saturated dot and extracts the infrared brightness value of that pixel. If the above conditions are not met, the correction unit 136 determines that the pixel is saturated or too far from the peripheral area of ​​a saturated dot and sets the distance value to an invalid value, for example, 65535. If the above conditions are not met, the correction unit 136 sets the infrared brightness value of the infrared image to 0 and makes it an invalid value. The reason for extracting only the peripheral area of ​​a saturated dot is that the amount of light received at the centroid of the dot becomes saturated, making distance measurement impossible, or even if distance measurement is possible, the accuracy of the distance measurement becomes significantly worse.

[0071] The depth image generation process described above is repeated for all pixels of the infrared image, thereby correcting the infrared image and the depth image corresponding to the position where the maximum value of all pixels in the infrared image is saturated. Once processing is complete for all pixels of the infrared image, the infrared image and depth image output by the centroid extraction unit 135 and the infrared image and depth image processed by the correction unit 136 are combined, and the output control unit 137, which will be described later, outputs the processed infrared image and depth image.

[0072] Next, an example of the distribution of infrared brightness values ​​in the X direction of an infrared image after correction processing by the correction unit 136 of the distance image generation device 100 according to this disclosure will be explained using Figure 10. Figure 10 is a diagram showing an example of the distribution of infrared brightness values ​​in the X direction of an infrared image after correction processing by the correction unit 136 of the distance image generation device 100 according to this disclosure. Similar to Figure 5, the horizontal axis shows the position of the pixels in the X direction of the infrared image, and the vertical axis shows the infrared brightness value at the position of the pixel in a certain Y coordinate of the infrared image. Comparing the infrared brightness values ​​of the infrared image before processing shown in Figure 5, as described above, with the infrared brightness values ​​of the infrared image after processing shown in Figure 10, it can be seen that by applying the correction processing by the correction unit 136 of the distance image generation device 100 according to this disclosure, only the area around saturated pixels is extracted.

[0073] The output control unit 137 integrates the infrared image and distance image output by the centroid extraction unit 135 with the infrared image and distance image output by the correction unit 136, respectively, and outputs them. The output control unit 137 controls the output of the integrated distance image and infrared image to an external device or equipment, or to the display unit 170. The output control unit 137 may output the integrated distance image and infrared image by displaying them on the display unit 180, for example. Alternatively, the output control unit 137 may output the data of the integrated distance image and infrared image to an external device or equipment connected via the communication unit 110.

[0074] Here, the distance image and infrared image output by the output control unit 137 before and after processing will be explained using Figures 11 and 12. Figure 11 is a diagram showing both the infrared image acquired by the distance image generation device 100 according to this disclosure and the infrared image after processing. Figure 12 is a diagram showing both the distance image acquired by the distance image generation device 100 according to this disclosure and the distance image after processing. In Figures 11 and 12, an infrared image captured by the light receiving unit 150 is shown in a situation where a retroreflective plate RB is placed at a distance of 300 mm and a white board DB is placed at a distance of 1100 mm. The distance measurement value of the light receiving unit 150 is 16 bits, and for example, if it is 300 mm, it will be output as 300, and the distance measurement value will be output as is. The infrared image IRIMG1 shown on the left side of Figure 11 is used as the input image, and the output image of the infrared image IRIMG2 after applying this disclosure is shown on the right side of Figure 11.

[0075] As shown on the left side of Figure 11, the infrared image IRIMG1 of the input image is saturated because the light emitted from the light emitter 140 is strongly reflected by the retroreflective plate RB. Specifically, in Figure 11, the white areas of the infrared image IRIMG1 of the input image are saturated. In the saturated infrared image IRIMG1 of the input image, some of the dots become invalid values ​​and are displayed in white.

[0076] On the other hand, as shown in Figure 12, in the corresponding distance image DPIMG1, distance data is obtained in the peripheral areas of the saturated dots, and these areas are displayed darkly. Thus, in the distance image DPIMG1 before the application of the distance image generation process corresponding to the saturated dots in the infrared image IRIMG1 before processing, there is inaccurate distance data that is far from the centroid of the reflected light dots.

[0077] In contrast, as shown on the right side of Figure 12, the processed distance image DPIMG2 extracts the areas around saturated dots and the centroids of unsaturated dots in the output infrared image IRIMG2, and the peaks of the distance values ​​are sharpened. As shown on the right side of Figure 12, in the whiteboard DB region, the areas corresponding to the centroids of the dots in the infrared image are extracted, and the size of the dots in the processed distance image DPIMG2 becomes smaller. Also, as shown in the processed distance image DPIMG2 on the right side of Figure 12, in the saturated region of the retroreflector RB, only the saturated area is extracted, indicating that the distance of the retroreflector RB is obtained with high accuracy.

[0078] (Effects of the First Embodiment) According to the distance image generation device 100 described above, the maximum value of the infrared brightness is obtained for each block of the infrared image which is divided into multiple blocks, and further compared with adjacent blocks to obtain the maximum value of the infrared brightness including the neighboring blocks of the block to be processed. As a result, even when a saturated dot crosses the divided blocks of the infrared image, the maximum value of the infrared brightness in the area surrounding the saturated dot can be detected.

[0079] Furthermore, in the case of blocks containing saturated dots, a dynamic threshold specifically for saturation is used to acquire only the infrared brightness value of the infrared image and the distance measurement value of the depth image for the area surrounding the saturated dots. In the case of blocks that do not contain unsaturated dots, a threshold for centroid extraction is used to acquire the infrared brightness value of the infrared image and the distance measurement value of the depth image for the centroid position of the dots. This makes it possible to acquire accurate depth images even when the dots of the light-receiving unit 150 are saturated due to optical diffuse reflection, etc. Therefore, a depth image generation device 100 can be provided that can obtain accurate depth images even when there is optical diffuse reflection.

[0080] (Distance Image Generation Method and Program) Next, a first embodiment of the distance image generation method and program according to this disclosure will be described with reference to Figure 13. Figure 13 is a flowchart of the first flow of the distance image generation method according to this disclosure. The first embodiment of the distance image generation method according to this disclosure will be described in accordance with the flow shown in Figure 13.

[0081] First, the acquisition unit 131 acquires the processed image (infrared image, depth image) captured by the light receiving unit 150 (step S100). Next, the maximum value detection unit 1321 divides the infrared image acquired by the acquisition unit 131 into unit blocks (step S101). Next, the maximum value detection unit 1321 detects the maximum value of the infrared brightness of the pixels in each unit block of the infrared image (step S102). Next, the maximum value detection unit 1321 sets the unit block to be corrected (step S103). Next, the maximum value detection unit 1321 calculates the maximum value of the infrared brightness of the pixels in the extended range, including blocks adjacent to the unit block to be corrected (step S104). Next, the saturation determination unit 133 determines whether the maximum value of the infrared brightness of the pixels in the extended range is saturated (step S105).

[0082] If the maximum value of the infrared brightness of the pixels in the extended range is saturated (step S105: Yes), the dynamic threshold calculation unit 134 calculates a dynamic threshold (step S106). Next, the correction unit 136 corrects the infrared brightness of the pixels in a unit block of the infrared image and the distance measurement value of the pixels in the distance image corresponding to the pixels in the unit block of the infrared image using the infrared brightness value of the filtered infrared image obtained by the filter processing unit 1322 on the infrared image acquired by the acquisition unit 131 and the dynamic threshold calculated by the dynamic threshold calculation unit 134 (step S107). Next, the control unit 130 determines whether the processing from steps S103 to S107 has been completed for all unit blocks (step S108). If the processing from step S103 to step S107 is completed for all unit blocks (step S108: Yes), the output control unit 137 integrates the infrared image and distance image output by the centroid extraction unit 135 with the infrared image and distance image output by the correction unit 136, respectively, and outputs them to an external device or equipment, or to the display unit 170 (step S110). After step S110, the control unit 130 terminates the processing of the distance image generation method.

[0083] In step S105, if the maximum value of the infrared brightness of the pixels in the extended range is not saturated (step S105: No), the centroid extraction unit 135 extracts the centroid of the dots in step S109 using the infrared brightness value of the pixels in the unit block that was determined not to be saturated and the distance measurement value of the pixels in the depth image corresponding to the pixels in the unit block that was determined not to be saturated (step S109). After the processing in step 109, the control unit 130 proceeds to step S108 and executes the subsequent processing. Also, in step S108, if the processing from steps S103 to S107 has not been completed for all unit blocks (step S108: No), the control unit 130 returns to step S103 and executes the subsequent processing.

[0084] According to the first embodiment of the distance image generation method described above, the maximum value of the infrared brightness is detected for each block of the infrared image, the infrared brightness value of the saturated block is corrected, and the distance measurement value of the distance image corresponding to that block of the infrared image is corrected. This provides a distance image generation method that can obtain an accurate distance image even when there is optical diffuse reflection.

[0085] Next, a second embodiment of the distance image generation method and program according to this disclosure will be described with reference to Figure 14. Figure 14 is a flowchart of the second flow of the distance image generation method according to this disclosure. The second embodiment of the distance image generation method according to this disclosure will be described in accordance with the flow shown in Figure 14.

[0086] The process in step S200 is the same as the process in step S100 shown in Figure 13, so the explanation will be omitted.

[0087] The filter processing unit 1322 performs a sum-of-products operation on the infrared brightness values ​​of the pixels in the infrared image acquired by the acquisition unit 131, and generates and stores the filtered infrared image (step S201).

[0088] The processes from step S202 to step S206 are the same as the processes from step S101 to step S105 shown in Figure 13, so their explanation will be omitted.

[0089] If the maximum brightness value of pixels in the extended range is saturated (step S206: Yes), the dynamic threshold calculation unit 134 sets the unit pixel to be extracted (step S207). Next, the dynamic threshold calculation unit 134 calculates a dynamic threshold from the brightness of the unit pixel (step S208). Next, the correction unit 136 determines whether the brightness of the corresponding pixel in the filtered image is at the maximum value or less than the dynamic threshold (step S209). The correction unit 136 refers to the filtered infrared image generated and held in step S201, and if the infrared brightness value of the corresponding pixel in the filtered infrared image is at the maximum value or less than the dynamic threshold, the unit pixel information is deleted (step S210). The control unit 130 determines whether the processing from steps S207 to S210 has been completed for all unit pixels (step S211). If the processing from steps S207 to S210 has been completed for all unit pixels (step S211: Yes), the control unit 130 proceeds to step S212. Furthermore, if the processing from step S207 to step S210 is not completed for all unit pixels (step S211: No), the control unit 130 returns to step S207 and executes the subsequent processing.

[0090] Furthermore, in step S209, if the brightness of the corresponding filtered image pixels is not less than the maximum value or dynamic threshold (step S209: No), the distance image generation device 100 proceeds to step S211 and performs the subsequent processing.

[0091] The processes from step S212 to step S214 are the same as the processes from step S108 to step S110 shown in Figure 13, so their explanation will be omitted.

[0092] According to the second embodiment of the distance image generation method described above, the maximum value is detected for each block of the infrared image, the infrared brightness value of the saturated block of the infrared image is corrected, and the distance measurement value of the distance image corresponding to that block of the infrared image is corrected. This provides a distance image generation method that can obtain an accurate distance image even when there is optical diffuse reflection.

[0093] (Second Embodiment) (Configuration of Distance Image Generating Device) Next, a second embodiment of the distance image generating device 100 according to the present disclosure will be described with reference to Figure 15. Figure 15 is a diagram showing an example of the configuration of the second embodiment of the distance image generating device 100 according to the present disclosure. As shown in Figure 15, the second embodiment of the distance image generating device 100 differs from the first embodiment of the distance image generating device 100 in that the acquisition unit 131 has a processing unit, the control unit 130 has a switching unit 1315, and the light projection unit 140 has a first light projection unit 140A and a second light projection unit 140B. In the following, the configuration of the second embodiment of the distance image generating device 100 that differs from the configuration of the first embodiment of the distance image generating device 100 will be described, and the description of other configurations will be omitted.

[0094] The acquisition unit 131 acquires information regarding the operating mode input to the input unit 160. Here, the operating mode may include, for example, a long-distance measurement mode and a short-distance measurement mode. The user selects one of these and inputs it via the input unit 160.

[0095] The switching unit 1315 switches the operating mode of the light-emitting unit 140. As shown in Figure 15, the light-emitting unit 140 includes a first light-emitting unit 140A used for measuring distance in long-range distance measurement mode and a second light-emitting unit 140B used for measuring distance in short-range distance measurement mode, and the switching unit 1315 may switch whether or not to use both. That is, the switching unit 1315 switches the operating mode of the light-emitting unit 140 based on the operating mode input by the user to the input unit 160. Accordingly, the switching unit 1315 outputs a control signal to the first light-emitting unit 140A or the second light-emitting unit 140B to output a predetermined light emission pulse, depending on whether it is in short-range distance measurement mode or long-range distance measurement mode.

[0096] In other words, when the operating mode is long-range distance measurement mode, the switching unit 1315 illuminates the long-range dot VCSEL of the first light-emitting unit 140A, and does not illuminate the short-range dot VCSEL of the second light-emitting unit 140B. Conversely, when the operating mode is short-range distance measurement mode, the short-range dot VCSEL of the second light-emitting unit 140B is illuminated, and the long-range dot VCSEL of the first light-emitting unit 140A is not illuminated. The illumination period may be adjusted to the optimal number of illumination pulses for each of the long-range distance measurement mode and short-range distance measurement mode. For example, the width of one pulse in the illumination period is switched to the optimal width for the distance measurement range for each of the long-range distance measurement mode and short-range distance measurement mode.

[0097] Then, the switched first light-emitting unit 140A or the second light-emitting unit 140B determines the shutter timing and the light emission pulse for the VCSEL that are appropriate for the operating mode and operates accordingly.

[0098] Here, an example of the dot irradiation pattern and block division method of a long-range dot-irradiation type VCSEL, which is one example of the realization of the first light-emitting unit 140A, will be explained using Figure 16. Figure 16 is a diagram showing an example of the number of dots, output power, and method of dividing the distance image blocks in the long-range distance measurement mode of the light-emitting unit of the second embodiment of the distance image generation device 100 according to this disclosure. As shown in Figure 16, the long-range dot-irradiation type VCSEL, which is one example of the realization of the first light-emitting unit 140A, may have, for example, 2028 dots and an output power of 4W. Also, as shown in Figure 15, in the long-range distance measurement mode, the infrared image may be divided into blocks of, for example, 10 × 10 pixels in size.

[0099] Here, an example of the dot irradiation pattern and block division method of a short-range dot-irradiation type VCSEL, which is one example of realizing the second light-emitting unit 140B, will be explained using Figure 17. Figure 17 is a diagram showing an example of the number of dots, output power, and method of dividing the distance image blocks in the short-range distance measurement mode of the light-emitting unit of the second embodiment of the distance image generation device 100 according to this disclosure. As shown in Figure 17, the short-range dot-irradiation type VCSEL, which is one example of realizing the second light-emitting unit 140B, may have 4800 dots, which is more than the long-range type, and an output power of 2W. Also, as shown in Figure 17, in the short-range distance measurement mode, the infrared image may be divided into blocks of, for example, 6 x 6 pixels in size.

[0100] The filter processing unit 1322 switches the filter applied to the infrared image according to the long-range measurement mode and the short-range measurement mode, and performs filtering on the infrared image. That is, the filter processing unit 1322 uses either a long-range filter or a short-range filter according to the long-range measurement mode and the short-range measurement mode, respectively. The kernel size and kernel coefficients of the long-range filter may be set arbitrarily. Similarly, the kernel size and kernel coefficients of the short-range filter may also be set arbitrarily.

[0101] In this way, the VCSEL to be used is selected according to the operating mode, and once it is emitted, the centroid and saturated peripheral areas of the dot illumination are extracted from the infrared image block in order to correct the infrared image and the distance image, similar to the first embodiment 1. The size of the frame at this time can be set more finely in the short-range measurement mode to match the dot spacing of the VCSEL; for example, it may be set to a size of 6x6 pixels.

[0102] Up to this point, we have explained the case where there are two operating modes, but it is also possible to increase the number of types of dot-illuminated VCSELs to increase the number of operating modes and change the block size accordingly to several different sizes. In this way, even if multiple types of VCSELs with different dot spacings and numbers are used depending on the operating mode, optimal correction can be obtained according to the operating mode by setting the corresponding block size.

[0103] According to the second embodiment of the distance image generation device 100 described above, when measuring long distances, distance measurement is performed using a long-distance dot-illuminating VCSEL, and when measuring short distances, distance measurement is performed using a short-distance dot-illuminating VCSEL with low light intensity. By setting the appropriate block size, even when using multiple types of VCSELs with different dot spacings and numbers, it becomes possible to obtain an infrared image and a distance image with optimal correction according to the distance measurement mode. Therefore, it is possible to provide a distance image generation device 100 that can obtain accurate distance images even when there is optical diffuse reflection.

[0104] (Configuration and Effects) The distance image generation device 100 according to the first embodiment includes: an acquisition unit 131 that acquires a distance image and an infrared image obtained by receiving reflected light reflected by a subject from emitted light; a maximum value detection unit 1321 that divides the infrared image into a plurality of blocks and acquires the maximum value of the infrared brightness from the pixels of the block to be processed and the pixels of the blocks surrounding the block to be processed; a saturation determination unit 133 that determines whether the maximum value of the infrared brightness obtained by the maximum value detection unit 1321 is a saturation threshold; and a correction unit 136 that corrects at least one of the infrared brightness value of the infrared image and the distance measurement value of the distance image based on the infrared brightness value of the infrared image and the determination result of the saturation determination unit 133.

[0105] According to this configuration, the infrared image is divided into multiple blocks, the maximum infrared brightness value is obtained from the pixels of the block to be processed and the pixels of the surrounding blocks, and it is determined whether the maximum infrared brightness value is the saturation threshold. Based on the infrared brightness value of the pixels in the infrared image and the determination result of the saturation determination unit 133, if it is determined that the maximum infrared brightness value is the saturation threshold, the distance measurement value of the pixels in the distance image corresponding to the pixels of the block to be processed can be corrected. As a result, distance measurement values ​​that have poor measurement accuracy due to saturation of the amount of received light can be corrected, and an accurate distance image can be obtained. Therefore, it is possible to provide a distance image generation device 100 that can obtain an accurate distance image even when there is optical diffuse reflection.

[0106] The distance image generation device 100 according to the second embodiment is the distance image generation device 100 according to the first embodiment, further comprising a dynamic threshold calculation unit 134 that calculates a dynamic threshold set for each pixel of the block to be processed, and the correction unit 136 invalidates the distance measurement value of the distance image corresponding to the position of the pixel when the infrared brightness value of the pixel of the block to be processed is less than the dynamic threshold.

[0107] With this configuration, if it is determined that the maximum value of the infrared brightness is less than the saturation threshold, a dynamic threshold set for each position in the infrared image is calculated, and if the infrared brightness value of a pixel in the block to be processed is less than the dynamic threshold, the distance measurement value of the distance image corresponding to the position of that pixel can be invalidated. Therefore, distance measurement values ​​with poor measurement accuracy that fall around saturated dots can be invalidated. Thus, a distance image generation device 100 can be provided that can obtain an accurate distance image even when there is optical diffuse reflection.

[0108] The distance image generation device 100 according to the third embodiment is the distance image generation device 100 according to the first or second embodiment, further comprising a filter processing unit 1322 that performs a filter processing on an infrared image with predetermined kernel coefficients, wherein the correction unit 136 invalidates the distance measurement value of the distance image corresponding to the position of the pixel if the infrared brightness value of the pixel of the processing target block of the infrared image after the filter processing on the infrared image is less than a dynamic threshold.

[0109] This configuration allows for the extraction of the area surrounding a saturated dot that is closest to the saturated dot by applying a filter to the infrared image. Furthermore, if the infrared brightness value of the filtered infrared image falls below a dynamic threshold, the distance value of the distance image corresponding to the pixel's position can be invalidated. Therefore, distance measurements with poor accuracy that fall around saturated dots can be invalidated. Consequently, a distance image generation device 100 can be provided that can obtain accurate distance images even in the presence of optical diffuse reflection.

[0110] The distance image generation device 100 according to the fourth embodiment is a distance image generation device 100 according to any one of the first to third embodiments, wherein the acquisition unit 131 acquires information regarding the operating mode and further comprises a switching unit 1315 that switches the light emission unit 140 that emits light used for capturing infrared images and distance images based on the operating mode, and the maximum value detection unit 1321 changes the size of the unit block when dividing the infrared image into a plurality of blocks based on the operating mode.

[0111] With this configuration, the light-emitting unit 140 that emits light used for capturing infrared and depth images can be switched based on the operating mode, and the size of the unit block when dividing the infrared image into multiple blocks can be changed based on the operating mode. Therefore, even if multiple types of light-emitting units 140 with different dot spacings and dot numbers are switched and used, by setting the corresponding unit block size, it is possible to obtain infrared and depth images with optimal correction according to the operating mode. Thus, it is possible to provide a depth image generation device 100 that can obtain accurate depth images even when there is optical diffuse reflection.

[0112] A distance image generation method according to the fifth embodiment includes the steps of: acquiring a distance image and an infrared image obtained by receiving reflected light reflected by a subject from emitted light; dividing the infrared image into a plurality of blocks and acquiring the maximum value of the infrared brightness from the pixels of the block to be processed and the pixels of the blocks surrounding the block to be processed; determining whether the maximum value of the infrared brightness obtained in the step of acquiring the maximum value is a saturation threshold; and correcting at least one of the infrared brightness value of the pixels in the infrared image and the distance measurement value of the pixels in the distance image based on the infrared brightness value of the pixels in the infrared image and the determination result of the step of determining whether it is a saturation threshold.

[0113] With this configuration, the infrared image is divided into multiple blocks, the maximum infrared brightness value is obtained from the pixels of the block to be processed and the pixels of the surrounding blocks, and it is determined whether the maximum infrared brightness value is a saturation threshold. Based on the infrared brightness value of the pixels in the infrared image and the determination result of the step to determine whether it is a saturation threshold, at least one of the infrared brightness value of the pixels in the infrared image or the distance measurement value of the pixels in the distance image can be corrected. As a result, distance measurement values ​​that have poor accuracy due to saturation of the amount of received light can be corrected, and an accurate distance image can be obtained. Therefore, a distance image generating device 100 can be provided that can obtain an accurate distance image even when there is optical diffuse reflection.

[0114] The program according to the sixth embodiment includes the steps of: acquiring a distance image and an infrared image obtained by receiving reflected light reflected by a subject from emitted light; dividing the infrared image into a plurality of blocks and acquiring the maximum value of the infrared brightness from the pixels of the block to be processed and the pixels of the blocks surrounding the block to be processed; determining whether the maximum value of the infrared brightness obtained in the step of acquiring the maximum value is a saturation threshold; and correcting at least one of the infrared brightness value of the pixels in the infrared image and the distance measurement value of the pixels in the distance image based on the infrared brightness value of the pixels in the infrared image and the determination result of the step of determining whether it is a saturation threshold, wherein in the step of determining the saturation threshold, if it is determined that the maximum value of the infrared brightness obtained in the step of acquiring the maximum value is a saturation threshold, the program causes the computer to correct the distance measurement value of the pixels in the distance image corresponding to the block to be processed.

[0115] This configuration allows for the division of an infrared image into multiple blocks, the acquisition of the maximum infrared brightness value from the pixels of the block to be processed and the pixels of the surrounding blocks, a determination of whether the maximum infrared brightness value is a saturation threshold, and correction of at least one of the infrared brightness value of the pixels in the infrared image or the distance measurement value of the pixels in the depth image based on the infrared brightness value of the pixels in the infrared image and the determination result of the step in determining whether it is a saturation threshold. As a result, it is possible to correct distance measurement values ​​that are not accurate due to saturation of the amount of received light, thereby obtaining an accurate depth image. Therefore, it is possible to provide a program that can obtain an accurate depth image even when there is optical diffuse reflection.

[0116] Although embodiments of the present disclosure have been described above, the embodiments are not limited to those described herein. Furthermore, the aforementioned components include those that can be easily conceived by those skilled in the art, those that are substantially the same, and those that fall within the so-called equivalent range. Moreover, the aforementioned components can be combined as appropriate. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the gist of the embodiments described above.

[0117] The distance image generation device, distance image generation method, and program disclosed herein can be used, for example, in Time-of-Flight (Tof) distance measurement technology for measuring the distance to an object.

[0118] 10 Prism 20 Lens 100 Distance image generation device 110 Communication unit 120 Storage unit 121 Infrared image storage unit 122 Distance image storage unit 130 Control unit 131 Acquisition unit 1315 Switching unit 1321 Maximum value detection unit 1322 Filter processing unit 133 Saturation determination unit 134 Dynamic threshold calculation unit 135 Centricular extraction unit 136 Correction unit 137 Output control unit 140 Light projection unit 140A First light projection unit 140B Second light projection unit 150 Light receiving unit 150A Light receiving unit 150B Light receiving unit 160 Input unit 170 Display unit A Subject

Claims

1. A distance image generation device comprising: an acquisition unit that acquires a distance image and an infrared image obtained by receiving reflected light reflected by a subject from emitted light; a maximum value detection unit that divides the infrared image into a plurality of blocks and acquires the maximum value of the infrared brightness from the pixels of the block to be processed and the pixels of the blocks surrounding the block to be processed; a saturation determination unit that determines whether the maximum value of the infrared brightness obtained by the maximum value detection unit is a saturation threshold; and a correction unit that corrects at least one of the infrared brightness values ​​of the pixels in the infrared image and the distance measurement values ​​of the pixels in the distance image based on the infrared brightness values ​​of the pixels in the infrared image and the determination result of the saturation determination unit.

2. A distance image generation device according to claim 1, further comprising: a dynamic threshold calculation unit that calculates a dynamic threshold set for each pixel of the block to be processed, wherein the correction unit invalidates the distance measurement value of the distance image corresponding to the position of the pixel when the infrared brightness value of the pixel of the block to be processed is less than the dynamic threshold.

3. A distance image generation device according to claim 1, further comprising: a filter processing unit that performs a filter processing on the infrared brightness values ​​of pixels in the infrared image with predetermined kernel coefficients set, wherein the correction unit invalidates the distance value of a pixel in the distance image corresponding to the position of a pixel if the infrared brightness value of a pixel in the processing target block of the infrared image after performing a filter processing on the infrared brightness values ​​of pixels in the infrared image falls below a dynamic threshold.

4. The distance image generation apparatus according to any one of claims 1 to 3, wherein the acquisition unit acquires information regarding the operating mode, and further comprises a switching unit that switches a light emission unit that emits light used for capturing the infrared image and the distance image based on the operating mode, and the maximum value detection unit changes the size of the unit block when dividing the infrared image into a plurality of blocks based on the operating mode.

5. A method for generating a distance image, comprising the steps of: acquiring a distance image and an infrared image obtained by receiving reflected light reflected by a subject from emitted light; dividing the infrared image into a plurality of blocks and acquiring the maximum value of the infrared brightness from the pixels of the block to be processed and the pixels of the blocks surrounding the block to be processed; determining whether the maximum value of the infrared brightness obtained in the step of acquiring the maximum value is a saturation threshold; and correcting at least one of the infrared brightness value of the pixels of the infrared image and the distance measurement value of the pixels of the distance image based on the infrared brightness value of the pixels of the infrared image and the determination result of the step of determining whether it is a saturation threshold.

6. A program that causes a computer to perform the following steps: acquiring a distance image and an infrared image obtained by receiving reflected light reflected by a subject from emitted light; dividing the infrared image into a plurality of blocks and acquiring the maximum value of the infrared brightness from the pixels of the block to be processed and the pixels of the blocks surrounding the block to be processed; determining whether the maximum value of the infrared brightness obtained in the step of acquiring the maximum value is a saturation threshold; and correcting at least one of the infrared brightness values ​​of the pixels in the infrared image and the distance measurement values ​​of the pixels in the distance image based on the infrared brightness values ​​of the pixels in the infrared image and the determination result of the step of determining whether it is a saturation threshold.

Images