Distance image capturing device, and distance image capturing method
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2026-02-03
- Publication Date
- 2026-08-13
Smart Images
Figure JP2026003727_13082026_PF_FP_ABST
Abstract
Description
Distance Image Capturing Device and Distance Image Capturing Method
[0001] The present invention relates to a distance image capturing device and a distance image capturing method. This application claims priority based on Japanese Patent Application No. 2025-016499 filed in Japan on February 4, 2025, and the contents thereof are incorporated herein by reference.
[0002] A time-of-flight (TOF) method distance image capturing device that measures the distance between a measuring device and an object based on the flight time of light in a measurement space by utilizing the fact that the speed of light is known has been realized (see, for example, Patent Document 1). In such a distance image capturing device, the delay time from the time when a light pulse, which is a pulsed near-infrared light, is irradiated until the light pulse reflected from the subject returns is obtained by accumulating the charges generated by a photoelectric conversion element in a plurality of charge accumulation parts, and the distance to the subject is calculated using the delay time and the speed of light (see, for example, Patent Document 1).
[0003] In such a TOF method distance image capturing device, increasing the measurable distance (measurement distance) range to a longer distance is an issue. In order to measure a long distance, it is difficult to accurately measure the distance unless a certain amount of charges corresponding to the reflected light from a distant subject can be accumulated in the charge accumulation part. Increasing the light amount of the light source is considered as a countermeasure for increasing the measurement distance, but increasing the light amount increases the power consumption. As a countermeasure, it is effective to use a structured light source (for example, dot light, line light, etc.) capable of partially increasing the power density of the light source.
[0004] Japanese Patent No. 4235729 Gazette
[0005] However, since structured light sources emit light in specific patterns such as dots and lines, the brightness and darkness corresponding to the dot or line-like patterns are reflected in the IR (infrared) image and depth image. Specifically, in IR images, the IR value (signal value corresponding to the amount of infrared light received by the pixel) can be increased for parts of the subject illuminated by light from the structured light source, while the IR value will be close to 0 (zero) for parts not illuminated by light from the structured light source. In depth images, the distance can be determined for parts of the subject illuminated by light from the structured light source, while it becomes difficult to calculate the distance for parts not illuminated by light from the structured light source. In other words, there is a problem in that the pixel values in the image vary according to the pattern of the structured light source.
[0006] The present invention has been made to solve the above problems, and its objective is to provide a distance image acquisition device and a distance image acquisition method that can suppress variations in pixel values caused by imaging using a structured light source.
[0007] A first aspect of the distance image imaging apparatus of the present invention comprises a light source unit that irradiates a measurement space with light pulses, a pixel circuit in which a plurality of pixels are arranged in a two-dimensional matrix and each of which has a photoelectric conversion element that generates an electric charge corresponding to the incident light and a plurality of charge storage units that store the electric charge, and a pixel driving circuit that distributes and stores electric charge to each of the plurality of charge storage units at a predetermined storage timing synchronized with the irradiation timing of the light pulses, and a distance image processing unit that controls the irradiation of the light pulses and calculates the distance to a subject in the measurement space based on the amount of electric charge stored in each of the plurality of charge storage units in response to the irradiation of the light pulses, wherein the light source unit has at least a structured light source that emits structured light, and the distance image processing unit performs pixel binning processing to correct pixel data, which is a signal value based on the amount of electric charge stored in the plurality of charge storage units of the first pixel among the plurality of pixels, based on the pixel data of the surrounding pixels, which are the plurality of pixels arranged around the first pixel.
[0008] The present invention relates to a distance image acquisition method performed by a distance image acquisition apparatus comprising: a light source unit that irradiates a measurement space with light pulses; a pixel circuit in which a plurality of pixels are arranged in a two-dimensional matrix and each of the photoelectric conversion elements that generate charge according to the incident light and a plurality of charge storage units that store charge; a pixel driving circuit that distributes and stores charge to each of the plurality of charge storage units at a predetermined storage timing synchronized with the irradiation timing of the light pulses; and a distance image processing unit that controls the irradiation of the light pulses and calculates the distance to a subject in the measurement space based on the amount of charge stored in each of the plurality of charge storage units in response to the irradiation of the light pulses. The distance image acquisition method is performed by a distance image acquisition apparatus comprising: a light source unit that irradiates a measurement space with light pulses; a pixel circuit in which a plurality of pixels are arranged in a two-dimensional matrix and each of the photoelectric conversion elements that generate charge according to the incident light and a plurality of charge storage units that store charge; a pixel driving circuit that distributes and stores charge to each of the plurality of charge storage units at a predetermined storage timing synchronized with the irradiation timing of the light pulses; and a distance image processing unit that controls the irradiation of the light pulses and calculates the distance to a subject in the measurement space based on the amount of charge stored in each of the plurality of charge storage units in response to the irradiation of the light pulses.
[0009] According to the present invention, it is possible to suppress variations in pixel values caused by imaging using a structured light source.
[0010] This is a block diagram showing an example configuration of a distance image acquisition device according to the embodiment. This is a block diagram showing an example configuration of a distance image sensor according to the embodiment. This is a circuit diagram showing an example of the pixel configuration of the embodiment. This is a diagram for explaining the pixel driving method performed by the distance image acquisition device according to the embodiment. This is a schematic diagram showing how reflected light from a structured light source according to the embodiment is received in the light-receiving area. This is a schematic diagram showing the output distribution of pixel data in Figure 5. This is a schematic diagram showing the output distribution of pixel data when Figure 6 is corrected by pixel binning processing. This is a flowchart showing the processing flow performed by the distance image acquisition device according to the embodiment.
[0011] The distance image acquisition device 1 of this embodiment will be described below with reference to the drawings.
[0012] Figure 1 is a block diagram illustrating the schematic configuration of a distance image capturing device 1 according to an embodiment. The distance image capturing device 1 comprises, for example, a light source unit 2, a light receiving unit 3, and a distance image processing unit 4. Figure 1 also shows the subject OB, which is the object whose distance is to be measured by the distance image capturing device 1.
[0013] The light source unit 2 irradiates the subject OB with an optical pulse PO in accordance with the control from the distance image processing unit 4. The light source unit 2 is, for example, a surface-emitting semiconductor laser module such as a vertical cavity surface-emitting laser (VCSEL). The light source unit 2 comprises a light source device 21 and a diffuser plate 22.
[0014] The light source device 21 is a light source that emits laser light in the near-infrared wavelength band (for example, a wavelength band of 850 nm to 940 nm) which becomes the light pulse PO irradiated onto the subject OB. The light source device 21 is, for example, a semiconductor laser light-emitting element. The light source device 21 emits pulsed laser light in response to control from the timing control unit 41.
[0015] The diffuser plate 22 is an optical component that diffuses the near-infrared wavelength laser light emitted by the light source device 21 over a surface area that illuminates the subject OB. The pulsed laser light diffused by the diffuser plate 22 is emitted as an optical pulse PO and irradiates the subject OB.
[0016] The light receiving unit 3 receives the reflected light RL of the light pulse PO reflected from the subject OB, and outputs a pixel signal corresponding to the received reflected light RL. The light receiving unit 3 includes a lens 31 and a distance image sensor 32.
[0017] Lens 31 is an optical lens that guides the incident reflected light RL to the distance image sensor 32. Lens 31 emits the incident reflected light RL towards the distance image sensor 32, causing it to be received (incident) by the pixels 321 provided in the light-receiving area of the distance image sensor 32.
[0018] The distance image sensor 32 is an image sensor. The distance image sensor 32 comprises a plurality of pixels 321 arranged in a two-dimensional matrix. Within each of the plurality of pixels 321 of the distance image sensor 32, there is a photoelectric conversion element, a plurality of charge storage units CS corresponding to this single photoelectric conversion element PD, and a component that distributes charge to each of the plurality of charge storage units CS. In other words, each of the plurality of pixels 321 is an image sensor with a distribution configuration that distributes and stores charge in the plurality of charge storage units.
[0019] The distance image sensor 32 distributes the charge generated by the photoelectric conversion element PD to each of the multiple charge storage units CS in accordance with the control from the timing control unit 41. The distance image sensor 32 also outputs a pixel signal corresponding to the amount of charge distributed to each of the multiple charge storage units CS. The distance image sensor 32 has multiple pixels 321 arranged in a two-dimensional matrix, and outputs a pixel signal for one frame corresponding to each of the multiple pixels 321.
[0020] Here, the configuration of the distance image sensor 32 will be explained using Figure 2. Figure 2 is a block diagram showing the schematic configuration of the image sensor (distance image sensor 32) used in the distance image acquisition device 1 of this embodiment.
[0021] As shown in Figure 2, the distance image sensor 32 includes, for example, a light-receiving area 320 in which a plurality of pixels 321 are arranged in a two-dimensional matrix, and a pixel driving circuit 322. The pixel driving circuit 322 includes, for example, a vertical scanning circuit 323 having a distribution operation, a horizontal scanning circuit 324, a pixel signal processing circuit 325, and a control circuit 326. The plurality of pixels 321 is an example of a pixel circuit.
[0022] The light-receiving region 320 is a region in which multiple pixels 321 are arranged in a two-dimensional matrix. Figure 2 shows an example where they are arranged in a two-dimensional matrix with 8 rows and 8 columns. Each of the multiple pixels 321 accumulates a charge corresponding to the amount of light it receives and outputs an accumulation signal corresponding to the amount of accumulated charge.
[0023] The control circuit 326 comprehensively controls the distance image sensor 32. For example, the control circuit 326 controls the operation of the components of the distance image sensor 32 in response to instructions from the timing control unit 41 of the distance image processing unit 4. In addition, the control of the components of the distance image sensor 32 may be directly performed by the timing control unit 41, in which case the control circuit 326 can be omitted.
[0024] The vertical scanning circuit 323 controls multiple pixels 321 arranged in the light-receiving area 320 row by row in response to control from the control circuit 326. The vertical scanning circuit 323 causes the pixel signal processing circuit 325 to output a voltage signal corresponding to the amount of charge accumulated in each of the multiple charge storage units CS of each of the multiple pixels 321 to be controlled. For example, the vertical scanning circuit 323 distributes and stores the charge converted by the photoelectric conversion element PD to each of the multiple charge storage units CS of each of the multiple pixels 321 to be controlled at an accumulation timing synchronized with the irradiation of the optical pulse PO. In addition, the vertical scanning circuit 323 discharges the charge converted by the photoelectric conversion element PD from a charge discharge unit (drain gate transistor GD described later) during a period different from the accumulation period in which charge is accumulated in each of the multiple charge storage units CS (for example, a readout period).
[0025] The pixel signal processing circuit 325 performs predetermined signal processing (for example, noise suppression processing or A / D conversion processing) on the voltage signals output from multiple pixels 321 in each row to the corresponding vertical signal lines, in response to control from the control circuit 326.
[0026] The horizontal scanning circuit 324 sequentially outputs the signals from the pixel signal processing circuit 325 in a time series, in response to control from the control circuit 326. As a result, the accumulated signal for one frame is sequentially output to the distance image processing unit 4. In the following explanation, it is assumed that the pixel signal processing circuit 325 performs A / D conversion and that the accumulated signal is a digital signal.
[0027] Here, the configuration of the pixel 321 will be explained using Figure 3. Figure 3 is a circuit diagram showing an example of a pixel 321. Figure 3 shows an example of the configuration of one pixel 321 among a plurality of pixels 321 arranged within the light-receiving area 320. In this figure, one pixel 321 is shown to have four signal readout units RU (signal readout units RU1 to RU4).
[0028] Pixel 321 comprises one photoelectric conversion element PD, a drain gate transistor GD, and four signal readout units RU (RU1, RU2, RU3, RU4) that output voltage signals from corresponding output terminals O (O1, O2, O3, O4). Each of the four signal readout units RU comprises a read gate transistor G, a floating diffusion FD, a charge storage capacitor C, a reset transistor RT, a source follower transistor SF, and a selection transistor SL. The charge storage unit CS is composed of the floating diffusion FD and the charge storage capacitor C.
[0029] In Figure 3, the four signal readout units RU are distinguished by adding a number from "1" to "4" after the code "RU" of each unit. Similarly, each component of the four signal readout units RU is also distinguished by indicating a number representing the respective signal readout unit RU after its code.
[0030] In one pixel 321, the signal readout unit RU1 outputs a voltage signal from the output terminal O1. The signal readout unit RU1 comprises a readout gate transistor G1, a floating diffusion transistor FD1, a charge storage capacitor C1, a reset transistor RT1, a source follower transistor SF1, and a selection transistor SL1. The charge storage unit CS1 is composed of the floating diffusion transistor FD1 and the charge storage capacitor C1. The signal readout units RU2 to RU4 have a similar configuration.
[0031] The photoelectric conversion element PD is an embedded photodiode that converts incident light into electricity, generates an electric charge corresponding to the intensity of the incident light, and stores the generated charge. The structure of the photoelectric conversion element PD can be arbitrary. For example, the photoelectric conversion element PD may be a PN photodiode with a structure in which a P-type semiconductor and an N-type semiconductor are joined, or a PIN photodiode with a structure in which an I-type semiconductor is sandwiched between a P-type semiconductor and an N-type semiconductor. Furthermore, the photoelectric conversion element PD is not limited to a photodiode, and may be, for example, a photogate type photoelectric conversion element.
[0032] The drain gate transistor GD is a transistor used to discard the charge generated in the photoelectric conversion element PD. When the drain gate transistor GD is controlled to the ON state by the pixel driving circuit 322, it discards the charge generated in the photoelectric conversion element PD (i.e., it resets the photoelectric conversion element PD).
[0033] The pixel driving circuit 322 drives the pixel 321, and distributes the charge generated by the photoelectric conversion element PD's photoelectric conversion of incident light to each of the four charge storage units CS. It then outputs voltage signals corresponding to the amount of charge of each distributed charge to the pixel signal processing circuit 325.
[0034] For example, the pixel driving circuit 322 controls the storage drive signals TX1 to TX4, corresponding to each of the charge storage units CS1 to CS4, in sequence to turn on when driving the pixel 321, in synchronization with the irradiation timing of the optical pulse PO. This causes the read gate transistors G1 to G4, corresponding to each of the charge storage units CS1 to CS4, to conduct in sequence, distributing and storing the charge in the corresponding charge storage units CS1 to CS4. As a result, charge is stored in the charge storage units CS1, CS2, CS3, and CS4 in that order.
[0035] Furthermore, the pixel 321 is not limited to a configuration with four signal readout units RU as shown in Figure 3, but can be any pixel with multiple signal readout units RU. In other words, the number of signal readout units RU (charge storage units CS) provided by each of the multiple pixels 321 arranged in the distance image sensor 32 may be two, three, or five or more.
[0036] Furthermore, Figure 3 shows an example in which the charge storage unit CS is composed of a floating diffusion FD and a charge storage capacitance C. However, the charge storage unit CS only needs to be composed of a floating diffusion FD, and the pixel 321 may not have a charge storage capacitance C.
[0037] Returning to the explanation of Figure 1, the distance image processing unit 4 controls the distance image acquisition device 1 and calculates the distance to the subject OB. The distance image processing unit 4 comprises a timing control unit 41, a distance calculation unit 42, and a measurement control unit 43.
[0038] The timing control unit 41 controls the timing of outputting various control signals required for measurement, in accordance with the control of the measurement control unit 43. These various control signals include, for example, a signal to control the irradiation of the light pulse PO, a signal to distribute and store reflected light RL in multiple charge storage units CS, and a signal to control the number of integration cycles per frame. The number of integration cycles is the number of times the process of distributing and storing charge in the charge storage units CS (see Figure 3) per frame is repeated. The product of this number of integration cycles and the time (storage time) for storing charge in each charge storage unit CS in one charge distribution cycle is the exposure time per frame.
[0039] The distance calculation unit 42 outputs distance information calculated based on the pixel signal output from the distance image sensor 32, determining the distance to the subject OB. The distance calculation unit 42 calculates the delay time from the irradiation of the light pulse PO to the reception of the reflected light RL based on the amount of charge accumulated in the multiple charge storage units CS. The distance calculation unit 42 calculates the distance to the subject OB according to the calculated delay time.
[0040] The distance calculation unit 42 calculates the delay time Td using, for example, the following equation (1). Equation (1) assumes that the amount of charge corresponding to the ambient light component, which is included in the amount of charge stored in the charge storage units CS1 and CS2, is the same as the amount of charge stored in the charge storage unit CS3.
[0041] Td = To × (Q2 - Q3) / (Q1 + Q2 - 2 × Q3) … (Equation 1) However, To is the period during which the optical pulse PO is irradiated. Q1 is the amount of charge accumulated in the charge accumulation unit CS1. Q2 is the amount of charge accumulated in the charge accumulation unit CS2. Q3 is the amount of charge accumulated in the charge accumulation unit CS3.
[0042] The distance calculation unit 42 multiplies the delay time Td obtained by Equation (1) by the speed of light (velocity) in the case of the short-distance light-receiving pixels to calculate the round-trip distance to the subject OB. Then, the distance calculation unit 42 measures the distance to the subject OB by setting the calculated round-trip distance to 1 / 2.
[0043] The measurement control unit 43 controls the timing control unit 41. For example, the measurement control unit 43 sets the integration number and the accumulation time in one frame, and controls the timing control unit 41 so that imaging is performed with the set content.
[0044] With such a configuration, in the distance image capturing device 1, the light receiving unit 3 receives the reflected light RL of the near-infrared wavelength band optical pulse PO irradiated by the light source unit 2 to the subject OB, and the distance image processing unit 4 calculates the distance to the subject OB and outputs distance information.
[0045] In FIG. 1, the distance image capturing device 1 having a configuration in which the distance image processing unit 4 is provided inside the distance image capturing device 1 is shown. However, the distance image processing unit 4 may be provided outside the distance image capturing device 1.
[0046] The driving of the pixel 321 will be explained using Figure 4. Figure 4 is a diagram illustrating the method of driving the pixel 321 performed by the distance image imaging device 1 of the embodiment. Driving the pixel 321 means controlling the timing signals (signals corresponding to each of the items "G1" to "G4" and "GD" shown in Figure 4) so that charge is accumulated in the charge accumulation unit CS provided by the pixel 321. Figure 4 shows a timing chart when the pixel 321 is driven with reference to the irradiation timing of the light pulse PO. In this figure, timing charts for elements corresponding to each of the items "LI", "G1" to "G4", and "GD" are shown. "LI" indicates the irradiation timing of the light pulse PO, where the ON state means that light is irradiated and the OFF state means that light is not irradiated. "G1" to "G4" indicate the accumulation timing of the readout gate transistors G1 to G4, where the ON state means that charge is accumulated and the OFF state means that charge is not accumulated. "GD" indicates the drive timing of the drain gate transistor GD. When it is turned ON, charge is discharged, and when it is turned OFF, charge is not discharged. As shown in this figure, when driving the pixel 321, charge is sequentially accumulated in the four charge accumulation units CS1 to CS4 at an accumulation timing synchronized with the irradiation timing of the light pulse PO. For example, at the same timing as the irradiation timing when the light pulse PO is irradiated, the drain gate transistor GD is turned OFF, and the read gate transistors G1 to G4 are sequentially turned ON. Specifically, the distance image processing unit 4, via the pixel driving circuit 322, turns OFF the drain gate transistor GD and turns ON the read gate transistor G1 at the irradiation timing. After a specific accumulation time (for example, the same time as the irradiation time when the light pulse PO is irradiated) has elapsed since turning ON the read gate transistor G1, the read gate transistor G1 is turned OFF. At the same time that the read gate transistor G1 is turned OFF, the read gate transistor G2 is turned ON. The read gate transistor G2 is turned ON, and after the storage time To has elapsed, the read gate transistor G2 is turned OFF.At the moment the read gate transistor G2 is turned off, the read gate transistor G3 is turned on. After the storage time To has elapsed since the read gate transistor G3 was turned on, the read gate transistor G3 is turned off. At the moment the read gate transistor G3 is turned off, the read gate transistor G4 is turned on. After the storage time To has elapsed since the read gate transistor G4 was turned on, the read gate transistor G4 is turned off, and the drain gate transistor GD is turned on. In one frame, the distance image processing unit 4 repeatedly performs the drive (charge storage in the charge storage unit CS) according to the drive pattern shown in Figure 4 a predetermined number of times, and then reads out the charge stored in the charge storage unit CS as pixel data. The distance image processing unit 4 generates at least one of an IR image or a distance image using the read out pixel data.
[0047] In this embodiment, the light source unit 2 has at least a structured light source. A structured light source is a light source that emits light in a specific pattern, such as dots, lines, or a combination thereof. For example, a structured light source is a dot light source that emits dot light, or a line light source that emits line light. By using a structured light source, the power of the emitted light (irradiation intensity per unit area) can be increased without increasing power consumption, such as by increasing the light source output, and the distance that can be reached by the emitted light can be extended, thereby expanding the measurable distance. On the other hand, when using a structured light source, it becomes difficult to accurately measure the distance in areas that are not illuminated by light.
[0048] As a countermeasure, in this embodiment, the distance image processing unit 4 performs pixel binning processing. Pixel binning processing is a process of correcting pixel data of a target pixel 321 (first pixel) among a plurality of pixels 321 based on pixel data of other pixels 321 arranged around the first pixel. By performing pixel binning processing, it is possible to perform corrections such as averaging the pixel data of each of the plurality of pixels 321 provided in the light receiving region 320 according to the pixel data of the surrounding pixels 321, and to suppress variations (non-uniformity) in pixel values for each pixel 321 caused by the pattern of the structured light source.
[0049] The distance image processing unit 4 may implement pixel binning processing by hardware binning or software binning. Hardware binning is a process of executing pixel binning processing by an IC (integrated circuit) or the like as hardware. In hardware binning, for example, a pixel binning circuit that executes pixel binning processing is provided on the chip of the distance image sensor 32, and pixel binning processing is performed by executing the pixel binning processing. Software binning is a process of executing pixel binning processing by causing a program (software) related to pixel binning processing to be executed by a CPU or the like as hardware.
[0050] The pixel data here is a pixel value based on the amount of charge stored in the charge storage unit CS of the pixel 321. The pixel data may be a distance value calculated based on equation (1), or an IR value. The IR value is the amount of infrared light received by each of the multiple pixels 321 in response to irradiation with an infrared light pulse PO. For example, the IR value is a value corresponding to the amount of charge (Q1 + Q2 + Q3 + Q4). Here, Q1 is the amount of charge stored in the charge storage unit CS1. Q2 is the amount of charge stored in the charge storage unit CS2. Q3 is the amount of charge stored in the charge storage unit CS3. Q4 is the amount of charge stored in the charge storage unit CS4. Alternatively, the IR value may be a value corresponding to the amount of charge (Q1 + Q2 + Q3 + Q4 - 4 × Qb). Here, Qb is the amount of charge corresponding to the ambient light component stored in each of the charge storage units CS1 to CS4. For example, among the charge storage units CS1 to CS4, the amount of charge stored in charge storage unit CS, which stores charge at a time when reflected light RL is not received, can be defined as the amount of charge Qb corresponding to the ambient light component. The amount of charge Qb corresponding to the ambient light component is assumed to be the same amount of charge stored in each of the charge storage units CS1 to CS4.
[0051] The distance image processing unit 4 averages the pixel data of multiple adjacent pixels 321 in the light-receiving area 320 by simply adding them together and averaging. Multiple adjacent pixels 321 may be a group of pixels arranged adjacent to each other vertically and horizontally, or a group of pixels arranged in a row vertically or horizontally. For example, when using four adjacent pixels 321, one can use a group of pixels enclosed by a (2x2) movable window where pixels are adjacent vertically and horizontally, a group of pixels enclosed by a (1x4) movable window arranged in a vertical row, or a group of pixels enclosed by a (4x1) movable window arranged in a horizontal row. The distance image processing unit 4 performs pixel binning by using the average value obtained by averaging the pixel data of multiple adjacent pixels 321 as the corrected pixel data for each of those pixels 321. Furthermore, when using a dot light source or a line light source, more effective image correction is possible by switching the range of the pixel binning process according to the radius of the dot or the width of the line. For example, when the dot or line size is large, binning is effective in correcting peripheral pixels, allowing for the acquisition of uniform long-distance images. On the other hand, when the dot or line size is small, excessive binning can blur the dot / line pattern, potentially impairing image resolution and sharpness. However, by appropriately controlling the binning range according to the dot radius and line width, it is possible to suppress variations in pixel values while maintaining the characteristics of the dot / line pattern, thereby maintaining optimal image quality according to the application and imaging conditions. Furthermore, in this embodiment, the weighting method can be selected and changed according to the dot size of the dot light source and the line width of the line light source during pixel binning. For example, when the dot size or line width is large, the variation in pixel values is relatively small, so applying binning by simple averaging including peripheral pixels can effectively homogenize the overall image. On the other hand, when the dot size or line width is small, the difference in signal values between the center or the center and periphery of the line is large, and simple averaging may blur the dot / line pattern.Therefore, when the dot size or line width is small, a weighted averaging process is performed in which a high weighting coefficient is set for the central pixel (or the pixel that receives the most light at the center of the line), and a low weighting coefficient or zero is set for peripheral pixels with low signal values (charge amount, etc.). Specifically, spatial weighting based on a Gaussian distribution (a method that reduces the weight according to the distance from the center or the center of the line), setting weighting coefficients according to the ratio of received charge amounts, and setting adaptive weights based on the S / N ratio can be utilized.
[0052] Figure 5 is a schematic diagram showing how reflected light from a structured light source is received in the light-receiving area 320 of the embodiment. In the distance image acquisition device 1, when the structured light source is irradiated, the reflected light RL, which is the light pulse PO irradiated from the structured light source reflected off the subject OB, is received in the light-receiving area 320. As shown in this figure, the light-receiving area 320 receives reflected light RL according to the pattern of the structured light source (in this case, a dot pattern). As a result, the light-receiving area 320 contains a mixture of pixels 321A, which receive reflected light RL across the entire pixel 321; pixels 321B, which receive reflected light RL in part of the pixel 321; and pixels 321C, which do not receive reflected light RL.
[0053] Figure 6 is a schematic diagram showing the output distribution of pixel data for each pixel 321 in Figure 5. As shown in this figure, pixel 321A, which receives reflected light RL across its entire pixel, has a large pixel output (signal value of pixel data). Pixel 321B, which receives reflected light RL in only a portion of its pixel, has a moderate pixel output (signal value of pixel data). Pixel 321C, which does not receive reflected light RL, has a small pixel output (signal value of pixel data).
[0054] Figure 7 is a schematic diagram showing the output distribution of pixel data when Figure 6 is corrected by pixel binning. As shown in this figure, by performing pixel binning, the output values of each pixel data of the pixels 321 arranged in the light-receiving area 320 can be made uniform.
[0055] Thus, in this embodiment, by using a structured light source, the light pulse PO can reach distant locations where it is difficult to reach with diffused light from a diffuse light source, and the reflected light RL reflected from the distant subject OB can be received by the light receiving area 320. On the other hand, using a structured light source creates areas where the light pulse PO is not irradiated, and to counteract the resulting variation in the output values of the pixel data, pixel binning is performed. This allows the output values of the pixel data to be corrected so that they are large enough to measure distance even in areas where the light pulse PO is not irradiated, thereby generating a distance image or an IR image without variation.
[0056] Here, in the distance image acquisition device 1, the light source unit 2 may be configured to have both a structured light source and a diffuse light source. A diffuse light source is a light source that diffuses light to illuminate the subject OB uniformly. This allows the distance image acquisition device 1 to use different light sources depending on the situation. For example, a diffuse light source can be used when the subject OB is a nearby object located relatively close, and a structured light source can be used when the subject OB is a distant object located relatively far away, thus allowing the light source to be used differently depending on the situation.
[0057] Furthermore, if the light source unit 2 has both a structured light source and a diffuse light source, the distance image processing unit 4 performs pixel binning when the structured light source is used. In other words, when generating a distance image or IR image using the amount of charge accumulated in multiple charge storage units CS in response to the irradiation of the structured light source as an optical pulse PO, pixel binning is performed. This prevents variations in the pixel values of the distance image or IR image that correspond to the pattern of the structured light source.
[0058] On the other hand, the distance image processing unit 4 does not need to perform pixel binning when a diffuse light source is used. In the case of a diffuse light source, the subject OB is uniformly illuminated with light pulses PO, so there is no variation in the pixel values of the distance image or IR image that corresponds to the light irradiation pattern, as is the case with structured light sources. By not performing pixel binning when a diffuse light source is used, the processing burden of performing pixel binning can be reduced.
[0059] Furthermore, the distance image processing unit 4 may determine whether or not there is movement in the subject OB, and whether or not to use a structured light source or a diffuse light source for irradiating with the light pulse PO in subsequent imaging. Any method can be used to detect whether or not there is movement in the subject OB (motion detection).
[0060] For example, the distance image processing unit 4 extracts the region (subject region) in which the subject OB is captured from the IR image or distance image. The distance image processing unit 4 determines that there is movement in the subject OB if the shape of the subject region extracted this time is similar to the subject region extracted last time and the position of the subject region has changed. On the other hand, the distance image processing unit 4 determines that there is no movement in the subject OB if the shape of the subject region extracted this time is similar to the subject region extracted last time and the position of the subject region has not changed. Furthermore, the distance image processing unit 4 determines that there is no movement in the subject OB if the shape of the subject region extracted this time is not similar to the subject region extracted last time, assuming that a different subject OB was captured than last time. Alternatively, the distance image processing unit 4 can apply image processing technology to the IR image and perform motion detection using the result of determining whether or not a moving object such as a person or vehicle is captured in the IR image by performing object recognition, for example. Alternatively, motion detection may be performed using an external sensor different from the distance image capturing device 1. As an external sensor, for example, a passage sensor that detects the movement of a subject (e.g., a person) in response to the interruption of laser light emitted from the light transmitting unit can be used. In this case, the distance image capturing device 1 and the external sensor are connected in a communicative manner, and the detection result from the external sensor is output to the distance image capturing device 1. Based on the detection result from the external sensor, the distance image capturing device 1 controls the irradiation of the optical pulse PO using a diffuse light source if there is movement in the subject OB, and a structured light source if there is no movement.
[0061] The distance image processing unit 4 detects whether or not there is movement in the subject OB by motion detection, and if no movement is detected, it drives each of the multiple pixels 321 in accordance with the illumination of the structured light source as an optical pulse PO. In this case, the distance image processing unit 4 may reduce the frequency of illumination of the optical pulse PO by irradiating the subject OB intermittently, for example, every second. This is because if the subject OB is not moving, there is no need to measure the change in distance due to movement in detail. By making the illumination of the optical pulse PO by the structured light source intermittent, the measurable distance can be expanded compared to when a diffuse light source is used while suppressing the increase in power consumption. Furthermore, by intermittently continuing the illumination of the optical pulse PO by the structured light source and the driving of the multiple pixels 321 in accordance with the illumination of the optical pulse PO, motion detection can be performed intermittently. Therefore, when movement occurs in the subject OB, it is possible to appropriately take action such as changing from intermittent driving to continuous driving or returning to the diffuse light source, and appropriate imaging can be performed according to the conditions of the measurement space.
[0062] The distance image processing unit 4 detects whether or not there is movement in the subject OB by motion detection. If no movement is detected, and the structured light source is irradiated as an optical pulse PO, the unit corrects the pixel data of each of the multiple pixels 321 driven in response to the irradiation of the structured light source as an optical pulse PO by pixel binning. When pixel binning is performed, the pixel data becomes uniform, which may reduce the effective resolution (number of pixels). However, if no movement is detected in the subject OB, even if the resolution (number of pixels) is reduced and the subject OB is captured coarsely in the image, it is less likely to be perceived by the human eye and less likely to cause discomfort. For this reason, when there is movement in the subject OB, the structured light source is irradiated as an optical pulse PO, and the pixel data of each of the multiple pixels 321 driven in response to the irradiation of the structured light source as an optical pulse PO is corrected by pixel binning, thereby widening the measurable distance compared to when a diffuse light source is used, without causing discomfort.
[0063] Alternatively, the distance image processing unit 4 may detect whether or not there is movement in the subject OB by motion detection, and if no movement is detected, it may irradiate the structured light source as an optical pulse PO. In this case, the number of pixels 321 to be driven in response to the irradiation of the structured light source as an optical pulse PO may be limited to the pixels 321 that receive reflected light RL. In this case, the distance image processing unit 4 generates the pixel data of the undriven pixels 321 by performing pixel binning using the pixel data of the driven pixels 321. By reducing the number of driven pixels 321, the increase in processing load can be suppressed. For example, the distance image processing unit 4 may pre-determine which pixels 321 receive reflected light RL (reflected light receiving pixels) and which pixels 321 do not receive reflected light RL (non-reflected light receiving pixels) in response to the irradiation of the structured light source as an optical pulse PO. The distance image processing unit 4 detects whether or not there is movement in the subject OB by motion detection, and if no movement is detected, it irradiates the structured light source as an optical pulse PO. The distance image processing unit 4 controls the operation of reflected light receiving pixels among the multiple pixels 321 arranged in the light receiving area 320, in response to the irradiation of the structured light source as an optical pulse PO, while controlling the operation of non-reflected light receiving pixels. After driving the reflected light receiving pixels, the distance image processing unit 4 reads out the pixel data of the reflected light receiving pixels. The distance image processing unit 4 calculates the average value of the pixel data of the reflected light receiving pixels arranged around the non-reflected light receiving pixels, and uses the calculated average value as the pixel data of that non-reflected light receiving pixel.
[0064] Furthermore, if the subject OB is moving, the distance image processing unit 4 may determine whether the light source used to irradiate the light pulse PO should be a structured light source or a diffuse light source, depending on the distance to the subject OB and the direction in which the subject OB is moving.
[0065] The distance image processing unit 4 pre-classifies the distances that the distance image imaging device 1 can measure into three categories: short distance, medium distance, and long distance. For example, the short distance is the distance that can be reached by a light pulse PO from a diffuse light source located near the distance image imaging device 1 while maintaining a sufficient amount of light, and it is a region in which the distance can be measured even if the subject OB moves away from the distance image imaging device 1. The medium distance is a region in which a light pulse PO from a diffuse light source can reach, but it may be difficult to accurately measure the distance if the subject OB is moving away from the distance image imaging device 1. The long distance is a region in which a light pulse PO from a structured light source can reach, but a light pulse PO from a diffuse light source cannot reach, or even if it does, it is difficult to accurately measure the distance because it is difficult to receive the reflected light RL with a sufficient amount of light.
[0066] The distance image processing unit 4 calculates the distance to the subject OB captured in the distance image using equation (1), and determines whether the subject OB is at a short, medium, or long distance based on the calculated distance. The distance image processing unit 4 also detects whether or not there is movement in the subject OB.
[0067] The distance image processing unit 4, when the subject OB is at close range, will perform the next image using a diffuse light source, regardless of whether the subject OB is moving or not. The distance image processing unit 4, when the subject OB is at far distance, will perform the next image using a structured light source, regardless of whether the subject OB is moving or not. The distance image processing unit 4, when the subject OB is at medium distance and is not moving, will perform subsequent images using a diffuse light source. Furthermore, when the subject OB is at medium distance and is moving, the distance image processing unit 4 will decide whether to perform subsequent images using a diffuse light source or a structured light source, depending on the direction of movement. Specifically, if the subject OB is moving and the direction of movement is toward the distance image imaging device 1, subsequent images will be performed using a diffuse light source. On the other hand, if the subject OB is moving and the direction of movement is toward the distance image imaging device 1, subsequent images will be performed using a structured light source.
[0068] Furthermore, when performing pixel binning, the distance image processing unit 4 may determine whether or not to perform pixel binning for each pixel 321. For example, the distance image processing unit 4 may determine whether or not to perform pixel binning according to the amount of light received by each of the multiple pixels 321.
[0069] Generally, as long as none of the multiple pixels 321 are saturated, the greater the amount of reflected light RL received by each of the multiple pixels 321, the higher the accuracy of the pixel data. Saturation here refers to the phenomenon where the amount of light received by each of the multiple pixels 321 exceeds the upper limit of the amount of light corresponding to the amount of charge that the charge storage unit CS can store. This is because it is known that a certain amount of noise N caused by circuit configuration, signal processing, etc., is mixed into the pixel data, and the greater the amount of reflected light RL received by each of the multiple pixels 321 (signal value S), the larger the ratio of noise N to signal value S (S / N ratio). In other words, assuming that none of the multiple pixels 321 are saturated, the accuracy of the pixel data in pixel 321A, where the entire pixel 321 receives reflected light RL, as shown in Figure 5, will be high. The accuracy of the pixel data in pixel 321B, where only a part of the pixel 321 receives reflected light RL, will be moderate. The accuracy of pixel data in pixels 321C that did not receive reflected light RL will be low. Therefore, when a structured light source is used to irradiate with an optical pulse PO, it is possible to accurately calculate the distance for pixels 321 that receive reflected light RL with an intensity of light above the threshold. For this reason, from the standpoint of accuracy, pixel binning is not always necessary. On the other hand, for pixels 321 that receive only reflected light RL with an intensity of light below the threshold, it is considered difficult to accurately calculate the distance, so performing pixel binning can improve the accuracy of the pixel data.
[0070] From this perspective, the distance image processing unit 4 does not perform pixel binning on pixels 321 where the amount of charge accumulated in each of the multiple pixels 321 in response to irradiation with an optical pulse PO from a structured light source is greater than or equal to a threshold. On the other hand, the distance image processing unit 4 performs pixel binning on pixels 321 where the amount of charge accumulated in each of the multiple pixels 321 in response to irradiation with an optical pulse PO from a structured light source is less than a threshold. Here, the amount of charge accumulated in each of the multiple pixels 321 may be the sum of the charge amounts accumulated in each of the multiple charge storage units CS provided in each of the multiple pixels 321 (Q1 + Q2 + Q3 + Q4 - 4 × Qb), or it may be the charge amount excluding the ambient light component (Q1 + Q2 + Q3 + Q4 - 4 × Qb). Here, Q1 is the amount of charge accumulated in charge storage unit CS1. Q2 is the amount of charge accumulated in charge storage unit CS2. Q3 is the amount of charge accumulated in charge storage unit CS3. Q4 is the amount of charge stored in the charge storage unit CS4. Qb is the amount of charge corresponding to the ambient light component stored in each of the charge storage units CS1 to CS4. Alternatively, as the amount of charge stored in the pixel 321, a representative value (for example, the maximum value or average value) of the amount of charge stored in each of the four charge storage units CS of the pixel 321 may be used. In this case, the distance image processing unit 4 does not perform pixel binning on pixels 321 whose representative value is greater than or equal to a threshold, and performs pixel binning on pixels 321 whose representative value is less than a threshold.
[0071] Furthermore, when performing pixel binning, the distance image processing unit 4 may average the pixel data of multiple adjacent pixels 321 by weighted addition. For example, the distance image processing unit 4 can use a weighting coefficient corresponding to the amount of light received by the pixel 321. This allows the pixel data of pixels 321 that received more charge (i.e., high-precision pixel data) to be reflected more significantly through averaging during pixel binning. For example, the distance image processing unit 4 may set the weighting coefficient of pixels 321 that have accumulated less charge than a threshold among the surrounding pixels arranged around the target pixel to be subjected to pixel binning to 0 (zero). In this case, averaging can be performed using only the pixel data of pixels 321 that received reflected light RL with an amount of light greater than or equal to the threshold. More specifically, the distance image processing unit 4 may target the target pixel 321 with an accumulated charge less than a threshold for pixel binning, and perform pixel binning using reflected light receiving pixels among the surrounding pixels arranged around the target pixel whose accumulated charge is greater than or equal to the threshold.
[0072] Here, the processing flow of the depth image imaging device 1 will be explained using Figure 8. Figure 8 is a flowchart showing the processing flow of the depth image imaging device 1 according to the embodiment. First, the depth image imaging device 1 performs motion detection (step S10). The depth image imaging device 1 determines whether or not motion of the subject OB has been detected (step S11), and if no motion is detected, it drives the pixels 321 by irradiating them with light pulses using a structured light source (step S12). As a result, reflected light RL according to the light ray pattern from the structured light source is accumulated in the pixels 321. The depth image imaging device 1 calculates pixel data for each pixel 321 (step S13). The depth image imaging device 1 corrects the pixel data by pixel binning (step S14). The depth image imaging device 1 generates an image (IR image or depth image) based on the corrected pixel data (step S15). On the other hand, if motion is detected in step S11, the depth image imaging device 1 drives the pixels 321 by irradiating them with light pulses using a diffuse light source (step S16). The depth image acquisition device 1 calculates pixel data for each pixel 321 (step S17). The depth image acquisition device 1 generates an image (IR image or depth image) based on the pixel data (step S18). After generating an image in step S15 or S18, the depth image acquisition device 1 returns to the process shown in step S10 and performs subsequent imaging.
[0073] As described above, the distance image imaging device 1 of this embodiment comprises a light source unit 2, a light receiving unit 3, and a distance image processing unit 4. The light source unit 2 has at least a structured light source that emits structured light. The distance image processing unit 4 performs pixel binning processing to correct the pixel data of a first pixel among a plurality of pixels 321 arranged in the light receiving area 320 based on the pixel data of surrounding pixels arranged around the first pixel. As a result, in the distance image imaging device 1 of this embodiment, pixel data can be made uniform by performing pixel binning processing in imaging corresponding to irradiation with an optical pulse PO using a structured light source. Therefore, it is possible to suppress variations in pixel values caused by imaging using a structured light source.
[0074] The distance image acquisition device 1 and distance image processing unit 4 in the above-described embodiment may be implemented in whole or in part by a computer. In that case, the program for implementing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be loaded into a computer system and executed. The term "computer system" here includes hardware such as an OS and peripheral devices. The term "computer-readable recording medium" refers to portable media such as flexible disks, magneto-optical disks, ROMs, CD-ROMs, and storage devices such as hard disks built into a computer system. Furthermore, the term "computer-readable recording medium" may also include materials that dynamically hold programs for a short period of time, such as communication lines used when transmitting programs via networks such as the Internet or communication lines such as telephone lines, and materials that hold programs for a certain period of time, such as volatile memory inside a computer system that acts as a server or client in that case. The program may be a program for implementing a part of the functions described above, or it may be a program that can implement the functions described above in combination with a program already recorded in the computer system, or it may be a program implemented using a programmable logic device such as an FPGA.
[0075] Although embodiments of this invention have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments and includes designs, device configurations, correction processes, filtering processes, etc., that do not depart from the spirit of this invention.
[0076] 1...Distance image acquisition device 2...Light source unit 3...Light receiving unit 32...Distance image sensor 321...Pixel (pixel circuit) 323...Vertical scanning circuit 4...Distance image processing unit 41...Timing control unit 42...Distance calculation unit 43...Measurement control unit CS...Charge storage unit PO...Optical pulse
Claims
1. A distance image imaging device comprising: a light source unit that irradiates a measurement space with light pulses; a pixel circuit in which a plurality of pixels are arranged in a two-dimensional matrix and each of the photoelectric conversion elements that generate charge according to the incident light and a plurality of charge storage units that store charge; a pixel driving circuit that distributes and stores charge to each of the plurality of charge storage units at a predetermined storage timing synchronized with the irradiation timing of the light pulses; and a distance image processing unit that controls the irradiation of the light pulses and calculates the distance to a subject in the measurement space based on the amount of charge stored in each of the plurality of charge storage units in response to the irradiation of the light pulses, wherein the light source unit has at least a structured light source that emits structured light, and the distance image processing unit performs pixel binning processing to correct pixel data, which is a signal value based on the amount of charge stored in the plurality of charge storage units of a first pixel among the plurality of pixels, based on the pixel data of surrounding pixels, which are a plurality of pixels arranged around the first pixel.
2. The distance image imaging apparatus according to claim 1, wherein the light source unit comprises both the structured light source and a diffuse light source that emits diffuse light.
3. The distance image processing unit corrects the pixel data of each of the plurality of pixels driven in response to the irradiation of the structured light source as the light pulse by the pixel binning process, as described in claim 1.
4. The distance image processing unit detects whether or not there is movement in the subject, and if no movement is detected, irradiates the structured light source as the light pulse, as described in claim 1.
5. The distance image processing unit detects whether or not there is movement in the subject, irradiates the subject with the structured light source as a light pulse if no movement is detected, and corrects the pixel data of each of the plurality of pixels driven in response to the irradiation of the structured light source as a light pulse by the pixel binning process, as described in claim 1.
6. The distance image capturing apparatus according to claim 1, wherein the light source unit has both the structured light source and the diffuse light source that emits diffuse light, and the distance image processing unit detects whether or not there is movement in the subject, irradiates with the structured light source as the light pulse when no movement is detected, and irradiates with the diffuse light source as the light pulse when movement is detected.
7. The distance image imaging apparatus according to claim 1, wherein the structured light source is a dot light source that emits dot light, or a line light source that emits line light.
8. A light receiving unit having a light source unit that irradiates a measurement space with light pulses; a pixel circuit in which a plurality of pixels are arranged in a two-dimensional matrix and each of the photoelectric conversion elements that generate charge according to the incident light and a plurality of charge storage units that store charge; a pixel driving circuit that distributes and stores charge to each of the plurality of charge storage units at a predetermined storage timing synchronized with the irradiation timing of the light pulses; a distance image processing unit that controls the irradiation of the light pulses and calculates the distance to an object present in the measurement space based on the amount of charge stored in each of the plurality of charge storage units in response to the irradiation of the light pulses; wherein the light source unit has at least a structured light source that emits structured light; the distance image processing unit performs pixel binning processing to correct pixel data, which is a signal value based on the amount of charge stored in the plurality of charge storage units of the first pixel among the plurality of pixels, based on the pixel data of the surrounding pixels, which are the plurality of pixels arranged around the first pixel; and the light source unit has both the structured light source and a diffuse light source that emits diffuse light. The distance image processing unit detects whether or not there is movement in the subject, and if no movement is detected, irradiates the subject with the structured light source as a light pulse, in a distance image acquisition device.
9. The distance image processing unit pre-classifies the distances that the distance image imaging device can measure into three categories: short distance, medium distance, and long distance, as described in claim 1.
10. A distance image imaging method performed by a distance image imaging device, the device comprising: a light source unit that irradiates a measurement space with light pulses; a pixel circuit in which a plurality of pixels are arranged in a two-dimensional matrix and each of the photoelectric conversion elements that generate charge according to the incident light and a plurality of charge storage units that store charge; a pixel driving circuit that distributes and stores charge to each of the plurality of charge storage units at a predetermined storage timing synchronized with the irradiation timing of the light pulses; and a distance image processing unit that controls the irradiation of the light pulses and calculates the distance to a subject in the measurement space based on the amount of charge stored in each of the plurality of charge storage units in response to the irradiation of the light pulses, wherein the light source unit has at least a structured light source that emits structured light, and the distance image processing unit performs pixel binning processing to correct pixel data, which is a signal value based on the amount of charge stored in the plurality of charge storage units of a first pixel among the plurality of pixels, based on the pixel data of surrounding pixels, which are a plurality of pixels arranged around the first pixel.