Light detection device

Abstract

[Problem] To detect an event by a simple method regardless of illuminance, and to realize area saving and power saving. [Solution] A light detection device comprises: a photodiode that performs avalanche multiplication in accordance with incident photons; a control circuit that controls switching between a first state in which the photodiode can detect the photons and a second state in which the photodiode is recharged; a counter that counts the number of times the photons are detected by the photodiode; and an event detection circuit that detects, on the basis of a difference between a threshold that is constant regardless of illuminance and a count value of the counter, an event when a change in brightness exceeds the threshold.

Description

Light detection device

[0001] This disclosure relates to a photodetector.

[0002] A technique for measuring brightness changes using a first count circuit that counts the number of photons and a second counter circuit that counts time is known (see Patent Document 1). In Patent Document 1, in order to compare the time it takes for the number of photons to reach a certain reference value, the difference between the current count value of the second counter circuit and the past count value of the second counter circuit is compared with a threshold to detect an event.

[0003] Japanese Patent Publication No. 2022-067623

[0004] However, the difference described in Patent Document 1 requires that the threshold value changes depending on the reference brightness, thus necessitating the preparation of a threshold value corresponding to the reference brightness, which complicates the control of event detection. Furthermore, Patent Document 1 also states that the ratio of count values ​​may be used. While using the ratio of count values ​​allows for a constant threshold value, expanding the dynamic range of event detection requires the use of a high-speed clock and an increase in the number of bits in the first and second counters, making it difficult to reduce the size and power consumption.

[0005] Therefore, this disclosure provides a light detection device that can detect events using a simple method regardless of illuminance, and that can achieve area saving and power saving.

[0006] To solve the above problems, the present disclosure provides a light detection device comprising: a photodiode that performs avalanche multiplication in response to incident photons; a control circuit that controls switching between a first state in which the photodiode can detect the photons and a second state in which the photodiode is recharged; a counter that counts the number of times the photons have been detected by the photodiode; and an event detection circuit that detects an event when a change in brightness exceeds a threshold value, based on the difference between a certain threshold value and the count value of the counter, regardless of the illuminance.

[0007] The control circuit may switch the recharging period of the photodiode using two or more multiplication periods of the reference period.

[0008] The event detection circuit may combine multiple recharge periods to detect the event based on the count value which has a logarithmic response to illuminance.

[0009] The control circuit generates a pulse signal corresponding to the potential change of the cathode or anode of the photodiode, and the counter may count the number of pulse signals.

[0010] The counter distinguishes and counts the photons incident during the first period and the photons incident during the second period, which is longer than the first period, and the control circuit may use the sum of the first and second periods as the recharge period of the photodiode.

[0011] The counter may update its count value twice when a photon is incident during the first period, and update its count value once when a photon is incident during the second period.

[0012] The counter may change the count value by distinguishing between photons incident during two or more periods.

[0013] The device comprises a plurality of pixels and a stacked first substrate and a second substrate, each of the plurality of pixels having a photodiode and a grayscale readout circuit that outputs grayscale information corresponding to the count value of the counter, the plurality of photodiodes of the plurality of pixels arranged on the first substrate, and the counter and the grayscale readout circuit may be arranged on the second substrate.

[0014] The event detection circuit may be provided for at least some of the pixels.

[0015] The counter operates as an up counter or a down counter for each frame, and includes a holding circuit that holds the count value in the frame in which the counter operates as the up counter. The counter may operate as the up counter in the first frame in which the count value is reset, and as the down counter in the second frame following the first frame, downcounting from the count value held in the holding circuit.

[0016] The counter includes a comparator that compares the down count value at the end of the second frame in which it operates as the down counter with the threshold value, and if the comparator determines that the down count value is less than the threshold value, the counter may operate as the down counter in the third frame following the second frame, down-counting from the count value held in the holding circuit.

[0017] The event detection circuit detects the event when the comparator determines that the count value is equal to or greater than the threshold, and the counter may reset the count value and operate as the up counter in the next frame.

[0018] Two or more pixels may share one comparator, and each of the two or more pixels may compare the count value of a counter provided for each pixel with the threshold in a time-division manner.

[0019] The circuit may include one that adds the count value of the counter on a frame-by-frame basis and outputs it as a grayscale value.

[0020] A correction processing circuit may be provided to correct the correspondence between the count value of the counter and the frequency of the incident photon.

[0021] The event detection circuit and the correction processing circuit may output at different frame rates.

[0022] comprising a plurality of pixels, and a first substrate and a second substrate that are stacked, wherein each of the plurality of pixels comprises the photodiode, the control circuit, the counter, the event detection circuit, a first readout circuit that outputs the event, and a second readout circuit that outputs gradation information according to a count value of the counter, the photodiode is disposed on the first substrate, and at least one of the control circuit, the counter, the event detection circuit, the first readout circuit, and the second readout circuit may be disposed on the second substrate.

[0023] comprising a third substrate stacked on the second substrate, wherein among the control circuit, the counter, the event detection circuit, the first readout circuit, and the second readout circuit, any component not disposed on the second substrate may be disposed on the third substrate.

[0024] comprising: a signal line connected to two or more pixels arranged in a column direction; a first holding circuit that holds the count value of the counter of each pixel transmitted through the signal line; a second holding circuit that holds the count value held in the first holding circuit when the event is detected; a subtraction processing circuit that detects a difference between the count value held in the first holding circuit from the signal line and the count value held in the second holding circuit when the count values of the counter of a plurality of pixels for one row arranged in a row direction are read out in parallel; and a comparator that compares the difference with the threshold, wherein when the comparator determines that the difference is equal to or greater than the threshold, the event detection circuit detects the event and causes the second holding circuit to hold the count value held in the first holding circuit, and when the comparator determines that the difference is less than the threshold, the event detection circuit may detect the event for a plurality of pixels of one column in a next row adjacent in the column direction.

[0025] wherein the counter is reset for each frame and performs an up-count operation every time a photon is detected by the photodiode, and the event detection circuit may detect the event for each frame based on a comparison result obtained by the comparator.

[0026] comprising an addition circuit that adds, for each row, the count values of the counter for each pixel transmitted via the signal line, wherein the first holding circuit holds the count value added by the addition circuit, and the event detection circuit detects the event for each of a plurality of pixels adjacent in a column direction; an addition circuit that overlaps and adds count values of an adjacent or surrounding pixel and the same pixel may be provided.

[0027] a circuit that adds the count values of the counter for each pixel transmitted via the signal line in units of frames and outputs a gradation value may be provided.

[0028] a correction processing circuit that corrects a correspondence relationship between the count value of the counter and the frequency of incident photons may be provided.

[0029] the event detection circuit and the gradation value output circuit may perform output at different frame rates.

[0030] comprising: a plurality of pixels; a first readout circuit that outputs the event; a second readout circuit that outputs gradation information according to the count value of the counter; and a stacked first substrate and second substrate, wherein each of the plurality of pixels includes the photodiode, the control circuit, and the counter, the event detection circuit, the first readout circuit, the first holding circuit, the second holding circuit, the subtraction processing circuit, and the comparator are shared by the plurality of pixels, the photodiode is disposed on the first substrate, and at least one of the control circuit, the counter, the event detection circuit, the first readout circuit, the first holding circuit, the second holding circuit, the subtraction processing circuit, and the comparator may be disposed on the second substrate.

[0031] comprising a third substrate stacked on the second substrate, wherein among the control circuit, the counter, the event detection circuit, the first readout circuit, and the second readout circuit, any component not disposed on the second substrate may be disposed on the third substrate.

[0032] A block diagram showing the main parts of the photodetector according to the first embodiment. A timing diagram of the up / down counter. A flowchart showing the event detection process of the photodetector according to the first embodiment. A diagram showing the relationship between the event determination result and the operation of the up / down counter in the next frame. A block diagram showing the main parts of the photodetector according to the second embodiment. A block diagram showing the main parts of the photodetector according to the third embodiment. A diagram showing the correspondence between the incident photon frequency and the count value. A diagram showing the shape of the curve after correction processing. A block diagram showing the main parts of the photodetector according to the fourth embodiment. A timing diagram showing the operation of the pixel circuit and column processing circuit according to the fourth embodiment. A diagram showing the relationship between the code output from the event detection circuit and the first readout circuit, the event determination result, and the control of the next frame. A block diagram showing the main parts of the photodetector according to the fifth embodiment. A flowchart showing the processing operation of the photodetector according to the fifth embodiment. A block diagram showing the main parts of the photodetector according to the sixth embodiment. A block diagram showing the main parts of the photodetector according to the seventh embodiment. A timing diagram of the photodetector according to the seventh embodiment. A circuit diagram of the pixel circuit in the photodetector according to the eighth embodiment. A timing diagram showing the processing operation of the pixel circuit in Figure 16. A circuit diagram of a pixel circuit in a photodetector according to the ninth embodiment. A layout diagram showing a first example in which the photodetectors according to the first to ninth embodiments are arranged on multiple substrates. A layout diagram showing a second example in which the photodetectors according to the first to ninth embodiments are arranged on a single substrate. A layout diagram of a third example having multiple types of pixels. A diagram showing an example of expanding the area of ​​the second substrate stacked on the first substrate. A diagram showing an example of providing a third substrate stacked on the second substrate. A diagram showing an example of multiple recharge cycles. A diagram showing the correspondence between reference brightness and count value. A diagram showing the correspondence between reference brightness and the difference in equation (5). A block diagram showing the main parts of the photodetector according to the thirteenth embodiment. A diagram showing the event detection timing. A diagram explaining the processing operation of the photodetector according to the thirteenth embodiment. A diagram explaining the processing operation of the photodetector according to the thirteenth embodiment. A flowchart showing the processing operation of the photodetector according to the thirteenth embodiment. A diagram explaining the processing operation of the low-power mode. A diagram explaining the processing operation of the normal mode. A timing diagram of the photodetector according to the thirteenth embodiment. A block diagram of the main parts of the photodetector according to the fourteenth embodiment.A diagram illustrating the processing operation of the 14th embodiment. A block diagram showing the main part of the photodetector according to the 15th embodiment. A diagram illustrating the processing operation of the photodetector according to the 15th embodiment. A diagram showing a first example of binning. A diagram showing a second example of binning. A diagram showing a third example of binning. A diagram showing a fourth example of binning. A diagram showing a fifth example of binning. A block diagram showing the main part of the photodetector according to the 17th embodiment. A diagram showing the processing operation of the photodetector according to the 17th embodiment. A block diagram showing the main part of the photodetector according to the 18th embodiment. A plan view of the pixel array section of the photodetector according to the 18th embodiment. A circuit diagram of a grayscale pixel. A circuit diagram of an EVS pixel. A block diagram showing the main part of the photodetector according to one modification of the 18th embodiment. A plan view of the pixel array section of the photodetector according to one modification of the 18th embodiment. A diagram showing an example of a two-layer stacked structure. A diagram showing an example of a three-layer stacked structure. A block diagram showing the main part of the photodetector according to the 20th embodiment. A diagram showing the pixel region to which the first threshold is applied and the pixel region to which the second threshold is applied. A timing diagram of the photodetector according to the 21st embodiment. A diagram illustrating the decimation process according to the 22nd embodiment. A diagram showing an example of decimation processing on a column-by-column basis. A diagram showing an example of decimation processing on a pixel-by-pixel basis. A diagram illustrating the processing operation of the photodetector according to the 23rd embodiment. A diagram showing a modified exposure period in low-power mode. A diagram showing a modified exposure period in normal mode. A block diagram of the main parts of the photodetector according to the 24th embodiment. A diagram showing the circuit operation in low-power mode. A diagram showing the circuit operation in normal mode. A block diagram of the main parts of the photodetector according to the 25th embodiment. A diagram showing the operation of two adjacent pixels in low-power mode. A diagram showing the operation of two adjacent pixels in normal mode. A block diagram showing an example of the schematic configuration of the vehicle control system. An explanatory diagram showing an example of the installation position of the external information detection unit and the imaging unit.

[0033] The embodiments of the photodetector will be described below with reference to the drawings. While the main components of the photodetector will be described below, there may be components and functions not shown or described in the drawings. The following description does not exclude any components or functions not shown or described.

[0034] (First Embodiment) Figure 1 is a block diagram showing the main parts of the light detection device 1 according to the first embodiment. As shown in Figure 1, the light detection device 1 according to the first embodiment includes a pixel circuit 2 and a pulse generation circuit 3.

[0035] The light detection device 1 according to the first embodiment includes a plurality of pixels arranged in the row and column directions, and each pixel has a pixel circuit 2 as shown in Figure 1. The pulse generation circuit 3 is shared, for example, by a plurality of pixel circuits 2 arranged in the row direction.

[0036] The pixel circuit 2 in Figure 1 includes a SPAD (Single Photon Avalanche Diode) 4, a control circuit 5, an up-down counter (UDC) 6, a holding circuit 7, a comparator 8, an event detection circuit 9, and a first readout circuit 10.

[0037] The SPAD4 is an avalanche photodiode that operates in Geiger mode and can respond to even a single photon incidence.

[0038] The control circuit 5 includes a PMOS transistor Q1 and an inverter IV. The source of the PMOS transistor Q1 is connected to the power supply voltage node, the drain is connected to the cathode of SPAD4 and the input node of inverter IV, and the gate is input to the clock signal CK1 output from the pulse generation circuit 3. When the clock signal CK1 output from the pulse generation circuit 3 transitions from high to low, the PMOS transistor Q1 turns on and enters a recharge state (second state) that raises the cathode voltage of SPAD4. When the clock signal CK1 transitions from low to high, the PMOS transistor Q1 turns off, and SPAD4 enters a standby state (first state) where photons can be detected. The control circuit 5 is not limited to this configuration, and any configuration that switches between the standby state and the recharge state and shapes the waveform according to the potential of the cathode or anode and outputs a pulse signal is acceptable.

[0039] As will be described later, the control circuit 5 may switch the recharge period of SPAD4 to two or more multiplication periods of the reference period. The control circuit 5 generates a pulse signal corresponding to the potential change of the cathode or anode of SPAD4.

[0040] When SPAD4 detects a photon, the cathode voltage of SPAD4 decreases, and the output node of inverter IV transitions from low to high.

[0041] The output node of inverter IV is connected to the clock terminal of up / down counter 6. When the output node of inverter IV transitions from low to high, the up / down counter 6 performs a counting operation.

[0042] The up / down counter 6 switches between up-counting and down-counting, for example, for each frame. The up / down counter 6 performs up-counting or down-counting in synchronization with the pulse signal generated by the control circuit 5. The up / down counter 6 distinguishes between photons incident during two or more periods of the recharge cycle and changes the count value accordingly.

[0043] The comparator 8 compares the difference between the up-count and down-count of the up-down counter 6 with a threshold value. The threshold value is input from outside the pixel circuit 2. As will be described later, the threshold value according to this embodiment is a constant value that does not change with illuminance.

[0044] The event detection circuit 9 detects events for each pixel based on the comparison result of the comparator 8. The event detection circuit 9 detects an event when the amount of change in brightness exceeds a threshold. The event detection circuit 9 may also detect a Positive (also called POS) event when the amount of change from low to high brightness (amount of change in brightness) exceeds a threshold, and a Negative (also called NEG) event when the amount of change from high to low brightness (amount of change in brightness) exceeds a threshold.

[0045] The first readout circuit 10 outputs the event detection result of the event detection circuit 9 to the event output line SL2 based on the readout selection control signal supplied via the readout selection control line SL1. The event output line SL2 extends in the column direction and is connected to multiple pixel circuits 2 in the same column. The readout selection control line SL1 is sometimes called the row selection line.

[0046] Figure 2 is a timing diagram of the up / down counter 6. In Figure 2, the horizontal axis represents time, and the vertical axis represents the count value. As shown in Figure 2, the up / down counter 6 switches its counting operation every frame. In the first frame, it performs an up-counting operation, and in the next frame, it performs a down-counting operation. In this way, the up / down counter 6 switches between up-counting and down-counting operation depending on whether or not an event has been detected.

[0047] Figure 3 is a flowchart showing the event detection process of the light detection device 1 according to the first embodiment. Figure 4 is a diagram showing the relationship between the event determination result and the operation of the up / down counter 6 in the next frame. The operation of the light detection device 1 according to the first embodiment will be described below based on Figures 2 to 4.

[0048] As described above, in the first frame (times t1 to t2 in Figure 2), the up / down counter 6 performs an up-count operation (step S1). The up-count value at the end of that frame is held in the holding circuit 7 (step S2).

[0049] In the next frame (for example, from time t2 to t3), the up / down counter 6 performs a down-count operation (step S3). The comparator 8 determines whether the down-count value at the end of that frame is greater than or equal to a threshold (step S4). Here, the down-count value is the difference between the count value of the current down-count operation and the count value at the end of the previous up-count operation.

[0050] If step S4 is NO, the up-count value held in the holding circuit 7 is preset to the up-down counter 6 (step S5), and in the next frame (for example, time t3 to t4), the processing from step S3 onwards is repeated. In this way, the down-count operation is repeated by presetting the count value at the end of the previous up-count until the down-count value exceeds the threshold.

[0051] If step S4 is YES, the event detection circuit 9 detects an event (step S6). Then, the up / down counter 6 resets its count value (step S7), and the process from step S1 onwards is repeated.

[0052] As shown in Figure 4, if an event is detected (step S4 is YES), the up / down counter 6 resets the count value and then performs an up-count operation. If no event is detected (step S4 is NO), the up / down counter 6 presets the count value and then performs a down-count operation.

[0053] In the optical detection device 1 according to the first embodiment, event detection is not performed during the period when the up-down counter 6 is performing an up-count operation. Event detection is performed only during the period when the up-down counter 6 is performing a down-count operation. Until an event is detected, the count value held in the holding circuit 7 is preset to the up-down counter 6 for each frame, and then event detection is performed.

[0054] The comparator 8 according to this embodiment compares the down count value with a constant threshold value regardless of illuminance. Brightness is the number of photons per unit time and is proportional to the photon rate, which is the number of photons per second.

[0055] In this specification, a constant contrast refers to a change in luminance by a fixed percentage relative to a reference luminance. When determining event detection based on constant contrast, it is desirable to use a constant threshold; however, since the threshold changes with illuminance, it is not possible to use a constant threshold regardless of illuminance.

[0056] In this invention, by adjusting and combining the settings of multiple recharge cycles, a relationship between count values ​​that has a logarithmic response to luminance is created, and the property that the threshold remains constant when the difference between count values ​​is taken is utilized. Equation (1) is an equation that shows the relationship between the reference luminance (reference luminance) fref, the count value Nref obtained at the reference luminance, the changed luminance fref × (1 + CT), the count value Ncmp obtained at the changed luminance, and the difference between count values ​​that has a logarithmic response to luminance. CT is the contrast change set as the threshold.

[0057] Nfef - Ncmp = αlog(fref) - αlog(fref × (1 + CT)) = αlog(1 / (1 + CT)) ... (1) α represents the slope of the logarithmic response curve. Equation (1) shows that the difference between the count value obtained at the reference brightness and the count value obtained at the changed brightness is a constant value independent of the reference brightness. Therefore, the comparator 8 according to this embodiment can perform a comparison with the down count value using a constant threshold regardless of the illuminance.

[0058] Thus, in the first embodiment, events can be detected by comparing a constant threshold value with the count value (downcount value) regardless of illuminance, thereby achieving area reduction and power saving for the pixel circuit 2. Furthermore, the event detection circuit 9 according to the first embodiment has the upcount value held in the holding circuit 7 preset in the up / down counter 6, and downcounts each time an event is detected. When the downcount value falls below the threshold value, an event is detected. This makes it possible to detect events when the brightness change is at a lower frequency.

[0059] (Second Embodiment) Figure 5 is a block diagram showing the main parts of the light detection device 1 according to the second embodiment. In the light detection circuit according to the second embodiment, a plurality of pixel circuits 2 share one comparator 8. Figure 5 shows an example in which two adjacent pixel circuits 2 in the column direction share one comparator 8. More specifically, the two pixel circuits 2 share the comparator 8, a first selection circuit 11, and a second selection circuit 12.

[0060] Each pixel circuit 2 includes a SPAD 4, a control circuit 5, an up / down counter (UDC) 6, a holding circuit 7, an event detection circuit 9, and a first readout circuit 10. Hereinafter, two pixel circuits 2 that share a comparator 8 will be referred to as the first pixel circuit 2a and the second pixel circuit 2b.

[0061] The first selection circuit 11 alternately selects either the count value of the up / down counter 6 in the first pixel circuit 2a or the count value of the up / down counter 6 in the second pixel circuit 2b. The comparator 8 compares the count value selected by the first selection circuit 11 with a threshold value.

[0062] The second selection circuit 12 time-division switches whether to input the comparison result output from the comparator 8 to the event detection circuit 9 in the first pixel circuit 2a or to the event detection circuit 9 in the second pixel circuit 2b.

[0063] The comparator 8 selects the count value of the first pixel circuit 2a or the count value of the second pixel circuit 2b in a time-division manner and compares it with a threshold value, and alternately inputs the comparison result to the event detection circuit 9 in the first pixel circuit 2a or the event detection circuit 9 in the second pixel circuit 2b.

[0064] Since the comparator 8 has a larger area than other circuit blocks in the pixel circuit 2, sharing one comparator 8 among multiple pixel circuits 2 makes it possible to reduce the area and power consumption of the pixel circuit 2.

[0065] (Third Embodiment) Figure 6 is a block diagram showing the main parts of the light detection device 1 according to the third embodiment. The pixel circuit 2 of the light detection circuit according to the third embodiment shown in Figure 6 has a second readout circuit 13 in addition to the configuration in Figure 1. The second readout circuit 13 outputs the count value or grayscale value of the up / down counter 6 to the signal line SL3. In this specification, the second readout circuit 13 may be referred to as the grayscale readout circuit.

[0066] Multiple pixel circuits 2 arranged in a column are connected to a common event output line SL2 and signal line SL3. Event detection results output from each pixel circuit 2 are transmitted via the event output line SL2. Count values ​​or grayscale values ​​output from each pixel circuit 2 are transmitted via the signal line SL3. The first readout circuit 10 and the second readout circuit 13 may output event information and grayscale information at different frame rates.

[0067] Since the count value of each pixel correlates with the grayscale value, the grayscale value of each pixel can be obtained from the count value without acquiring the difference. The reconstruction from the count value to the grayscale value may be performed outside the pixel circuit 2 or outside the light detection device 1.

[0068] Figure 7A shows the correspondence between the incident photon frequency and the count value. In Figure 7A, the horizontal axis represents the incident photon frequency, and the vertical axis represents the count value. The incident photon frequency is equivalent to the grayscale value. Ideally, as shown by the dashed line in Figure 7A, it is desirable for the incident photon frequency and the count value to have a linear relationship. However, in reality, as shown by the curve in Figure 7A, when the incident photon frequency is low, the relationship between the incident photon frequency and the count value is almost linear, but as the incident photon frequency increases, the count value becomes nonlinear and converges to the number of recharges and stops changing. Therefore, when the count value becomes larger than a certain size, it becomes impossible to accurately determine the incident photon frequency.

[0069] Therefore, the shape of the curve representing the correspondence between the incident photon frequency and the count value may be corrected by a correction process. Figure 7B shows the shape of the curve after the correction process. Through the correction process, even in the region of high incident photon frequencies, the gradation value, which is the incident photon frequency, can be accurately restored from the count value.

[0070] The correction process described above may be performed outside the pixel circuit 2 read from the second readout circuit 13, or outside the light detection device 1.

[0071] Thus, in the third embodiment, the count value of the up / down counter 6 is output from the pixel circuit 2 to the signal line SL3, so that the grayscale value can be restored from the count value while event detection is performed, and the event detection image and the grayscale image can be generated in parallel.

[0072] (Fourth Embodiment) Figure 8 is a block diagram showing the main parts of the light detection device 1 according to the fourth embodiment. The pixel circuit 2 of the light detection device 1 according to the fourth embodiment restores the grayscale value from the count value of the up / down counter 6 and outputs the restored grayscale value to the signal line SL3.

[0073] The internal configuration of the pixel circuit 2 in Figure 8 is the same as that in Figure 6, so its explanation is omitted. Multiple pixel circuits 2 arranged in the column direction are connected to a common signal line SL3. The count value output from each pixel circuit 2 to the signal line SL3 is input to the column processing circuit 21. The column processing circuit 21 has a selection circuit 22 and an adder 23. The frame processing circuit 29, which switches for each row read, has a count holding memory 24 and an image frame memory 25.

[0074] The selection circuit 22 selects the count holding memory 24 if the event code output from the event output line SL2 is 01, and selects the adder 23 otherwise.

[0075] The adder 23 performs the following operation as shown in equation (2): subtracting the difference between the up-count value Nup and the down-count value Ndn from the sum of the count value Mx and the up-count value Nup of the previous frame. The count-holding memory 24 holds the up-count value Nup.

[0076] The output of adder 23 = Mx + Nup - (Nup - Ndn) ... (2) The count value, which is the output of adder 23, is stored in the image frame memory 25.

[0077] Figure 9 is a timing diagram showing the operation of the pixel circuit 2 and column processing circuit 21 according to the fourth embodiment. Between times t1 and t2, the up-down counter 6 performs an up-counting operation. At time t2, the up-down counter 6 stops the up-counting operation. The count value (up-count value) N0 at time t2 is output from the second read circuit 13 at time t3. At time t4, the count holding memory 24 holds the up-count value in the holding circuit 7 of the pixel circuit 2.

[0078] During the frames from time t1 to t4, the up / down counter 6 performs an up-counting operation, so the comparator 8 does not perform a comparison operation. At time t3, the event detection circuit 9 and the first readout circuit 10 output the code "01" to indicate that it is an up-counting period. During the frame period from time t1 to t4, the image frame memory 25 stores the count value Mx of the previous frame.

[0079] At time t4, the next frame begins, and the up / down counter 6 performs a down-count operation each time the SPAD 4 detects a photon. At time t5, the down-count value is N1. Since N1 is below the threshold, the event detection circuit 9 and the first readout circuit 10 output the code "00" at time t6, instructing the up / down counter 6 to be preset. Upon receiving this code, at time t7, when the next frame begins, the count value of the up / down counter 6 is preset to the count value N0 held in the holdout circuit 7. At time t6, the count value D1 of the up / down counter 6 is output from the second readout circuit 13. At time t4, the read data is added to the image frame memory 25, and the count value Mx + N0 is stored.

[0080] At time t7, the next frame begins, and the up / down counter 6 performs a down-count operation each time the SPAD 4 detects a photon. At time t8, the down-count value is N2. Since N2 is above the threshold, the event detection circuit 9 and the first readout circuit 10 output code "11" at time t9, instructing the up / down counter 6 to be reset. Upon receiving this code, the count value of the up / down counter 6 is reset at time t10, when the next frame begins. Therefore, the up / down counter 6 performs an up-count operation from time t10 onward. At time t9, the count value D2 of the up / down counter 6 is output from the second readout circuit 13. At time t7, the count value Mx + N0 + N1 is stored in the image frame memory 25 by adding the count value N1 recovered from the adder 23.

[0081] Figure 10 shows an example of the code output from the event detection circuit 9 and the first readout circuit 10, the event determination result, and the relationship between these and the control of the next frame. As shown in Figure 10, code "11" indicates the detection of a Positive (POS) event, and code "10" indicates the detection of a Negative (NEG) event. In the case of these codes, in the next frame, the count value of the up / down counter 6 is reset, and the up / down counter 6 performs an up-counting operation. Code "00" indicates that no event was detected, and in the next frame, the count value of the up / down counter 6 is preset, and the up / down counter 6 performs a down-counting operation. Code "01" indicates an up-counting period, and no event detection is performed.

[0082] Thus, in the fourth embodiment, similar to the third embodiment, the count value can be output in parallel with event detection. Since the addition of the count value per frame is performed by the column processing circuit 21 and the frame processing circuit 29, the internal configuration of the pixel circuit 2 can be simplified.

[0083] (Fifth Embodiment) Figure 11 is a block diagram showing the main parts of the light detection device 1 according to the fifth embodiment. The light detection device 1 according to the fifth embodiment includes a pixel array section 20 in which a plurality of pixels are arranged in the row direction and column direction, a pulse generation circuit 3, a row control circuit 30, and a column processing circuit 21.

[0084] The row control circuit 30 is also present in the light detection device 1 according to the first to fourth embodiments, but it is not shown in the first to fourth embodiments. The row control circuit 30 controls, for example, the read selection control line (row control line) SL1. The row control line SL1 controls the operation of the counter 26, the holding circuit 7, the first read circuit 10, etc. of the pixel circuit 2.

[0085] The pixel circuit 2 according to the fifth embodiment shown in Figure 11 includes a SPAD 4, a control circuit 5, a counter 26, and a first readout circuit 10. The counter 26 is not an up / down counter, but for example, an up counter. The first readout circuit 10 outputs the count value of the counter 26. The counter 26 is reset each frame and performs an up-count operation each time a photon is detected by the SPAD 4.

[0086] Two or more pixel circuits 2 arranged in a column are connected to a signal line SL2, and the count value of each pixel circuit 2 is input to the column processing circuit 21 via the signal line SL2.

[0087] As shown in Figure 11, the pixel circuit 2 according to the fifth embodiment differs from the pixel circuit 2 according to the first to fourth embodiments in that it does not have a comparator 8 and an event detection circuit 9. Therefore, it is possible to reduce the area and power consumption of the pixel circuit 2. The pixel circuit 2 in Figure 11 counts the number of times the SPAD 4 detects a photon for each frame.

[0088] The column processing circuit 21 includes a count holding circuit 31, a subtraction processing circuit 32, a frame processing circuit 29, a comparator 8, and an event detection circuit 9. The frame processing circuit 29 includes a reference count frame memory 33.

[0089] The count holding circuit 31 holds the count value output from each pixel circuit 2 to the signal line SL3.

[0090] The subtraction circuit 32 detects the difference between the count value held by the count holding circuit 31 and the count value held in the reference count frame memory 33.

[0091] The reference count frame memory 33 stores the count value held in the count holding circuit 31 when the pixel circuit 2 detects an event.

[0092] The comparator 8 compares the difference detected by the subtraction circuit 32 with a threshold value.

[0093] The event detection circuit 9 detects an event when the comparator 8 determines that the difference is greater than or equal to a threshold, and stores the count value held by the count holding circuit 31 in the reference count frame memory 33. If the comparator 8 determines that the difference is less than the threshold, it detects events for multiple pixels in one column of the next row adjacent in the column direction. The event signal output from the event detection circuit 9 is output from the photodetector 1 via the output interface circuit 34.

[0094] Figure 12 is a flowchart showing the processing operation of the light detection device 1 according to the fifth embodiment. The count holding circuit 31 holds the count values ​​output from each pixel circuit 2 for each row (step S11). The count holding circuit 31 holds the count values ​​of each pixel in each row.

[0095] Next, the subtraction circuit 32 detects the difference between the count value held in the count holding circuit 31 and the count value held in the reference count frame memory 33 (step S12). The subtraction circuit 32 detects the above-mentioned difference for each pixel of each row.

[0096] Next, it is determined whether the difference is greater than or equal to a threshold (step S13). Here, the threshold is a constant threshold regardless of illuminance, as in the first to fourth embodiments. There may be two thresholds: one for detecting a positive event when illuminance increases, and another for detecting a negative event when illuminance decreases.

[0097] If step S13 is YES, the event detection circuit 9 detects an event (step S14) and updates the count value held by the reference count frame memory 33 to the count value held by the count holding circuit 31 (step S15). If step S13 is NO, it is determined that no event has been detected (step S16).

[0098] If the processing in step S15 or step S16 is completed, the count value of the next row is read (step S17), and the processing from step S11 onwards is repeated.

[0099] Thus, in the fifth embodiment, since event detection is performed by the column processing circuit 21, the pixel circuit 2 can be made smaller in area and more power-efficient. Furthermore, since the column processing circuit 21 performs event detection by comparing the difference between the new count value and the count value at the time of the previous event detection with a threshold, a constant threshold can be used regardless of the illuminance, enabling event detection at a constant contrast.

[0100] (Sixth Embodiment) Figure 13 is a block diagram showing the main parts of the photodetector 1 according to the sixth embodiment. The photodetector 1 according to the sixth embodiment shown in Figure 13 has a different internal configuration of the column processing circuit 21 compared to Figure 11.

[0101] The column processing circuit 21 in Figure 13 has an adder circuit 35 in addition to the configuration of the column processing circuit 21 in Figure 11. The adder circuit 35 adds the count values ​​output from each pixel circuit 2 to the signal line SL2 row by row. For example, the adder circuit 35 counts the count values ​​for multiple adjacent pixel rows. The number of pixels to be added may be n × n pixels or more, and the addition process may be performed so that the pixel count values ​​overlap.

[0102] The count holding circuit 31 holds the count value added by the addition circuit 35. The processing from the subtraction circuit 32 onward is the same as in Figures 11 and 12.

[0103] As described above, the light detection device 1 according to the sixth embodiment detects events in units of multiple pixel rows. This eliminates the need for the count holding circuit 31 and the reference count frame memory 33 to store count values ​​for each pixel row, thereby reducing memory capacity. Furthermore, since the count values ​​of multiple pixel rows are added together and then compared with a threshold, event detection in low light conditions can be performed with high accuracy. Additionally, by performing the addition process in an overlapping manner, it is possible to increase the resolution of event detection.

[0104] (Seventh Embodiment) Figure 14 is a block diagram showing the main parts of the photodetector 1 according to the seventh embodiment. The photodetector 1 according to the seventh embodiment shown in Figure 14 has a different internal configuration of the column processing circuit 21 compared to Figures 11 and 13.

[0105] The frame processing circuit 29 in Figure 14 has a grayscale value frame memory 37 in addition to the configuration of the frame processing circuit 29 in Figure 11. The grayscale value frame memory 37 adds the count values ​​output from each pixel circuit 2 to the signal line SL2 for each pixel on a frame basis. The count values ​​output from the second readout circuit 13 are output from the light detection device 1 via the output interface circuit 36. The output interface circuit 34 and the output interface circuit 36 ​​may output at different frame rates. This allows the event detection circuit 9 and the second readout circuit 13 to output event information and grayscale information at different frame rates.

[0106] As described above, the count value is correlated with the incident photon frequency, but the relationship between the incident photon frequency and the count value is not necessarily linear, and especially as the incident photon frequency increases, the count value saturates and stops changing. Therefore, the second readout circuit 13 may perform a correction process to bring the count value closer to the incident photon frequency corresponding to the count value before outputting the count value or the grayscale value.

[0107] Figure 15 is an example of a timing diagram of the photodetector 1 according to the seventh embodiment. In the frame starting at time t1, the holding circuit 7 in the pixel circuit 2 holds the count value N-1 from the previous frame. This count value N-1 is held in the count holding circuit 31. The reference frame memory also holds the count value Nx from when the most recent event was detected. The counter 26 in the pixel circuit 2 counts up from time t1 each time the SPAD 4 detects a photon.

[0108] At time t2, while the counter 26 is counting up, the subtraction circuit 32 detects the difference (N-1 - Nx) between the count value N-1 held by the count holding circuit 31 and the count value Nx held by the reference count frame memory 33. Since this difference is less than the threshold, the event detection circuit 9 does not detect an event. The grayscale value frame memory 37 stores the count value (Mx-1 + N-1) obtained by adding the count value N-1 held by the count holding circuit 31 to the count value Mx-1 in the previous frame.

[0109] The pixel circuit 2's holding circuit 7 and count holding circuit 31 hold the count value N0 immediately before the frame switch time t3. The reference count frame memory 33 also holds the count value Nx that the count holding circuit 31 held before time t3. After a series of row operations is completed, processing for the next row begins, and the addresses of the reference count frame memory 33 and the grayscale value frame memory are switched.

[0110] During the processing of the same line in the next frame, at time t4 while counter 26 is counting up, subtraction circuit 32 outputs the difference (N0 - N-1) between the count value N0 held by count holding circuit 31 and the count value N-1 held by reference count frame memory 33. In this example, this difference was determined to be greater than or equal to the Positive threshold, so event detection circuit 9 holds a Positive (POS) event. The count value in reference count frame memory 33 is updated at time t5 to the count value N0 held by count holding circuit 31.

[0111] The grayscale value frame memory 37 stores a count value (Mx + N0) at time t3, which is obtained by adding the count value N0 held by the count holding circuit 31 to the count value Mx = Mx - 1 + N - 1 from the previous frame.

[0112] The pixel circuit 2's holding circuit 7 and count holding circuit 31 hold the count value N1 immediately before the frame switch time t6. The reference count frame memory 33 also holds the count value N0 that the count holding circuit 31 held before time t6. After completing a series of row processing operations, the processing of the next row begins, and the addresses of the reference count frame memory and the grayscale value frame memory are switched.

[0113] Figure 15 shows an example where the brightness increases at time t6. In this case, the counter 26 counts up at a faster rate than before time t4.

[0114] At time t7, while the counter 26 is counting up, the subtraction circuit 32 detects the difference (N1 - N0) between the count value N1 held by the count holding circuit 31 and the count value N0 held by the reference count frame memory 33. Since this difference is less than the threshold, the event detection circuit 9 does not detect an event. The grayscale value frame memory 37 stores the count value (Mx + N0 + N1) obtained by adding the count value N1 held by the count holding circuit 31 to the count value Mx + N0 from the previous frame.

[0115] The pixel circuit 2's holding circuit 7 and count holding circuit 31 hold the count value N2 immediately before the frame switch time t8. The reference count frame memory 33 also holds the count value N0 that the count holding circuit 31 held before time t8.

[0116] At time t9, while counter 26 is counting up, the subtraction circuit 32 detects the difference (N2 - N0) between the count value N2 held by the count holding circuit 31 and the count value N0 held by the reference count frame memory 33. Since this difference is greater than or equal to the Positive threshold, the event detection circuit 9 detects a Positive (POS) event and stores the result. The count value in the reference count frame memory 33 is updated at time t10 to the count value N2 held by the count holding circuit 31.

[0117] The second readout circuit 13, at time t8, stores a count value (Mx + N0 + N1 + N2) obtained by adding the count value N2 held by the count holding circuit 31 to the count value Mx + N0 + N1 from the previous frame.

[0118] Thus, in the seventh embodiment, by adding a frame-by-frame count value in parallel with event detection to maintain grayscale value information, it is possible to output at a slower rate than the frame rate of the event detection output while maintaining a high SNR, thereby enabling the generation of an event detection image and a grayscale image for the same pixel for each frame.

[0119] (Eighth Embodiment) Figure 16 is a circuit diagram of the pixel circuit 2 in the photodetector 1 according to the eighth embodiment. The photodetector 1 according to the eighth embodiment is similar to the first to seventh embodiments in that it counts when photons are incident on it and output pulses. The pixel circuit 2 in Figure 16 is also applicable to the photodetector 1 according to the first to seventh embodiments.

[0120] The pixel circuit 2 in Figure 16 differs from the pixel circuit 2 in Figure 11 in the configuration of the control circuit 5. Furthermore, the pulse generation circuit 3 outputs two types of clock signals, CK1 and CK2. The pixel circuit 2 uses the clock signals CK1 and CK2 to count the number of photons detected by the SPAD 4.

[0121] The control circuit 5 in Figure 16 has a NAND gate G1 that outputs a negative logical AND signal of the output node of inverter IV and the clock signal CK2 output from pulse generation circuit 3. The counter 26 performs up-counting or down-counting operations in synchronization with the pulse signal output from NAND gate G1.

[0122] The counter 26 distinguishes and counts photons incident during the first period and photons incident during the second period, which is longer than the first period. The control circuit 5 sets the recharge period of the SPAD 4 to the sum of the first and second periods. For example, if a photon is incident during the first period, the counter 26 updates its count value twice, and if a photon is incident during the second period, it updates its count value once.

[0123] Figure 17 is a timing diagram showing the processing operation of the pixel circuit 2 in Figure 16. The clock signals CK1 and CK2 output from the pulse generation circuit 3 are clock signals with changing frequencies. The arrows in Figure 7 indicate the photon incidence timing. Because the frequencies of the clock signals CK1 and CK2 change, the pixel circuit 2 has two cases: one in which it counts the number of photons detected by the SPAD4 separately for count periods τ0 and τ1 and recharges the SPAD4 in a recharge period (τ0 + τ1); and another in which it counts the number of photons detected by the SPAD4 separately for count periods τ2 and τ3 and recharges the SPAD4 in a recharge period (τ2 + τ3).

[0124] By varying the lengths of periods τ0 to τ3, the dynamic range can be widened. Furthermore, by performing two or more counting operations per recharge cycle, the number of recharges can be reduced, leading to power savings.

[0125] The count value Ncount for a frame that includes the recharge period (τ0 + τ1) can be expressed by the following equation (3), using the count value Nreg at the recharge period (τ0 + τ1).

[0126] Ncount = Nreg((1 - e - fτ0) + (1 - e - f(τ0 + τ1))) ... (3) In this way, in the eighth embodiment, the dynamic range can be widened by providing multiple count periods and recharge periods of different lengths.

[0127] Alternatively, by inputting multiple pulses from CK2 while CK1 recharges once, photons incident during two or more periods can be counted separately. For example, if one recharge period is divided into n periods, the photon incidence timings would be n, n-1, ..., 1, starting from the earliest, and no photons would be counted if none were incident.

[0128] (Ninth Embodiment) Figure 18 is a circuit diagram of the pixel circuit 2 in the photodetector 1 according to the ninth embodiment. The photodetector 1 according to the ninth embodiment is similar to the first to eighth embodiments in that it can detect photons as pulses by causing avalanche multiplication when incident on them. The pixel circuit 2 in Figure 18 is also applicable to the photodetector 1 according to the fifth to seventh embodiments. Furthermore, by adding a clock signal CK2, it is also applicable to the eighth embodiment.

[0129] The pixel circuit 2 in Figure 18 differs from the pixel circuit 2 in the first to eighth embodiments in the configuration of the control circuit 5. Also, the cathode of the avalanche diode is connected to a high voltage that puts it into Geiger mode. The control circuit 5 in Figure 18 has an NMOS transistor Q2 and an inverter IV. The drain of the NMOS transistor Q2 is connected to the anode of SPAD4 and the input node of inverter IV, the source is connected to the ground voltage node, and the gate is input to the clock signal CK1 from the pulse generation circuit 3.

[0130] Thus, in the first to eighth embodiments, the control circuit 5 in the pixel circuit 2 uses an NMOS transistor Q2 instead of a PMOS transistor Q1, connects the anode of SPAD4 and the input node of inverter IV to the drain of the NMOS transistor Q2, and can detect pulses using the anode.

[0131] (Tenth Embodiment) The photodetector 1 according to the first to ninth embodiments may be arranged on a single substrate, or it may be arranged on multiple stacked substrates. This makes it possible to miniaturize each substrate.

[0132] Figure 19A is a layout diagram showing a first example in which the photodetector 1 according to the first to ninth embodiments is arranged on multiple substrates. The number of substrates stacked is not limited, but below, an example in which the photodetector 1 is arranged on two substrates (hereinafter referred to as the first substrate SB1 and the second substrate SB2) will be mainly described. In the first example, multiple SPAD4 having multiple pixels are arranged on the first substrate SB1, and pixel circuits 2 other than the SPAD4 and sub-pixel circuits 27 such as column processing circuits 21 are arranged on the second substrate SB2.

[0133] Figure 19B is a layout diagram showing a second example in which the photodetector 1 according to the first to ninth embodiments is arranged on a single substrate SB. In the second example, the SPAD4 is placed in a first region AR1 of the substrate SB, and the pixel circuits 2 other than the SPAD4 and the column processing circuit 21, etc., are placed in a second region AR2 of the same substrate SB.

[0134] Figure 19C is a layout diagram of a third example having multiple types of pixels. Figure 19C is a layout diagram of the first substrate SB1 and the second substrate SB2 in a plan view. The first substrate SB1 has SPAD4s for each of the multiple types of pixels. The multiple types of pixels include, for example, a pixel PX1 that performs event detection, a pixel PX2 that performs photon counting, and a pixel PX3 that acquires depth information. The second substrate SB2 is provided with a region where a pixel circuit 2 and other under-pixel circuits for the event detection pixel PX1 are located directly below the SPAD4 of the photon counting pixel PX2, and a region where a pixel circuit 2 and other under-pixel circuits for the photon counting pixel PX3 are located directly below the SPAD4 of the photon counting pixel PX3.

[0135] Thus, in the tenth embodiment, each pixel can be arranged on multiple substrates, or on the same substrate in divided regions, or multiple types of pixels can be mixed together.

[0136] (Eleventh Embodiment) In the photodetector 1 according to the fifth to eighth embodiments, at least a portion of the column processing circuit 21, such as a reference count frame memory 33 or a grayscale value frame memory 37, which have a large area, may be arranged by expanding the area of ​​the second substrate SB2, or they may be arranged on a third substrate stacked on the second substrate SB2.

[0137] Figure 20A shows an example of expanding the area of ​​the second substrate SB2 which is laminated on the first substrate SB1. A count holding circuit 31 or a reference count frame memory 33, etc., is arranged on the expanded portion 28 of the second substrate SB2. In this case, the first substrate SB1 may be made wider than the light-receiving portion of the SPAD and bonded together with SB2 using the same chip size.

[0138] Figure 20B shows an example in which a third substrate SB3 is stacked on the second substrate SB2. The third substrate SB3 is equipped with a count holding circuit 31 or a reference count frame memory 33, etc.

[0139] Thus, in the 11th embodiment, by providing a separate area for arranging some circuit blocks with a large area, such as frame memory, the frame memory and the like can be arranged without affecting other circuit blocks.

[0140] (Twelfth embodiment) By setting multiple recharge cycles and performing event detection, a constant threshold can be used regardless of illuminance.

[0141] Figure 21 shows an example of multiple recharge periods. Figure 21 shows an example where the recharge period τi is repeated Ni times, with recharge period τ1 being repeated N1 times followed by recharge period τ2 being repeated N2 times. As shown in Figure 21, the recharge period can be controlled by the frequency of the clock signal CK1 output from the pulse generation circuit 3. Note that the recharge period may also be controlled by combining multiple clock signals. Furthermore, the order of long periods, short periods, etc., does not matter.

[0142] The count value Ncount of the counter (for example, the up / down counter 6 in Figure 1) that counts the pulse signals output from SPAD4 is expressed by the following equation (4), due to the characteristics of photon incidence following a Poisson process. Note that equation (4) assumes that the recharge period is varied in n ways, as shown in Figure 21. In equation (4), Nrcg is the number of recharge periods, fin is the photon rate, τmin is the shortest recharge period, and γ is the common ratio of the recharge periods.

[0143]

[0144] The difference between the counter's reference count value Nref and the count value Ncmp after the brightness change is expressed by the following equation (5). CT is the contrast change.

[0145]

[0146] As shown in equation (5), the difference in count values ​​is independent of the reference brightness. Therefore, this difference can be compared with a certain threshold. In this embodiment, events are detected by comparing the difference in equation (5) with a certain threshold.

[0147] Figure 22 shows the correspondence between reference luminance and count value. The horizontal axis of Figure 22 is the logarithm of the photon rate corresponding to the reference illuminance. The vertical axis is the count value. Figure 22 shows curve w1 for reference luminance and curve w2 for luminance that is, for example, 1.3 times the reference luminance. Curves w1 and w2 have a region that can be represented by a straight line on the graph and have a logarithmic response. This linear region is the event detection range. Within the event detection range, the photon rate can be uniquely identified from the count value.

[0148] Figure 23 shows the correspondence between the reference brightness and the difference in equation (5). The horizontal axis of Figure 23 is the logarithm of the photon rate corresponding to the reference brightness. The vertical axis is the count difference. Curve w3 in Figure 23 shows the correspondence between the count value at the reference brightness and the count value at a brightness 1.3 times the reference brightness. In this case, the area labeled as the event detection range is represented by a constant count difference value, but in other areas, the difference from the threshold shown by the dashed line is small, so no events are detected.

[0149] By combining multiple types of recharge cycles in this way, event detection can be performed using a constant threshold regardless of illuminance, and the dynamic range can be expanded.

[0150] (Third Embodiment) The optical detection device 1 according to the thirteenth embodiment detects events in a power-saving manner by reducing the power required for recharging while maintaining the event determination count.

[0151] Figure 24 is a block diagram showing the main parts of the light detection device 1 according to the thirteenth embodiment. The light detection device 1 according to the thirteenth embodiment shown in Figure 24 includes an event determination unit 38 in addition to the configuration of Figure 13, an addition processing circuit 39 instead of the addition circuit 35 in Figure 13, and a control unit 40 instead of the row control circuit 5 in Figure 13.

[0152] The addition circuit 39 in Figure 24 switches whether or not to perform binning processing for each set of pixels based on the frequency of events. Here, binning processing refers to the process of adding the number of events for each group of pixels containing multiple pixels. Binning processing is repeated for each frame.

[0153] The event determination unit 38 selects either the low-power mode or the normal mode based on the number of events detected by the event detection circuit 9. As described later, the low-power mode is a mode in which the number of events is counted by binning processing with a slow recharge cycle. The normal mode is a mode in which the number of events is counted for each pixel with a fast recharge cycle. In this specification, the low-power mode may be referred to as the first mode, and the normal mode as the second mode. Note that a slow recharge cycle means that the recharge cycle is long, and a fast recharge cycle means that the recharge cycle is long.

[0154] The control unit 40 controls the driving timing of each pixel in the pixel array 20 depending on whether it is in low-power mode or normal mode. The pulse generation circuit 3 generates a pulse signal synchronized with the recharge period based on the control signal from the control unit 40 and supplies it to each pixel.

[0155] The control unit 40 may select a mode other than the low-power mode or the normal mode. In this case, the control unit 40 selects one of several modes in which the period for recharging the photodiode and the pixel area for detecting events differ, based on the event detection result by the event detection circuit 9.

[0156] Figure 25 shows the event detection timing. The pulse generation circuit 3 generates a pulse signal CK1 synchronized with the recharge period. As mentioned above, the recharge period differs between the low-power mode and the normal mode. Figure 25 shows an example in which the pulse generation circuit 3 generates a pulse signal CK1 that temporarily transitions from a high level to a low level at the start timing of the recharge period.

[0157] When a photon is detected, the cathode voltage Vk of SPAD4 decreases, and the output signal OUT of the control circuit 5 transitions from a high level to a low level. As a result, the counter 26 updates its count value.

[0158] Figures 26A and 26B illustrate the processing operation of the optical detection device 1 according to the thirteenth embodiment. As shown in Figure 26A, when the number of detected events is less than or equal to a predetermined value, the low-power mode is selected. In the low-power mode, binning is performed with a slow recharge cycle to count the number of events (state ST1). As a result, event detection is performed for each group of pixels to be binned, and events are acquired at a low resolution (state ST2). Figure 26A shows an example in which events are detected by separating them into positive events and negative events for each group of pixels, but events may be detected without distinguishing between polarity.

[0159] On the other hand, as shown in Figure 26B, if the number of detected events exceeds a predetermined value, the normal mode is selected. In normal mode, the number of events is counted for each pixel with a high-speed recharge cycle (state ST3). This allows events to be acquired at high resolution (state ST4).

[0160] Figure 27 is a flowchart showing the processing operation of the light detection device 1 according to the thirteenth embodiment. First, the reference count value held in the reference count frame memory 33 is acquired (step S21).

[0161] Next, exposure is started, and the number of events is counted for each group of pixels to be binned using a slow recharge cycle for the binning process (step S22). Thus, at the stage when the process shown in Figure 27 is started, the low-power mode is selected, and the binning process is performed with a slow recharge cycle to count the number of events.

[0162] After exposure is complete, the count value of the number of events counted for each pixel group counted in step S22 is read out (step S23). Next, for each pixel group to be binned, the count value of the number of events detected in each pixel included in the pixel group is added (step S24). The subtraction circuit 32 outputs the difference between the count value added in step S24 and the reference count value (step S25).

[0163] Next, it is determined whether the difference is greater than or equal to a predetermined threshold (step S26). If step S26 is YES, it is determined that an event has occurred (step S27). In step S27, if there are both positive and negative events, it is determined that either a positive or negative event has occurred.

[0164] Next, the reference count value is updated to the count value added in step S24 (step S28).

[0165] On the other hand, if step S26 is NO, it is determined that no event occurred (step S29).

[0166] When the processing in step S28 or S29 is completed, it is determined whether the processing has been repeated a predetermined number of times N (step S30). Since the processing is performed for each frame, in step S30 it is determined whether the processing has been repeated for N frames.

[0167] If the number of repetitions has not reached N, it is determined whether the number of detected events is greater than or equal to a predetermined value (step S31). In step S31, if the difference in step S26 is determined to be greater than or equal to a predetermined threshold, it is determined to be YES, and if the difference in step S26 is determined to be less than the predetermined threshold, it is determined to be NO.

[0168] If step S31 is NO, the low-power mode is selected and the process from step S22 onwards is repeated.

[0169] If step S31 is YES, the normal mode is selected, and the number of events for each pixel is counted without binning processing, with a faster recharge cycle than the low-power mode (step S32).

[0170] After exposure is complete, the count value for each pixel group counted in step S32 is read out (step S33). Then, the processing from step S25 onwards is carried out.

[0171] Figure 28A illustrates the processing operation in low-power mode, and Figure 28B illustrates the processing operation in normal mode.

[0172] In low-power mode, as shown in Figure 28A, binning is performed with a slow recharge cycle to count the number of events. In this case, since the number of events is counted for each pixel group, a low-resolution event detection result is obtained. In low-power mode, grayscale images are not output.

[0173] On the other hand, in normal mode, as shown in Figure 28B, the number of events is counted for each pixel with a high charge period. In this case, high-resolution event detection results can be obtained. In normal mode, a high-resolution grayscale image can be generated based on the number of events per pixel.

[0174] Figure 29 is a timing diagram of the light detection device 1 according to the 13th embodiment. The time interval t1 to t2 is one frame period. In the following, during the one frame period from time t1 to t2, the number of events is counted for each pixel group in low-power mode.

[0175] Assume that the number of events detected for each pixel group during the period from time t1 to t2 is less than a predetermined threshold. In this case, the low-power mode is subsequently selected for the frame from time t2 to t3. For example, if the pixel group to be binned contains 4 pixels, the number of events K for each pixel included in each pixel group is calculated for each pixel row. ０ , L ０ M ０ , N ０ The number of events N after adding up the values. ０ The value is held in the count holding circuit 31.

[0176] The subtraction circuit 32 counts the number of events N held in the count holding circuit 31. ０' and the number of events Nx stored in the reference count frame memory 33. The event detection circuit 9 determines that an event (for example, a Negative event) has occurred based on this difference. Accordingly, the reference count frame memory 33 is updated to store the number of events N held in the count holding circuit 31 ０ ' (at time t3).

[0177] In the frames from time t3 to t4, since it has been determined that an event occurred in the immediately preceding frame, the number of events is counted for each pixel at a high-speed recharge period without performing binning processing. The count holding circuit 31 holds the count value N for each pixel １ .

[0178] At time t4, the subtraction processing circuit 32 outputs the difference between the count value N held by the count holding circuit 31 １ and the number of events N stored in the reference count frame memory 33 ０ '. The event detection circuit 9 determines that an event (for example, a Positive event) has occurred based on this difference. Accordingly, the reference count frame memory 33 is updated to store the number of events N held in the count holding circuit 31 １ (at time t6).

[0179] In the frames from time t6 to t9, since it has been determined that an event occurred in the immediately preceding frame, the number of events is counted for each pixel at a high-speed recharge period without performing binning processing. At time t6, the count holding circuit 31 holds the number of events N counted for each pixel ２ .

[0180] At time t7, the subtraction processing circuit 32 outputs the difference between the number of events N counted for each pixel ２ and the number of events N stored in the reference count frame memory 33 １ . The event detection circuit 9 determines that no event has occurred based on this difference. Therefore, the reference count frame memory 33 is not updated.

[0181] Power consumption P of each pixel ＳＰＡＤThis can be expressed by the following equation (6).

[0182] P ＳＰＡＤ = N ｐｉｘ ・N ｒｃｇ ・C ｋ ・V ｅｘ ・V ｂｄ f ｐｓ …(6) In equation (6), N ｐｉｘ is the number of pixels, N ｒｃｇ This is the number of SPAD4 recharges per frame, C ｋ V is the cathode capacitance. ｅｘ V is the excess voltage. ｂｄ is the breakdown voltage, f ｐｓ This is the frame frequency.

[0183] As can be seen from equation (6), by using a low-speed recharge period, N ｒｃｇ P becomes smaller ＳＰＡＤ This can reduce f ｐｓ P becomes smaller ＳＰＡＤ This can reduce the number of pixels. Also, by decimating at least one of the pixel rows or pixel columns, N ｐｉｘ P becomes smaller ＳＰＡＤ It can be reduced.

[0184] Thus, in the 13th embodiment, it is determined whether the number of events detected for each frame is greater than or equal to a predetermined value. If fewer than the predetermined value is detected, binning is performed with a slow recharge cycle, thereby reducing power consumption. Furthermore, if the number of detected events is greater than or equal to the predetermined value, event detection is performed for each pixel with a fast recharge cycle, enabling high-resolution event detection and the generation of high-resolution images.

[0185] (14th Embodiment) Figure 30 is a block diagram of the main parts of the light detection device 1 according to the 14th embodiment. In the light detection device 1 according to the 14th embodiment shown in Figure 30, a control line CL1 for controlling the recharge period is arranged for each pixel row. A control signal CK1 transmitted by each control line CL1 is generated by the control unit 40. Multiple control lines CL1 are connected to a pulse generation circuit 3.

[0186] The light detection device 1 according to the 14th embodiment has a block configuration similar to that shown in Figure 24. In the 14th embodiment, the pulse generation circuit 3 provides each pixel row with a different timing control signal to individually switch the recharge period for each pixel row.

[0187] The light detection device 1 according to the 14th embodiment performs processing operations according to a flowchart similar to that in Figure 27.

[0188] Figure 31 is a diagram illustrating the processing operation of the 14th embodiment. In step S31 of Figure 27, if it is determined that the number of events is less than a predetermined value (state ST11), a control signal from the pulse generation circuit 3 is used to perform binning processing on only a portion of the pixel sequences at a slow recharge cycle to count the number of events (state ST12).

[0189] On the other hand, if it is determined in step S31 of Figure 27 that the number of events is equal to or greater than a predetermined value (state ST13), the number of events is counted for each pixel in only a portion of the pixel sequences at a high recharge cycle (state ST14).

[0190] Thus, in the 14th embodiment, it is possible to control whether to perform binning processing with a slow recharge cycle to count the number of events for each pixel row, or to count the number of events for each pixel with a fast recharge cycle.

[0191] (15th Embodiment) Figure 32 is a block diagram showing the main part of the light detection device 1 according to the 15th embodiment. In the 15th embodiment, the pulse generation circuit 3 provides a first control signal CK1 with a different timing for each pixel row, and the control unit 40 provides a second control signal CK2 with a different timing for each pixel row. The first control signal CK1 and the second control signal CK2 are generated by the control unit 40. The pulse generation circuit 3 transmits a plurality of first control signals CK1 via a plurality of first control lines CL1 arranged for each pixel row of the pixel array unit 20, and transmits a plurality of second control signals CK2 via a plurality of second control lines CL2.

[0192] As shown in Figure 32, each pixel has a multiplexer 41 connected to the gate of a PMOS transistor Q1 in the control circuit 5, and a NAND gate 42.

[0193] The NAND gate 42 outputs a low signal when both the row control line SL1 and the column control line SL2 are selected. The multiplexer 41 selects one of the first and second control signals CK1 and CK2 from the pulse generation circuit 3 based on the output signal of the NAND gate 42. The first control signal CK1 is a low-speed clock signal, while the second control signal CK2 is a high-speed clock signal.

[0194] By controlling the row control line SL1, the column control line SL2, and the first and second control signals CK1 and CK2, it is possible to switch between selecting a high-speed recharge period for each pixel row, each pixel column, or each pixel.

[0195] Figure 33 is a diagram illustrating the processing operation of the light detection device 1 according to the 15th embodiment.

[0196] First, binning is performed with a slow recharge cycle (state ST21) to detect events at low resolution (state ST22). Next, for the pixel region where an event occurred, events are detected for each pixel with a high recharge cycle (state ST23). This makes it possible to acquire event information for pixel regions of any position and size in the pixel array 20 (state ST24).

[0197] (16th Embodiment) There are various variations in the binning process performed in low-power mode.

[0198] Figure 34A shows a first example of binning. In the first example, binning is performed on each pixel group, including adjacent 2x2 pixels, and the two adjacent pixel groups are arranged so that they do not overlap. As a result, the resolution is reduced to 1 / 4 of the original resolution by performing binning. Figure 34A shows an example of detecting event information such as Positive (POS) Event, Negative (NEG) Event, or no Event for each pixel group.

[0199] Figure 34B shows a second example of binning. In this second example, binning is performed on each pixel group, including adjacent 3x3 pixels, and the two adjacent pixel groups are arranged so that they do not overlap. As a result, the resolution is reduced to 1 / 9 of the original resolution by performing binning.

[0200] Figure 34C shows a third example of binning. In the third example, binning is performed on each pixel group containing adjacent 2x2 pixels, and the pixels are arranged so that two adjacent pixel groups in the row and column directions overlap by two pixels each. This allows binning to be performed without reducing the resolution.

[0201] Figure 34D shows a fourth example of binning. In the fourth example, binning is performed on each pixel group containing adjacent 3x3 pixels, so that two adjacent pixel groups in the row, column, and diagonal directions overlap by 6 or 4 pixels. This allows binning to be performed without reducing resolution.

[0202] Figure 34E shows a fifth example of binning. The binning process in the fifth example counts the number of events for pixels of a specific color for each pixel group. In the fifth example in Figure 34E, the number of events for green pixels is counted for each 10x10 pixel group.

[0203] (17th Embodiment) Figure 35 is a block diagram of the main part of the light detection device 1 according to the 17th embodiment. The pixel array section 20 of the light detection device 1 according to the 17th embodiment has a plurality of grayscale pixels arranged in a first direction (e.g., row direction) and a second direction (e.g., column direction). The grayscale pixels have a SPAD 4 for photon detection, a control circuit 5, a counter 26, a holding circuit 7, and a first readout circuit 10. Each grayscale pixel has a corresponding color element of a color filter.

[0204] Figure 36 shows the processing operation of the light detection device 1 according to the 17th embodiment. First, binning is performed on the entire pixel array 20 with a low recharge cycle (state ST31). This allows events to be acquired at a low resolution (state ST32).

[0205] Next, for the pixel regions where an event is detected, events are detected pixel by pixel with a high-speed recharge cycle (state ST33). As a result, events are acquired at high resolution only for the pixel regions where an event is detected (state ST34), and in parallel, a high-resolution grayscale image is generated for the pixel regions where an event is detected (state ST35).

[0206] (Eighteenth Embodiment) Figure 37 is a block diagram of the main part of the light detection device 1 according to the eighteenth embodiment, and Figure 38 is a plan view of the pixel array section 20 of the light detection device 1 according to the eighteenth embodiment.

[0207] As shown in Figures 37 and 38, the pixel array 20 of the 18th embodiment has a mixture of EVS pixels and grayscale pixels. EVS pixels are pixels that detect events and do not output grayscale information. Grayscale pixels are pixels that output grayscale information and do not detect events.

[0208] Figure 39A is a circuit diagram of a grayscale pixel, and Figure 39B is a circuit diagram of an EVS pixel. Both the grayscale pixel in Figure 39A and the EVS pixel in Figure 39B have a SPAD 4, a control circuit 5, a counter 26, a holding circuit 7, and a first readout circuit 10. In addition, the EVS pixel in Figure 39B has a NAND gate 42 and a multiplexer 41, similar to Figure 32. The color elements of a color filter are arranged in the grayscale pixel. Grayscale pixels are also called RGB pixels.

[0209] As shown in Figure 39B, the EVS pixels select whether to detect events for each pixel with a high recharge cycle or to perform binning processing for each group of pixels with a low recharge cycle, based on the clock signals CK1 and CK2 from the pulse generation circuit 3.

[0210] Figure 40 is a block diagram of the main part of the light detection device 1 according to one modified example of the 18th embodiment, and Figure 41 is a plan view of the pixel array section 20 of the light detection device 1 according to one modified example of the 18th embodiment.

[0211] As shown in Figures 40 and 41, the pixel array 20 according to one modified example of the 18th embodiment has a mixture of IR pixels and EVS pixels. The IR pixels are used, for example, to measure distance using the ToF (Time of Flight) method.

[0212] The IR pixel has the same circuit configuration as in Figure 39A. The EVS pixel has the same circuit configuration as in Figure 39B.

[0213] Thus, EVS pixels that perform event detection and other pixels may be mixed and arranged in the pixel array 20. In this case, the control unit 40 selects either a low-power mode or a normal mode for the EVS pixels based on the event detection results and counts the number of events.

[0214] (Ninthest Embodiment) The photodetector 1 according to the ninth embodiment has a laminated structure. Various variations of the laminated structure of the photodetector 1 are possible. A typical laminated structure will be described below, but other laminated structures are also possible.

[0215] Figure 42 shows an example of a two-layer laminated structure, and Figure 43 shows an example of a three-layer laminated structure. Hereafter, the first layer will be referred to as the first substrate SB1, the second layer as the second substrate SB2, and the third layer as the third substrate SB3.

[0216] In the two-layer structure shown in Figure 42, multiple SPADs 4 are arranged on the first substrate SB1. Hereinafter, pixels having SPADs 4 may be referred to as SPAD pixels PX. A pixel-under-circuit 43 is arranged in the region of the second substrate SB2 that overlaps with the first substrate SB1 in a plan view, and peripheral circuits 50 of the light detection device 1 are arranged around it. The peripheral circuits 50 include, for example, an addition processing circuit 39, a reference count frame memory 33, and an event detection circuit 9. The pixel-under-circuit 43 includes a control circuit 5, a counter 26, a holding circuit 7, and a first read circuit 10.

[0217] The first substrate SB1 and the second substrate SB2 are joined together by CCC (Cuper-to-Cuper Connection), which directly joins copper (Cu) pads to each other.

[0218] In the three-layer structure shown in Figure 43, multiple SPAD pixels PX are arranged on the first substrate SB1. The under-pixel circuits are arranged on the second substrate SB2. The under-pixel circuits include, for example, a control circuit 5, a counter 26, a holding circuit 7, and a first readout circuit 10. The peripheral circuits 50 of the light detection device 1 are arranged on the third substrate SB3. The peripheral circuits 50 include, for example, an addition processing circuit 39, a reference count frame memory 33, and an event detection circuit 9. By using a three-layer structure, the sizes of the first substrate SB1 to the third substrate SB3 can be standardized, and the external dimensions of the light detection device 1 can be reduced.

[0219] (20th Embodiment) Figure 44 is a block diagram showing the main parts of the photodetector 1 according to the 20th embodiment. The comparator of the photodetector 1 according to the 20th embodiment shown in Figure 44 has a first threshold V ｔｈ１ and the second threshold V ｔｈ２ The following is input. First threshold V ｔｈ１ This is the threshold for event determination when detecting events for each pixel with a high recharge cycle. Second threshold V ｔｈ２ This is the threshold used for event detection when binning is performed on each pixel group with a slow recharge cycle.

[0220] Figure 45 shows the first threshold V ｔｈ１ The pixel region to which it is applied, and the second threshold V ｔｈ２ This figure shows the pixel region to which the first threshold V is applied. As shown in Figure 45, the pixel region to which an event is detected for each pixel has a first threshold V ｔｈ１ The second threshold V is applied to the remaining pixel regions. ｔｈ２ This applies.

[0221] When binning is performed with a slow recharge cycle, the magnitude of the difference output from the subtraction circuit 32 differs compared to when events are detected for each pixel with a high recharge cycle. Therefore, in the 20th embodiment, different thresholds are set for when events are detected for each pixel with a high recharge cycle and when binning is performed with a high recharge cycle to determine whether or not an event has occurred.

[0222] This allows the system to determine whether or not an event has occurred using the same criteria, even when the mode changes.

[0223] (21st Embodiment) The light detection device 1 according to the 21st embodiment has a block configuration similar to that in Figure 24, for example. The light detection device 1 according to the 21st embodiment reduces power consumption in frames in which no events occur.

[0224] Figure 46 is a timing diagram of the optical detection device 1 according to the 21st embodiment. In Figure 46, times t1 to t5 represent the low-power mode, during which binning is performed with a slow recharge cycle. Times t6 to t7 represent the normal mode, during which events are detected for each pixel with a fast recharge cycle.

[0225] If no events are detected during the frame period from time t1 to t2, the multiple frame periods from time t2 to t3 are decimated. Decimation is performed, for example, by the control unit 40. In decimation, photon detection at each SPAD4 is forcibly stopped by not applying a recharge voltage to the SPAD4 of each pixel. This reduces power consumption. The period from time t1 to t3 is the time required to generate one image based on the photon detection results.

[0226] The system determines whether an event occurred during the frame period from time t3 to t4. If no event is detected again, the decimation process is repeated from time t4 onwards.

[0227] In normal mode from time t5 onward, event detection is performed for each frame, regardless of whether an event has occurred. Based on the event detection results over multiple frame periods from time t5 to t7, an event image or grayscale image is generated.

[0228] (22nd Embodiment) The decimation process described in the 21st embodiment may be performed on the entire pixel array 20, or on a portion of the pixel region of the pixel array 20.

[0229] Figure 47 is a diagram illustrating the decimation process according to the 22nd embodiment. The light detection device 1 according to the 22nd embodiment performs decimation on pixel regions of arbitrary position and size in the pixel array section 20. For each period during which an image is generated based on the event detection result, it is determined whether or not an event has occurred, and if no event is detected, decimation is performed (state ST41). Decimation is performed, for example, by the control unit 40.

[0230] When an event is detected (state ST42), the event is detected for each pixel across the entire pixel array 20, and a grayscale image is generated based on the event detection result in synchronization with the event detection (state ST43).

[0231] The decimation process may be performed on a row-by-row basis, a column-by-column basis, or a pixel-by-pixel basis. Figure 48A shows an example of column-by-column decimation, and Figure 48B shows an example of pixel-by-pixel decimation.

[0232] (23rd Embodiment) The photodetector 1 according to the 23rd embodiment switches the exposure time based on the event detection result.

[0233] Since binning is assumed to be performed during periods when no events are detected, a slow recharge period is applied, which can widen the dynamic range on the low-light side. When an event is detected and the system transitions to normal mode, event detection is performed for each pixel with a high recharge cycle, which narrows the dynamic range on the low-light side. Therefore, in the 23rd embodiment, the exposure time is made longer in normal mode than in low-power mode to prevent any difference in the dynamic range on the low-light side.

[0234] Figure 49 is a diagram illustrating the processing operation of the photodetector 1 according to the 23rd embodiment. When no events are detected, a low-power mode is selected, and binning processing is performed to detect events for each pixel group with a slow recharge cycle and a shorter exposure time than in the normal mode (state ST51).

[0235] When an event is detected in low-power mode (state ST52), the system transitions to normal mode and performs event detection for each pixel with a high-speed recharge cycle and a longer exposure time than in low-power mode (state ST53).

[0236] The timing of the exposure period within a single frame is arbitrary.

[0237] Figure 50A shows a modified example of the exposure period in low-power mode, and Figure 50B shows a modified example of the exposure period in normal mode. Figure 50A shows an example in which multiple exposure periods are distributed within one frame period. Figure 50B shows an example in which an exposure period is provided over almost the entire length of one frame period.

[0238] (24th Embodiment) Figure 51 is a block diagram of the main part of the light detection device 1 according to the 24th embodiment. The light detection device 1 according to the 24th embodiment is characterized in that two adjacent pixels share one counter 26 when in low power mode.

[0239] As shown in Figure 51, each pixel of the pixel array 20 has a SPAD 4, a control circuit 5, a counter 26, a holding circuit 7, and a first readout circuit 10. In addition, two adjacent pixels in a second direction (for example, in the column direction) share a counter switching circuit 44. The counter switching circuit 44 has a first switch 45 connected to the input node of the counter 26 of one pixel, a second switch 46 connected to the input node of the counter 26 of the other pixel, and an OR gate 47.

[0240] Figure 52A shows the circuit operation in low-power mode, and Figure 52B shows the circuit operation in normal mode. The OR gate 47 calculates the logical OR of the output signals of the inverters 48 for each pixel. The first switch 45 selects the output signal of the OR gate 47 in low-power mode as shown in Figure 52A, and selects the output signal of the inverter 48 in normal mode as shown in Figure 52B. The second switch 46 is off in low-power mode as shown in Figure 52A and does not select the output signal of the inverter 48. The second switch 46 selects the output signal of the inverter 48 in normal mode as shown in Figure 52B.

[0241] Thus, in the pixel configuration shown in Figure 51, during low-power mode, the counter 26 inside one of the two pixels counts the number of events detected by both pixels. This allows the counter 26 inside the other pixel to be stopped, thereby saving power.

[0242] (25th Embodiment) Figure 53 is a block diagram of the main parts of the light detection device 1 according to the 25th embodiment. The 25th embodiment is characterized in that a single counter 26 is shared by multiple pixels in low-power mode using a method different from that of the 24th embodiment.

[0243] A switch 49 is positioned between the cathode of the SPAD4 of one of two adjacent pixels and the cathode of the SPAD4 of the other pixel. The switch 49 is switched on by the signal logic of the control line SL3. More specifically, in low-power mode, the switch 49 is on, short-circuiting the cathodes of the SPAD4 of one pixel and the other pixel. In normal mode, the switch 49 is off.

[0244] Figure 54A shows the operation of two adjacent pixels in low-power mode, and Figure 54B shows the operation of two adjacent pixels in normal mode.

[0245] As shown in Figure 54A, in low-power mode, the cathodes of each SPAD4 of two adjacent pixels are short-circuited. Therefore, the counter 26 of one pixel counts the number of events detected by the two pixels.

[0246] As shown in Figure 54B, in normal mode, each counter 26 of two adjacent pixels counts the number of events separately.

[0247] <Examples of application to mobile devices> The technology disclosed herein (this technology) can be applied to various products. For example, the technology disclosed herein may be implemented as a device mounted on any type of mobile device such as automobiles, electric vehicles, hybrid electric vehicles, motorcycles, bicycles, personal mobility devices, airplanes, drones, ships, and robots.

[0248] Figure 55 is a block diagram showing a schematic configuration example of a vehicle control system, which is an example of a mobile control system to which the technology described herein may be applied.

[0249] The vehicle control system 12000 comprises a plurality of electronic control units connected via a communication network 12001. In the example shown in Figure 55, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an external information detection unit 12030, an internal information detection unit 12040, and an integrated control unit 12050. The functional configuration of the integrated control unit 12050 is shown in the figure, which includes a microcomputer 12051, an audio / image output unit 12052, and an in-vehicle network interface 12053.

[0250] The drivetrain control unit 12010 controls the operation of devices related to the vehicle's drivetrain according to various programs. For example, the drivetrain control unit 12010 functions as a control device for a drivetrain generating device that generates driving force for the vehicle, such as an internal combustion engine or a drive motor; a drivetrain transmission mechanism that transmits driving force to the wheels; a steering mechanism that adjusts the steering angle of the vehicle; and a braking device that generates braking force for the vehicle.

[0251] The body system control unit 12020 controls the operation of various devices mounted on the vehicle body according to various programs. For example, the body system control unit 12020 functions as a control device for a keyless entry system, a smart key system, a power window system, or various lamps such as headlights, reverse lights, brake lights, turn signals, or fog lights. In this case, the body system control unit 12020 may receive radio waves transmitted from a portable device that replaces a key or signals from various switches. The body system control unit 12020 receives these radio waves or signals and controls the vehicle's door lock system, power window system, lamps, etc.

[0252] The external information detection unit 12030 detects information from outside the vehicle equipped with the vehicle control system 12000. For example, an imaging unit 12031 is connected to the external information detection unit 12030. The external information detection unit 12030 causes the imaging unit 12031 to capture images of the outside of the vehicle and receives the captured images. Based on the received images, the external information detection unit 12030 may perform object detection processing such as detecting people, cars, obstacles, signs, or characters on the road surface, or distance detection processing.

[0253] The imaging unit 12031 is a light sensor that receives light and outputs an electrical signal corresponding to the amount of light received. The imaging unit 12031 can output the electrical signal as an image or as distance measurement information. The light received by the imaging unit 12031 may be visible light or invisible light such as infrared light.

[0254] The in-vehicle information detection unit 12040 detects information inside the vehicle. The in-vehicle information detection unit 12040 is connected to, for example, a driver status detection unit 12041 that detects the driver's state. The driver status detection unit 12041 includes, for example, a camera that captures images of the driver, and the in-vehicle information detection unit 12040 may calculate the driver's level of fatigue or concentration, or determine whether the driver is drowsy, based on the detection information input from the driver status detection unit 12041.

[0255] The microcomputer 12051 can calculate control target values ​​for the drive force generator, steering mechanism, or braking device based on information inside and outside the vehicle acquired by the external information detection unit 12030 or the internal information detection unit 12040, and output control commands to the drive system control unit 12010. For example, the microcomputer 12051 can perform cooperative control aimed at realizing ADAS (Advanced Driver Assistance System) functions, including collision avoidance or impact mitigation, following driving based on distance between vehicles, maintaining vehicle speed, vehicle collision warning, or vehicle lane departure warning.

[0256] Furthermore, the microcomputer 12051 can perform cooperative control for purposes such as autonomous driving, where the vehicle drives autonomously without driver intervention, by controlling the drive force generating device, steering mechanism, or braking device, etc., based on information about the vehicle's surroundings acquired by the external information detection unit 12030 or the internal information detection unit 12040.

[0257] Furthermore, the microcomputer 12051 can output control commands to the body system control unit 12020 based on external information acquired by the external information detection unit 12030. For example, the microcomputer 12051 can control the headlights according to the position of a preceding or oncoming vehicle detected by the external information detection unit 12030, and perform coordinated control aimed at reducing glare, such as switching from high beams to low beams.

[0258] The audio-image output unit 12052 transmits at least one of audio and image output signals to an output device capable of visually or audibly notifying information to the vehicle's occupants or to those outside the vehicle. In the example shown in Figure 55, the output devices include an audio speaker 12061, a display unit 12062, and an instrument panel 12063. The display unit 12062 may include, for example, at least one of an onboard display and a head-up display.

[0259] Figure 56 shows an example of the installation position of the imaging unit 12031.

[0260] In Figure 56, the imaging unit 12031 includes imaging units 12101, 12102, 12103, 12104, and 12105.

[0261] The imaging units 12101, 12102, 12103, 12104, and 12105 are installed, for example, on the front nose, side mirrors, rear bumper, back door, and the upper part of the windshield inside the vehicle 12100. The imaging unit 12101 installed on the front nose and the imaging unit 12105 installed on the upper part of the windshield inside the vehicle mainly acquire images of the front of the vehicle 12100. The imaging units 12102 and 12103 installed on the side mirrors mainly acquire images of the sides of the vehicle 12100. The imaging unit 12104 installed on the rear bumper or back door mainly acquires images of the rear of the vehicle 12100. The imaging unit 12105 installed on the upper part of the windshield inside the vehicle is mainly used for detecting preceding vehicles, pedestrians, obstacles, traffic lights, traffic signs, or lanes.

[0262] Figure 56 shows an example of the imaging ranges of imaging units 12101 to 12104. Imaging range 12111 indicates the imaging range of imaging unit 12101 located on the front nose, imaging ranges 12112 and 12113 indicate the imaging ranges of imaging units 12102 and 12103 located on the side mirrors, respectively, and imaging range 12114 indicates the imaging range of imaging unit 12104 located on the rear bumper or back door. For example, by superimposing the image data captured by imaging units 12101 to 12104, an overhead view image of the vehicle 12100 can be obtained.

[0263] At least one of the imaging units 12101 to 12104 may have a function for acquiring distance information. For example, at least one of the imaging units 12101 to 12104 may be a stereo camera consisting of multiple image sensors, or an image sensor having pixels for phase difference detection.

[0264] For example, the microcomputer 12051, based on distance information obtained from the imaging units 12101 to 12104, can determine the distance to each object within the imaging range 12111 to 12114 and the temporal change of this distance (relative speed to the vehicle 12100). In particular, it can extract the closest object on the vehicle 12100's path that is traveling in approximately the same direction as the vehicle 12100 at a predetermined speed (e.g., 0 km / h or more) as the preceding vehicle. Furthermore, the microcomputer 12051 can set a predetermined distance to be maintained before the preceding vehicle and perform automatic braking control (including follow-and-stop control) and automatic acceleration control (including follow-and-start control), etc. In this way, cooperative control aimed at autonomous driving, where the vehicle drives autonomously without driver intervention, can be performed.

[0265] For example, the microcomputer 12051 can use distance information obtained from imaging units 12101 to 12104 to classify and extract three-dimensional object data related to three-dimensional objects, such as motorcycles, passenger cars, large vehicles, pedestrians, utility poles, and other three-dimensional objects, and use this data for automatic obstacle avoidance. For example, the microcomputer 12051 identifies obstacles around the vehicle 12100 into obstacles that are visible to the driver of the vehicle 12100 and obstacles that are difficult to see. The microcomputer 12051 then determines the collision risk, which indicates the degree of risk of collision with each obstacle. If the collision risk is above a set value and there is a possibility of collision, the microcomputer 12051 can provide driving assistance to avoid collisions by outputting a warning to the driver via the audio speaker 12061 or the display unit 12062, or by performing forced deceleration or evasive steering via the drive system control unit 12010.

[0266] At least one of the imaging units 12101 to 12104 may be an infrared camera that detects infrared light. For example, the microcomputer 12051 can recognize pedestrians by determining whether or not pedestrians are present in the images captured by the imaging units 12101 to 12104. Such pedestrian recognition is performed, for example, by a procedure to extract feature points from the images captured by the imaging units 12101 to 12104 as infrared cameras, and a procedure to perform pattern matching on a series of feature points that indicate the contour of an object to determine whether or not it is a pedestrian. When the microcomputer 12051 determines that a pedestrian is present in the images captured by the imaging units 12101 to 12104 and recognizes a pedestrian, the audio-image output unit 12052 controls the display unit 12062 to superimpose a rectangular contour line for emphasis on the recognized pedestrian. The audio-image output unit 12052 may also control the display unit 12062 to display an icon indicating a pedestrian at a desired position.

[0267] The above describes an example of a vehicle control system to which the technology described herein may be applied. The technology described herein may be applied to the imaging unit 12031, etc., among the configurations described above.

[0268] Furthermore, this technology can take the following configurations: (1) A light detection device comprising: a photodiode that performs avalanche multiplication in response to incident photons; a control circuit that controls switching between a first state in which the photodiode can detect the photons and a second state in which the photodiode is recharged; a counter that counts the number of times the photons have been detected by the photodiode; and an event detection circuit that detects an event when the change in brightness exceeds a certain threshold, based on the difference between a certain threshold and the count value of the counter, regardless of the illuminance. (2) The light detection device according to (1), wherein the control circuit switches the period for recharging the photodiode between two or more periodic values ​​of a reference period. (3) The light detection device according to (1) or (2), wherein the event detection circuit combines a plurality of the recharging periods to detect the event based on the count value which has a logarithmic response to illuminance. (4) The control circuit generates a pulse signal corresponding to a change in the potential of the cathode or anode of the photodiode, and the counter counts the number of pulse signals, as described in any one of (1) to (3). (5) The counter distinguishes and counts the photons incident in a first period and the photons incident in a second period that is longer than the first period, and the control circuit sets the recharge period of the photodiode to the sum of the first period and the second period. as described in any one of (1) to (4). (6) The counter updates the count value twice when a photon is incident in the first period, and updates the count value once when a photon is incident in the second period, as described in (5). (7) The counter distinguishes and changes the count value for each of the two or more periods in which the photons are incident, as described in any one of (1) to (6).(8) The light detection device according to any one of (1) to (7), comprising a plurality of pixels and a stacked first substrate and a second substrate, each of the plurality of pixels having a photodiode and a grayscale readout circuit that outputs grayscale information corresponding to the count value of the counter, the plurality of photodiodes having the plurality of pixels arranged on the first substrate, and the counter and the grayscale readout circuit arranged on the second substrate. (9) The light detection device according to any one of (1) to (8), wherein the event detection circuit is provided for at least some of the pixels. (10) The light detection device according to (8) or (9), wherein the counter operates as an up counter or a down counter for each frame, and comprises a holding circuit that holds the count value in the frame in which the counter operates as the up counter, the counter operates as the up counter in the first frame in which the count value is reset, and operates as the down counter that counts down from the count value held in the holding circuit in the second frame following the first frame. (11) The optical detection device according to (10), comprising a comparator that compares the down count value at the end of a second frame in which the counter operates as the down counter with the threshold, wherein the counter operates as the down counter in the third frame following the second frame, when the comparator determines that the down count value is less than the threshold, by down-counting from the count value held in the holding circuit. (12) The optical detection device according to (11), wherein the event detection circuit detects the event when the comparator determines that the count value is greater than or equal to the threshold, and the counter resets the count value and operates as the up counter in the next frame. (13) The optical detection device according to (11) or (12), wherein two or more pixels share one comparator, and each of the two or more pixels compares the count value of a counter provided for each pixel with the threshold in a time-division manner. (14) The light detection device according to any one of (10) to (13), comprising a circuit that adds the count value of the counter on a frame-by-frame basis and outputs it as a grayscale value.(15) The photodetector according to (14), further comprising a correction processing circuit for correcting the correspondence between the count value of the counter and the frequency of the incident photon. (16) The photodetector according to (15), wherein the event detection circuit and the correction processing circuit are output at different frame rates. (17) The photodetector according to any one of (10) to (16), further comprising a plurality of pixels and stacked first and second substrates, wherein each of the plurality of pixels has a photodiode, a control circuit, a counter, an event detection circuit, a first readout circuit for outputting the event, and a second readout circuit for outputting gradation information corresponding to the count value of the counter, the photodiode being arranged on the first substrate, and at least one of the control circuit, the counter, the event detection circuit, the first readout circuit, and the second readout circuit being arranged on the second substrate. (18) The light detection device according to (17), further comprising a third substrate laminated on the second substrate, wherein the control circuit, the counter, the event detection circuit, the first readout circuit, and the second readout circuit that are not arranged on the second substrate are arranged on the third substrate. (19) An optical detection device according to any one of (1) to (7), comprising: a signal line connected to two or more pixels arranged in the column direction; a first holding circuit that holds the count value of the counter for each pixel transmitted on the signal line; a second holding circuit that holds the count value held in the first holding circuit when an event is detected; a subtraction processing circuit that detects the difference between the count value held in the first holding circuit and the count value held in the second holding circuit when reading out the count values ​​of the counters for a plurality of pixels for one row arranged in the row direction in parallel; and a comparator that compares the difference with a threshold, wherein the event detection circuit detects the event when the comparator determines that the difference is greater than or equal to the threshold, and causes the second holding circuit to hold the count value held in the first holding circuit; and detects the event for a plurality of pixels for one column of the next adjacent row in the column direction when the comparator determines that the difference is less than the threshold.(20) The optical detection device according to (19), wherein the counter is reset each frame and performs an up-count operation each time the photon is detected by the photodiode, and the event detection circuit detects the event each frame based on the comparison result by the comparator. (21) The optical detection device according to (20), comprising an adder circuit that adds the count value of the counter of each pixel transmitted on the signal line row by row, the first holding circuit holds the count value added by the adder circuit, and the event detection circuit detects the event for each of a plurality of pixels adjacent in the column direction. (22) The optical detection device according to (21), comprising an adder circuit that adds the count values ​​of adjacent or surrounding pixels and the same pixel with overlapping values. (23) The optical detection device according to any one of (19) to (22), comprising a circuit that adds the count value of the counter of each pixel transmitted on the signal line frame by frame and outputs a grayscale value. (24) The light detection device according to (23), further comprising a correction processing circuit for correcting the correspondence between the count value of the counter and the frequency of the incident photon. (25) The light detection device according to any one of (19) to (24), wherein the event detection circuit and the grayscale value output circuit output at different frame rates. (26) The optical detection device according to any one of (19) to (25), comprising: a plurality of pixels; a first readout circuit for outputting the event; a second readout circuit for outputting grayscale information corresponding to the count value of the counter; and a stacked first substrate and a second substrate, each of the plurality of pixels having: the photodiode; the control circuit; the counter; the event detection circuit, the first readout circuit, the first holding circuit, the second holding circuit, the subtraction processing circuit, and the comparator are shared by the plurality of pixels; the photodiode is arranged on the first substrate; and at least one of the control circuit, the counter, the event detection circuit, the first readout circuit, the first holding circuit, the second holding circuit, the subtraction processing circuit, and the comparator is arranged on the second substrate.(27) The photodetector according to (26), further comprising a third substrate laminated on the second substrate, wherein the control circuit, the counter, the event detection circuit, the first readout circuit, and the second readout circuit that are not arranged on the second substrate are arranged on the third substrate. (28) The photodetector according to (26), further comprising: a photodiode that performs avalanche multiplication in response to incident photons; a control circuit that controls switching between a first state in which the photodiode can detect the photons and a second state in which the photodiode is recharged; a counter that counts the number of times the photons have been detected by the photodiode; an event detection circuit that detects an event when the change in brightness exceeds a certain threshold based on the difference between a certain threshold and the count value of the counter, regardless of illuminance; and a control unit that selects one of a plurality of modes in which the period for recharging the photodiode and the pixel region for detecting the event are different, based on the event detection result by the event detection circuit. (29) The photodetector according to (28), wherein the control unit selects one of the plurality of modes based on the number of events detected by the event detection circuit. (30) The photodetector according to (28) or (29), wherein the plurality of modes include a first mode for low-power operation and a second mode for detecting the events with high resolution. (31) The photodetector according to (30), wherein the first mode counts the number of events by performing binning processing with a slower recharge cycle than the second mode, and the second mode counts the number of events for each pixel with a faster recharge cycle than the first mode. (32) The photodetector according to any one of (28) to (31), comprising a pixel array section having a plurality of pixels, wherein at least some of the plurality of pixels have the photodiode, the control circuit, and the counter, and the control unit selects the plurality of pixels by combining two or more of the modes.(33) The optical detection device according to (32), wherein the plurality of pixels comprises a plurality of first pixel groups arranged in a first direction and extending in a second direction, and a plurality of second pixel groups arranged in the second direction and extending in the first direction, and the control unit selects the mode using at least one of one or more of the first pixel groups or the second pixel groups as a unit. (34) The optical detection device according to (32), wherein the control unit selects the mode for each of the plurality of pixels. (35) The optical detection device according to (32), wherein the control unit selects the mode using a pixel group containing two or more of the pixels as a unit. (36) The optical detection device according to any one of (32) to (35), wherein the control unit generates a plurality of control signals to select a pixel region containing any pixel among the plurality of pixels, and each of the plurality of pixels selects the mode based on the plurality of control signals. (37) The light detection device according to any one of (32) to (36), further comprising an addition processing circuit that generates a plurality of pixel groups, each containing two or more pixels that perform binning processing, based on the mode selected by the control unit, and counts the number of events for each pixel group. (38) The light detection device according to (37), wherein the addition processing circuit generates the plurality of pixel groups such that the pixel groups do not overlap with each other. (39) The light detection device according to (37), wherein the addition processing circuit generates the plurality of pixel groups such that two adjacent pixel groups partially overlap. (40) The light detection device according to any one of (32) to (39), wherein each of the plurality of pixels outputs a grayscale value and the detection result of the event. (41) The light detection device according to any one of (32) to (40), wherein the plurality of pixels include a pixel for detecting the event, a pixel for acquiring grayscale information, and a pixel for measuring the distance of an object. (42) The photodetector according to any one of (32) to (41), comprising a stacked first substrate and a second substrate, wherein the first substrate has a plurality of photodiodes included in the plurality of pixels, and the second substrate includes at least one of the counter, the event detection circuit, and the control unit.(43) The photodetector according to any one of (32) to (41), comprising a stacked first substrate, a second substrate, and a third substrate, wherein the first substrate has a plurality of photodiodes included in the plurality of pixels, and the second substrate or the third substrate includes at least one of the counter, the event detection circuit, and the control unit. (44) The photodetector according to any one of (28) to (43), wherein the control unit selects one of the plurality of modes by comparing a plurality of thresholds corresponding to the plurality of modes with the detection result of the event. (45) The photodetector according to any one of (28) to (44), wherein the control unit performs a decimation process frame by frame when some of the modes are selected. (46) The photodetector according to any one of (32) to (43), wherein the control unit adjusts the exposure time of the pixels to which the selected mode is applied. (47) The photodetector according to any one of (32) to (43), wherein each of the two or more pixels to be binned among the plurality of pixels has a first switching circuit that switches whether or not to input a pulse signal generated by the control circuit to one of the two or more pixels to the counter. (48) The photodetector according to any one of (32) to (43), wherein each of the two or more pixels to be binned among the plurality of pixels has a second switching circuit that switches whether or not to connect the cathodes of the photodiodes.

[0269] The aspects of this disclosure are not limited to the individual embodiments described above, but include various modifications that a person skilled in the art could conceive, and the effects of this disclosure are not limited to those described above. In other words, various additions, modifications, and partial deletions are possible, as long as they do not depart from the conceptual idea and spirit of this disclosure derived from the claims and their equivalents.

[0270] 1. Light detection device, 2. Pixel circuit, 2a. First pixel circuit, 2b. Second pixel circuit, 3. Pulse generation circuit, 5. Control circuit, 6. Up / down counter, 7. Hold circuit, 8. Comparator, 9. Event detection circuit, 10. First read circuit, 11. First selection circuit, 12. Second selection circuit, 13. Second read circuit, 20. Pixel array section, 21. Column processing circuit, 22. Selection circuit, 23. Adder, 25. Image frame memory, 26. Counter, 27. Pixel sub-circuit, 28. Extension section, 30. Row control circuit, 31. Count hold circuit, 32. Subtraction processing circuit, 33. Reference count frame memory, 34. Output interface circuit, 35. Adder circuit, 36. Output interface circuit

Claims

1. A light detection device comprising: a photodiode that performs avalanche multiplication in response to incident photons; a control circuit that controls switching between a first state in which the photodiode can detect the photons and a second state in which the photodiode is recharged; a counter that counts the number of times the photons have been detected by the photodiode; and an event detection circuit that detects an event when a change in brightness exceeds a certain threshold, based on the difference between a certain threshold and the count value of the counter, regardless of illuminance.

2. The photodetector according to claim 1, wherein the control circuit switches the recharging period of the photodiode to two or more multiplication periods of a reference period.

3. The light detection device according to claim 1, wherein the event detection circuit detects the event based on the count value which has a logarithmic response to illuminance by combining a plurality of recharge periods.

4. The photodetector according to claim 1, wherein the control circuit generates a pulse signal corresponding to a change in the potential of the cathode or anode of the photodiode, and the counter counts the number of pulse signals.

5. The photodetector according to claim 1, wherein the counter distinguishes and counts photons incident in a first period and photons incident in a second period longer than the first period, and the control circuit sets the recharge period of the photodiode to be the sum of the first period and the second period.

6. The photodetector according to claim 5, wherein the counter updates the count value twice when the photon is incident during the first period, and updates the count value once when the photon is incident during the second period.

7. The photodetector according to claim 1, wherein the counter distinguishes between photons incident during two or more periods of the recharge cycle and changes the count value accordingly.

8. The photodetector according to claim 1, comprising a plurality of pixels and a stacked first substrate and a second substrate, each of the plurality of pixels having a photodiode and a grayscale readout circuit that outputs grayscale information corresponding to the count value of the counter, the plurality of photodiodes of the plurality of pixels arranged on the first substrate, and the counter and the grayscale readout circuit arranged on the second substrate.

9. The photodetector according to claim 1, wherein the event detection circuit is provided for at least some of the pixels.

10. The photodetector according to claim 8, wherein the counter operates as an up-counter or a down-counter for each frame, and includes a holding circuit that holds the count value in the frame in which the counter operates as the up-counter, and the counter operates as the up-counter in the first frame in which the count value is reset, and operates as the down-counter in the second frame following the first frame, down-counting from the count value held in the holding circuit.

11. The photodetector according to claim 10, further comprising a correction processing circuit for correcting the correspondence between the count value of the counter and the frequency of the incident photon, wherein the event detection circuit and the correction processing circuit output at different frame rates.

12. An optical detection device according to claim 1, comprising: a signal line connected to two or more pixels arranged in a column direction; a first holding circuit that holds the count value of the counter for each pixel transmitted on the signal line; a second holding circuit that holds the count value held in the first holding circuit when an event is detected; a subtraction processing circuit that detects the difference between the count value held in the first holding circuit and the count value held in the second holding circuit when reading the count values ​​of the counters for a plurality of pixels for one row arranged in a row direction in parallel; and a comparator that compares the difference with a threshold, wherein the event detection circuit detects the event when the comparator determines that the difference is greater than or equal to the threshold, and causes the second holding circuit to hold the count value held in the first holding circuit; and detects the event for a plurality of pixels for one column of the next adjacent row in the column direction when the comparator determines that the difference is less than the threshold.

13. The photodetector according to claim 12, wherein the counter is reset each frame and performs an up-count operation each time the photon is detected by the photodiode, and the event detection circuit detects the event each frame based on the comparison result by the comparator.

14. The optical detection device according to claim 12, comprising an adder circuit that adds the count value of the counter of each pixel transmitted on the signal line row by row, the first holding circuit holding the count value added by the adder circuit, and the event detection circuit detecting the event for each of a plurality of pixels adjacent in the column direction.

15. The photodetector according to claim 14, further comprising an addition circuit that overlaps and adds the count values ​​of adjacent or surrounding pixels with those of the same pixel.

16. The light detection device according to claim 12, further comprising a circuit that adds the count value of the counter for each pixel transmitted on the signal line on a frame-by-frame basis to output a grayscale value.

17. The photodetector according to claim 16, further comprising a correction processing circuit for correcting the correspondence between the count value of the counter and the frequency of the incident photon.

18. The photodetector according to claim 12, wherein each of the plurality of pixels has a photodiode and a grayscale readout circuit that outputs grayscale information corresponding to the count value of the counter, and outputs at a different frame rate than the event detection circuit and the grayscale readout circuit.

Images