Solid-state imaging device and electronic apparatus

Abstract

[Problem] To improve performance. [Solution] This solid-state imaging device comprises pixels, a register, a control circuit, and an output circuit. The pixels are arranged in an array, and acquire data relating to the same object by shifting the data in a predetermined direction for each unit time. The register is provided for the pixels and stores the acquired data. The control circuit controls the acquisition of the data. The output circuit converts the value of the register. The register copies the value of the same object from one unit time in the past, and the control circuit causes data to be acquired when the value of the register is not equal to or greater than a predetermined value, and stops the acquisition of data in at least one corresponding pixel when the value of the register is equal to or greater than the predetermined value. The output circuit converts and outputs the value in accordance with the timing at which the value of the register becomes equal to or greater than the predetermined value.

Description

Solid-state imaging device and electronic device

[0001] The present disclosure relates to a solid-state imaging device and an electronic device.

[0002] As a sensor in a device for counting the number of photons, there are photon counting elements such as avalanche photodiodes including single photon avalanche photodiodes (SPADs). A sensor including this photon counting element is widely used, for example, as a sensor of a device for measuring distance. When imaging a high-brightness subject, there is a problem that the state of counting photons continues for a long time and power consumption increases.

[0003] When using this sensor, since sufficient photons can be acquired at any time for a bright subject, the counting is stopped when a certain count that satisfies the signal-to-noise ratio criterion is obtained, and low power consumption can be realized by estimating the number of photons at any time from the time until then. This technology can be appropriately implemented in a two-dimensional array sensor typified by a so-called area sensor, but in the case of multiple exposure using a line scan type sensor using several lines of pixels, the exposure time is very short and multiple exposure is required, and since the same amount of power is consumed in counting photons in the next frame or sub-frame, there is a problem that the effect of reducing power consumption is not so great.

[0004] Japanese Patent Application Laid-Open No. 2023-543709

[0005] Therefore, one of the non-limiting problems to be solved by the embodiments of the present disclosure is to improve the performance in photon counting using a line scan method. The problems to be solved by the embodiments of the present disclosure can also be, as some further non-limiting examples, problems corresponding to the effects described in the embodiments. That is, the problems corresponding to any at least one of the effects described in the description of the embodiments of the present disclosure can be the problems to be solved in the present disclosure.

[0006] According to one embodiment, the solid-state imaging device comprises a plurality of pixels, a register, a control circuit, and an output circuit. The plurality of pixels are arranged in an array in a first direction and a second direction intersecting the first direction, and data for the same object is acquired by multiple exposure, shifting in the second direction at unit time intervals. The register is provided for each of the plurality of pixels and stores the data acquired by each of the plurality of pixels. The control circuit controls imaging at the pixels and data acquisition from the pixels. The output circuit converts the value stored in the register and outputs it. The register also copies the value of the register of the pixel that acquired information on the same object in the unit time interval, and the control circuit controls imaging and data acquisition at the pixel corresponding to the register if the value of the register is not equal to or greater than a predetermined value, and controls at least one of imaging or data acquisition at the pixel corresponding to the register to stop if the value of the register is equal to or greater than the predetermined value, and the output circuit converts the value stored in the register by a predetermined calculation according to the timing when the value of the register becomes equal to or greater than the predetermined value and outputs it.

[0007] The aforementioned pixel may be a photon counting element.

[0008] The register may operate as a counter for counting photons incident on the pixel, and the counter may store the value stored by adding a count value based on the output from the pixel to the stored value.

[0009] The register may include a first area for storing count values ​​and a second area for indicating whether the stored value is greater than or equal to a predetermined value. If the value in the first area exceeds the predetermined value, a value indicating that the value stored in the first area is greater than or equal to the predetermined value may be set in the second area.

[0010] The output circuit may set the value of the second region based on the timing at which the value of the first region exceeds the predetermined value.

[0011] The output circuit may output a value stored in the first region multiplied by a constant, depending on the value stored in the second region.

[0012] The output circuit may output the value stored in the first region after left-shifting it according to the value stored in the second region.

[0013] The value stored in the second region may be set based on the elapsed time from when the subject is first photographed until the first region exceeds the predetermined value.

[0014] The leading pixel in the second direction for photographing the target may be set to a lower sensitivity than the other pixels in the second direction to which that pixel belongs.

[0015] The second pixel from the beginning in the second direction for photographing the target may be set to have higher sensitivity than the first pixel, and lower sensitivity than the other pixels in the second direction to which the first pixel and the second pixel belong.

[0016] Each of the aforementioned plurality of pixels is equipped with a color filter selected from a plurality of colors based on the arrangement of the pixels, and data relating to the light transmitted through the color filter may be acquired, and the processing relating to the value of the register may be performed for each pixel equipped with the same color color filter.

[0017] Each of the aforementioned plurality of pixels is equipped with a color filter selected from a plurality of colors based on the arrangement of the pixels, and data relating to the light transmitted through the color filter may be acquired, and the processing relating to the value of the register may be performed for each pixel that is photographing the same object regardless of the color transmitted by the color filter.

[0018] According to one embodiment, the electronic device comprises a plurality of pixels, a register, and a control circuit. The plurality of pixels are arranged in an array in a first direction and a second direction intersecting the first direction, and data for the same object is acquired by multiple exposure, shifting in the second direction at unit time intervals. The register is provided for each of the plurality of pixels and stores the data acquired by each of the plurality of pixels. The control circuit controls imaging at the pixels and data acquisition from the pixels. The register also copies the value of the register of the pixel that acquired information for the same object in the unit time interval, and the control circuit controls imaging and data acquisition at the pixel corresponding to the register if the value of the register is not equal to or greater than a predetermined value, and controls at least one of imaging or data acquisition at the pixel corresponding to the register to stop if the value of the register is equal to or greater than the predetermined value.

[0019] A schematic block diagram showing at least a part of a solid-state imaging device according to one embodiment. A schematic diagram showing an example of the configuration of a pixel array according to one embodiment. A schematic diagram showing an example of multiple exposure according to one embodiment. A diagram showing an example of the relationship between the register value and the output value according to one embodiment. A diagram showing an example of the relationship between the register value and the output value according to one embodiment. A diagram showing an example of the relationship between the register value and the output value according to one embodiment. A diagram showing an example of the relationship between the register value and the output value according to one embodiment. A diagram showing an example of the relationship between the register value and the output value according to one embodiment. A diagram showing an example of the relationship between the register value and the output value according to one embodiment. A diagram showing an example of the relationship between a light-reducing pixel and a pixel according to one embodiment. A diagram showing an example of the relationship between the register value and the output value according to one embodiment. A schematic diagram showing an example of the relationship between the register value and the output value according to one embodiment. A schematic diagram showing an example of the circuit around a pixel according to one embodiment. A schematic diagram showing an example of the circuit around a pixel according to one embodiment. A schematic diagram showing an example of the circuit around a pixel according to one embodiment. A schematic diagram showing an example of the implementation of a solid-state imaging device according to one embodiment. A block diagram showing an example of the schematic configuration of a vehicle control system. An explanatory diagram showing an example of the installation position of the external information detection unit and the imaging unit.

[0020] The embodiments of this disclosure will now be described with reference to the drawings. The drawings are for illustrative purposes only, and the shape, size, or size ratio of each component in the actual device does not need to be exactly as shown in the drawings. Furthermore, the drawings are simplified, so any other components necessary for implementation should be appropriately provided in addition to those shown in the drawings.

[0021] This disclosure describes an example using a sensor that operates in a line-scan manner. An array sensor is used as the image sensor, in which a predetermined number of light-receiving pixels equipped with a photoelectric conversion area are arranged in lines. As an example, by scanning an object or an image of an object perpendicular to the lines, the same object or the same area of ​​the object is captured multiple times across multiple lines, thereby obtaining the brightness value (which can also be read as the number of photons) in each area of ​​the object.

[0022] This solid-state imaging device or electronic device having a line-scan sensor will be described in the following order: 1. Overview of the solid-state imaging device 2. Pixels and registers 3. Conversion of count values ​​and other control 4. Application examples

[0023] <1. Overview of Solid State Imaging Systems>

[0024] Figure 1 is a schematic block diagram showing at least some of the configurations in an example, not limited to, of a solid-state imaging device according to one embodiment. The solid-state imaging device 1 is configured, in an example, to receive light emitted, reflected, or transmitted from an object to generate an image, to perform object detection or motion detection based on the received light, to measure the distance to an object, or to generate and output data necessary for these functions. The solid-state imaging device 1 comprises a pixel array 10, a pixel control circuit 12, a counter 14, a control circuit 16, and a processing circuit 18.

[0025] In addition, the solid-state imaging device 1 includes registers (not shown) connected to each configuration as needed. Registers may be provided corresponding to each of the multiple pixels arranged in the pixel array 10. Furthermore, although it will be described below that there is one register for each pixel, the device is not limited to this, and each pixel may have a separate register for storing the count value and a register for storing the timing. Also, if high-speed data reading and writing is possible, at least a portion of the registers in this disclosure may be shared with other storage areas, or may be located in a predetermined area in another storage area. Registers may be composed of circuits such as latches and flip circuits.

[0026] The following configurations are for illustrative purposes only, and the operation of each circuit is not limited to these. For example, the operation of each circuit may be performed in combination with the operation of other circuits, or some operations may be isolated and configured as separate circuits. Also, although the data flow is shown by arrows as an example in the diagram, the data flow is not limited to this, and for example, there may be data flow in the reverse direction, data flow between unconnected stars, or data flow through the registers mentioned above.

[0027] The pixel array 10 is a region where pixels, each comprising a photoelectric conversion region (e.g., a photodiode), are arranged. The pixels are arranged, for example, in a line perpendicular to the scanning direction and in an array with a predetermined number of lines in the scanning direction.

[0028] The photodiode provided in the pixel may be, for example, an avalanche photodiode, and in particular, a SPAD. The case of a SPAD will be described below, but the forms described in this disclosure can be applied to other photodiodes as well, to the extent that no inconsistencies arise.

[0029] The pixel control circuit 12 is a circuit that controls the pixels arranged in the pixel array 10. For example, the pixel control circuit 12 controls the output from the pixel circuits arranged in the pixel array 10 to process them appropriately, or controls the pixels arranged in the pixel array 10 to perform initialization and / or imaging processing at appropriate timings.

[0030] The pixel control circuit 12 can, for example, quench and recharge the signal output from the pixel. The pixel control circuit 12 may also be configured to include a quench circuit and a recharge circuit, respectively. These processes may be passive or active. In recharging, the pixel control circuit 12 can perform control such as using a current source in the passive type or using a delay element for feedback reset in the active type.

[0031] The counter 14 counts the number of pulses output from the pixel control circuit 12 and stores the counted value in a register provided for each pixel. The counter 14 can be configured using, for example, a ripple counter. The counter 14 can also transfer (copy) the counted value to the register of an adjacent pixel. The counter 14 may be provided for every 100 pixels. The transfer circuit can use a circuit configuration using commonly known D flip-flops (including D latches) and multiplexers.

[0032] Furthermore, since this counter 14 has the function of retaining data when not performing counting operations, it can also be used as a register. In other words, it is possible to use the same circuit for both the counter 14 and a register. Similarly, when referred to as a register in this disclosure, it may be a circuit that operates as a counter 14. That is, the register may be at least a part of the counter 14.

[0033] Counter 14 counts photons at the pixel of interest, for example, and copies the value to the register corresponding to the pixel that will count photons in the same region at the next timing (unit time) in the scan direction. Counter 14 adds the current photon count to the value in the register that stores the count values ​​for the same region up to the previous timing (unit time) that was scanned at the pixel of interest, and copies this added count to the register of the pixel that will acquire photons from the same region at the next timing (unit time).

[0034] In this way, the counter 14 sequentially adds and transfers the photon count value in the scanning direction. For the first pixel in the scanning direction, the counter 14 may, as an example without limitation, reset the register value before starting counting. For the last pixel in the scanning direction, the counter 14 may, as an example without limitation, store the value in a predetermined register or transfer the register value to the control circuit 16 or the processing circuit 18.

[0035] The control circuit 16 is a circuit that controls the operation of the pixel control circuit 12 and the counter 14. For example, the control circuit 16 can refer to the value of a register and, based on this reference result, control the operation of at least one of the pixel control circuit 12 or the counter 14 to stop. The control to stop the operation can also be performed on a pixel-by-pixel basis.

[0036] For example, the control circuit 16 can control the operation of the pixel control circuit 12 and the counter 14 so that normal exposure and counting processing is performed on a pixel if the value of the register corresponding to that pixel is not equal to or greater than a predetermined value.

[0037] For example, if the value of a register corresponding to a pixel is greater than or equal to a predetermined value, the control circuit 16 can control the pixel to stop at least one operation performed by the pixel control circuit 12 or counter 14 for that pixel. Furthermore, at subsequent timings, the control circuit 16 can control the pixel control circuit 12 or counter 14 to stop at least one operation related to acquiring information for regions where the register value is greater than or equal to a predetermined value for pixels that are consecutive in the scan direction. By stopping at least one operation, the overall power consumption of the solid-state imaging device 1 can be reduced.

[0038] As described above, the control circuit 16 can perform control to stop at least one of the operations of the pixel control circuit 12 or the counter 14 by referring to the value of the register. At least one of the operations of the pixel control circuit 12 or the counter 14 may include, for example, controlling light reception (imaging) at the pixel, or acquiring data from the pixel. Acquiring data from the pixel may include operations that directly or indirectly affect data acquisition, such as converting the data output from the pixel into pulses, or counting pulses.

[0039] The control circuit 16 can also refer to the register value output from the counter 14, rather than actively referencing the register value itself. That is, the register value may be actively read by the control circuit 16, or the value read by the counter 14 may be output to the control circuit 16.

[0040] Furthermore, the control circuit 16 may include memory. The control circuit 16 may record in its memory when the register value exceeded a predetermined value. This control circuit 16 may be formed as part of the pixel circuit, or it may be located outside the pixel array 10.

[0041] More specifically, the control circuit 16 may include a sequential circuit configuration. The control circuit 16 can, for example, receive a clock signal and / or an enable signal from the outside and adjust the timing of control such as stopping the operations of the pixel control circuit 12 and / or the counter 14.

[0042] The processing circuit 18 can operate as a circuit that converts the values of the registers storing the results counted by the counter 14 into respective pixel values and outputs them. Also, at least a part of the processing executed by the control circuit 16 can be realized by the processing circuit 18.

[0043] For example, when the value of the register is greater than or equal to a predetermined value, the processing circuit 18 can operate as an output circuit that converts the value stored in the register by a predetermined operation and outputs it according to the timing when the value of the register becomes greater than or equal to the predetermined value.

[0044] <2. Pixel and Register>

[0045] FIG. 2 is a top view schematically showing an example of the configuration of the pixel array 10 according to an embodiment. In the pixel array 10, pixels 100 are arranged in an array, for example, in a first direction and a second direction intersecting the first direction. Here, the first direction is the line direction, and the second direction is the scan direction. That is, by scanning the pixels 100 continuous in the first direction in the second direction, imaging of an object to be imaged, such as an object, is realized.

[0046] By providing a plurality of pixels 100 in the second direction, information about the same region of the target can be obtained in multiple portions. For example, the pixels 100 continuous in the second direction continuously image the same target or the same region of the target for each unit time of imaging (exposure).

[0047] The figure on the right shows the pixels 100 that are continuous in a certain second direction, with the pixels enclosed by a dotted line extracted. For example, pixel 100a, which is the first pixel in the second direction, is the pixel that first acquires information from the object among the pixels that are continuous in the second direction to which the pixel belongs. Pixel 100b acquires information of the same object one unit time after the acquisition timing of the information of pixel 100a. Similarly, pixel 100c acquires information one unit time after pixel 100b, pixel 100d acquires information one unit time after pixel 100c, pixel 100e acquires information one unit time after pixel 100d, pixel 100f acquires information one unit time after pixel 100e, pixel 100g acquires information one unit time after pixel 100f, and pixel 100h acquires information from the same region one unit time after pixel 100g, respectively.

[0048] The solid-state imaging device 1 has registers corresponding to each pixel 100. Values corresponding to the number of photons detected in each pixel 100 are stored in these registers. The counter 14 described in FIG. 1 can add the number of photons counted to the register corresponding to each pixel 100. Further, the counter 14 can transfer (copy) the added count value to the register corresponding to the pixel 100 that acquires information of the same region one unit time later.

[0049] For example, the counter 14 stores the value obtained by counting the output from pixel 100a during a certain period in the register corresponding to pixel 100a, and transfers this value to the register corresponding to pixel 100b so that it can be used by pixel 100b one unit time later. For example, at the timing one unit time later, the counter 14 adds and stores the value obtained by counting the output from pixel 100b in the register corresponding to pixel 100b, and transfers this value to the register corresponding to pixel 100c so that it can be used by pixel 100c one unit time later.

[0050] To properly implement this process, for example, counter 14 adds the value obtained by counting the output from pixel 100h to the value of the register corresponding to pixel 100h, and outputs it via control circuit 16. Counter 14 then transfers the value from the register of pixel 100g to the register of pixel 100h at a later timing. By processing from the back of the scan in this way, counting and transfer can be achieved seamlessly.

[0051] By counting and transferring data at each unit of time, for example, the value acquired at pixel 100a can be counted along with the values ​​acquired at each pixel, and the sum can be output from the register corresponding to pixel 100h. By adding the outputs from multiple pixels 100 using a register in this way, appropriate quantization can be achieved, and the signal-to-noise ratio (SNR) of the output signal can be improved.

[0052] Furthermore, the method is not limited to using up to pixel 100h. For example, instead of using all the pixels aligned in the second direction, it is possible to start from pixel 100a and extract signals from intermediate pixels. This control of the number of pixels may be performed, for example, in accordance with the set value of the maximum exposure time.

[0053] On the other hand, when acquiring information for areas with high brightness, if information for all pixels in the second direction is acquired, there is a possibility that the count value for that area may overflow at one of the pixels. If processing is performed for all pixels 100 belonging to the same second direction when an overflow occurs, power consumption will increase. To avoid this, this disclosure performs processing when an overflow is detected.

[0054] The number of bits forming a single register can be set arbitrarily. This can be set according to the data communication speed, data capacity, the size of the data value, etc. For example, if the size of the register is set to 16 bits, 12 bits can be used as the first area where the count value is stored, and the remaining 4 bits can be used as the second area where the timing of the count value overflow is stored. For example, if one capture is 16 [usec], then it is sufficient to be able to quantize about 4000 photons per capture, so the first area can be set to 12 bits as described above.

[0055] In other words, the solid-state imaging device 1 may include a first region for storing count values ​​as registers corresponding to each of the pixels 100, and a second region for storing values ​​indicating whether the stored value is greater than or equal to a predetermined value and the timing at which it becomes greater than or equal to the predetermined value.

[0056] The following explanation will describe a case where the first area of ​​the register is 12 bits and the second area is 4 bits, but the bit width setting is not limited to this. Also, although we have said that the areas of a single register are divided, this is not limited to the case where there is a memory area to store the count value corresponding to the pixel and a memory area to store when the count value has overflowed and the timing.

[0057] Figure 3 is a schematic diagram illustrating an example of multiple exposure according to one embodiment. The asterisk indicates the target T whose information is acquired at pixel 100a at timing t0. The vertical axis represents time, and the horizontal axis represents the relative position of the sensor that acquires the photons of target T. That is, pixels at the same coordinate on the horizontal axis acquire information about the same target at the same position.

[0058] First, the register corresponding to pixel 100a is initialized in the initial state. Pixel 100a detects a photon related to target T at time t0. Counter 14 counts the photon based on the output from pixel 100a and stores it in the first area of ​​the register corresponding to pixel 100a. Counter 14 transfers the value of the register corresponding to pixel 100a to the register corresponding to pixel 100b at the appropriate timing, for example, after the values ​​stored in each register from pixel 100b to pixel 100h have been appropriately transferred to the next register.

[0059] Pixel 100b detects a photon related to target T at time t1 after a short period of time. Counter 14 counts the photon in conjunction with the output from pixel 100b and adds this count to the value in the first area of ​​the register corresponding to pixel 100b where the transferred value is stored, and stores it. Counter 14 transfers the value of the register corresponding to pixel 100b to the register corresponding to pixel 100c at an appropriate timing, for example, after the values ​​stored in each register from pixel 100c to pixel 100h have been appropriately transferred to the next register.

[0060] By performing this process sequentially, after time t7, the first region of the register stored in pixel 100h can be obtained to contain the number of photons of the target T detected by pixels 100a to 100h. Based on this photon count, processing is added when the value in the first region exceeds a predetermined value.

[0061] <3. Conversion of count values ​​and other control>

[0062] Next, the photon counting of the solid-state imaging device 1 in this disclosure and the conversion of the counted values ​​to output data will be described. In Figures 2 and 3, there are eight consecutive pixels 100 in the second direction, but this is not limited to these. In the following description, there are 16 or more consecutive pixels 100 in the second direction, and of these, 16 consecutive pixels 100 from the start of imaging will be described.

[0063] (First Embodiment)

[0064] Figures 4 to 8 show an example of the relationship between register values ​​and output values ​​according to one embodiment. In these figures, the time axis is taken to the right, and the register values ​​corresponding to pixels L0 to LF that are continuous in the second direction, acquiring information for the same region for each unit time, are shown along the time axis.

[0065] For illustrative purposes, the register is configured with 16 bits, with the first area consisting of the lower 12 bits and the second area consisting of the upper 4 bits. However, the register configuration is not limited to this configuration; any configuration that can appropriately obtain similar counting and timings exceeding a predetermined value is acceptable.

[0066] For example, counter 14 sets the second region based on whether the most significant bit b11 of the first region is 0 or 1. If the value of bit b11 is 1, counter 14 sets a predetermined bit in the second region to 1 based on the timing, that is, the elapsed time from when the first photon is acquired from the target (region) until a predetermined value is exceeded.

[0067] Note that the determination is made based on whether the most significant bit of the first area is 0 or 1, but this is just an example and not the only way to proceed. What is important here is whether the value of the first area exceeds a predetermined threshold, and the timing at which the most significant bit becomes 1 is given as an example of exceeding this predetermined threshold. Similarly, the diagrams and the following explanations also perform processing based on the value of the most significant bit, but all of these can be appropriately reinterpreted as "processing based on whether or not a predetermined value has been exceeded."

[0068] Figure 4 shows an example where bit b11 in the register corresponding to pixel L0 is 1, that is, the photon count value of pixel L0 is greater than or equal to a predetermined value (in this case, 2048). Counter 14 refers to the value of the register corresponding to pixel L0, counts the photons incident on pixel L0 based on the output from pixel L0, adds it to the register value, and stores it. In the initial state, it is desirable that the value of the register corresponding to pixel L0 be initialized to 0, and in particular, that the second region be initialized to 0.

[0069] If counter 14 counts the most significant bit b11 of the first region to be 1 at this time, it writes 1 to the most significant bit of the second region at a time before transferring the value of the register corresponding to pixel L0 to the register corresponding to pixel L1, or at a time before obtaining information about the same object in pixel L1.

[0070] In other words, if bit b11 is 1 at the time of counting of pixel L0, which is a different timing from that in Figure 4, then counter 14 may set the most significant bit of the second region to 1 at that time. In other words, in the state of Figure 4, the most significant bit of the register corresponding to pixel L0 may be set to 1.

[0071] If the second region has not been initialized, counter 14 can also initialize the second region to 0 when the counting of photons for pixel L0 is complete, and then execute the above process.

[0072] The control circuit 16 refers to the value of the register and, if a bit of 1 exists in the second region of the register, stops at least one of the operations of the pixel 100 corresponding to that register, or the counting of photons based on the output from that pixel 100. In the case of Figure 4, since a bit of 1 exists in the second region, the control circuit 16 controls the operation to stop one of the photon counting operations for pixels that were scheduled to count photons from pixel L1 onward.

[0073] The control circuit 16 may, for example, control the exposure of the corresponding pixel L1, etc., to prevent photons from being incident. The control circuit 16 may, for example, control the pixel control circuit 12 to prevent the formation of pulses in the signal output by the pixel L1, etc. The control circuit 16 may, for example, control the photon counting in the counter 14 that counts photons from the output of the pixel L1, etc. It may also control the stopping of at least two of the above processes in combination. From the viewpoint of reducing power consumption, it is desirable for the control circuit 16 to control the stopping of at least both the exposure and the counting process in the counter 14.

[0074] Even if the control circuit 16 controls the counter 14 to stop counting, the counter 14 may still perform the transfer of the register values ​​corresponding to each pixel. Alternatively, the transfer of register values ​​may be performed by another circuit at an appropriate timing, rather than by the counter 14.

[0075] In this case, the output of the first region of the register corresponding to pixel LF will ultimately be the same as the count value of the photons counted at pixel L0. The processing circuit 18 converts the value stored in the first region by referring to the value in the second region and outputs it as the number of photons corresponding to the pixel corresponding to the final target region. For example, the processing circuit 18 can output the value stored in the first region multiplied by a constant depending on the value stored in the second region.

[0076] For example, in Figure 4, a bit sequence of 1000 is stored in the first region. In this case, the processing circuit 18 focuses on the number of bits from the least significant bit of the set bits in the second region and determines to shift by 4 bits. It then shifts the bits stored in the first region left by 4 bits (multiplying the value by 16) and converts them into an output value, which is then output as the pixel value of the target.

[0077] For example, if bit b11 is 1 in the first pixel in the scanning direction, it is considered that the brightness of the object being imaged is very high. In this case, it can be assumed that the value will overflow, and since determining the value without referring to the lower bits does not pose a major problem, the above process can be performed.

[0078] If the most significant bit b11 of the photon count value for pixel L0 is 0, processing from the subsequent pixel L1 is executed. Figure 5 shows an example of the register values ​​in this process. In Figure 5, as an example, the count value of pixel L0 does not exceed a predetermined value, and when the count value of pixel L1 is added to the count value of pixel L0, it exceeds the predetermined value.

[0079] When counter 14 determines that bit b11 in pixel L1 has become 1, it sets the second most significant bit in the second area to 1. It is desirable that this setting of the bit value in the second area be performed before processing related to pixel L2 begins, as in the case of Figure 4. As a result of this process, 0100 is stored in the second area of ​​the register corresponding to pixel L2.

[0080] Since the second region contains 1, the control circuit 16 performs control to stop one of the processes related to photon counting from pixel L2 onwards, similar to the above.

[0081] The processing circuit 18, in order to obtain the output value, refers to the register corresponding to pixel LF to which the register value at the stage when counting at pixel L1 is completed has been transferred. The processing circuit 18 refers to the value 0100 in the second area and converts the value in the first area to the output value by left-shifting it by 3 bits (multiplying the value by 8).

[0082] Furthermore, the solid-state imaging device 1 does not need to perform the same processing on pixel L2, or it may perform the same processing even if bit b11 is 1 on pixel L2. This is because, in situations where flicker is not occurring, if the sum up to pixel L1 is not equal to or greater than a predetermined value, it is highly likely that the sum up to pixel L2 will also not be equal to or greater than a predetermined value. The same reason applies below. On the other hand, when taking measures against flicker, it is possible to address this by performing the same processing on pixel L2 as well.

[0083] If bit b11 is 0 in pixel L2, processing from pixel L3 is executed. Figure 6 shows an example of the register values ​​in this process. In Figure 6, the case where the sum of the count values ​​up to pixel L2 and the count value of pixel L3 exceeds a predetermined value is shown.

[0084] When counter 14 determines that bit b11 in pixel L3 has become 1, it sets the third bit from the most significant in the second area to 1. It is desirable that this setting of the bit value in the second area be performed before processing related to pixel L4 begins, as in the case of Figure 4. As a result of this process, 0010 is stored in the second area of ​​the register corresponding to pixel L4.

[0085] Since the second region contains 1, the control circuit 16 performs control to stop one of the processes related to photon counting from pixel L4 onwards, similar to the above.

[0086] The processing circuit 18, in order to obtain the output value, refers to the register corresponding to pixel LF to which the value of the register at the stage when counting at pixel L3 is completed has been transferred. The processing circuit 18 refers to the value 0010 in the second area and converts the value in the first area to the output value by left-shifting it by 2 bits (multiplying the value by 4).

[0087] Note that the solid-state imaging device 1 does not need to perform the same processing for pixels L4, L5, or L6, or it may perform the same processing if bit b11 is 1 in pixels L4, L5, and / or L6. If bit b11 is 0 in pixel L7, processing from pixel L7 is performed. Figure 7 shows an example of the register values ​​in this processing. In Figure 7, the case where the count value of pixel L7 is added to the count values ​​up to pixel L6 is shown to be greater than or equal to a predetermined value.

[0088] When counter 14 determines that bit b11 in pixel L7 has become 1, it sets the fourth bit from the most significant in the second area to 1. It is desirable that this setting of the bit value in the second area be performed before processing related to pixel L8 begins, as in the case of Figure 4. As a result of this process, 0001 is stored in the second area of ​​the register corresponding to pixel L8.

[0089] Since the second region contains 1, the control circuit 16 performs the same control as above to stop one of the processes related to photon counting from pixel L8 onwards.

[0090] The processing circuit 18, in order to obtain the output value, refers to the register corresponding to pixel LF to which the register value at the stage when counting at pixel L7 is completed has been transferred. The processing circuit 18 refers to the value 0001 in the second area, and then left-shifts the value in the first area by 1 bit (doubles the value) to convert it into the output value.

[0091] The solid-state imaging device 1 does not need to perform the same processing for pixels L8, L9, LA, LB, LC, LD, LE, or LF, or it may perform the same processing if bit b11 is 1 in pixels L8, L9, LA, LB, LC, LD, LE, and / or LF. If bit b11 is 0 in any pixel, as shown in Figure 8, the processing circuit 18 can use the value of the first region, which is the sum of the count values ​​up to pixel LF, as the output value.

[0092] As described above, according to this embodiment, by stopping any of the photon counting processes, including imaging, in areas with high brightness values ​​that do not significantly affect the image, it is possible to acquire brightness values ​​in areas where the brightness value is appropriate and above a predetermined value without reducing the resolution in areas where the brightness value does not exceed a predetermined value (e.g., 2048), while also reducing power consumption and maintaining a high signal-to-noise ratio. As a result, it is possible to generate a high dynamic range image without extreme degradation of the signal-to-noise ratio for each grayscale.

[0093] For example, in Figure 4, the count values ​​from 65520 to 32768 can be appropriately represented; in Figure 5, from 32760 to 16384; in Figure 6, from 16380 to 8192; in Figure 7, from 8190 to 4096; and in Figure 8, from 4095 to 0.

[0094] Furthermore, since the number of bits to be quantized can be kept constant, the number of bits in the register to be held can be fixed to a predetermined value, making it possible to form the register and other circuits for implementing this control in the same layer as or directly below the circuit for controlling the pixels. As a result, the device itself can be miniaturized.

[0095] For example, when forming a circuit beneath a pixel, there are limitations to the number of circuits that can be mounted. However, by using this embodiment, even if the number of pixels to be integrated increases, it is not necessary to increase the number of registers, thus enabling the expansion of exposure.

[0096] Furthermore, even if flicker is present in the target, it is possible to stop subsequent processing at the time the value of the area where the flicker occurs is obtained, and to increase the brightness value of the flicker-causing area.

[0097] In this embodiment, all bits in the second region were initialized to 0, but this is not the only way to do so. For example, the control circuit 16 can check whether or not to execute processing for each pixel, starting from the most significant bit in the second region, and stop processing according to the position of the 1 bit.

[0098] Specifically, the control circuit 16 can be defined to, for example, stop if the most significant bit of the second region is 1 in pixel L1. In this case, it is not necessary to refer to the other bit values ​​of the second region. Similarly, the control circuit 16 can be defined to, for example, stop if the most significant bit or the second-to-last bit of the second region is 1 in pixels L2 and L3, and it is not necessary to refer to the values ​​of the third and fourth bits of the second region. Thus, the values ​​of bit b11 in the first region sequentially set the values ​​from the most significant bit of the second region, and the control circuit 16 may decide whether or not to perform the stop process depending on the position of this 1 in the second region.

[0099] Furthermore, while the second region is set to 4 bits, it is not limited to this. For example, in the above example, the processing circuit 18 could expand the dynamic range by achieving a maximum 4-bit shift. Therefore, it is sufficient to set a second region in the register that represents values ​​from 0 to 4.

[0100] In other words, in this embodiment, the first region may consist of 3 bits. In this case, the counter 14 can, for example, set the second region to 100 in the case of Figure 4, 011 in the case of Figure 5, 010 in the case of Figure 6, 001 in the case of Figure 7, and 000 in the case of Figure 8. The control circuit 16 can stop the processing related to the photon counting of subsequent pixels if there is a 1 in the bit. The processing circuit 18 can obtain the number of bits to shift the first region by referring to the value of the second region.

[0101] This configuration makes it possible to further reduce the number of bits in the register or increase the number of bits in the first area. Increasing the number of bits in the first area reduces the power consumption reduction effect, but it makes it possible to improve the resolution in the low-brightness area. In other words, it expands the scope for balancing power consumption reduction and brightness resolution, allowing for more flexible design.

[0102] (Second Embodiment)

[0103] In the above-described embodiment, the brightness value received by each pixel remained constant, but it is also possible to vary the brightness value received by each pixel. For example, the first pixel or multiple pixels in the scanning direction may be set to acquire a lower brightness value (number of photons) than the other pixels.

[0104] Figure 9 shows an example of the relationship between the register value and the output value according to one embodiment. The pixel array 10 may include, for example, attenuation pixels H0 and M0 as pixels that image the target before pixel L0. This shows, for example, in Figure 2, that attenuation pixels are provided to the right of pixel 100a in the second direction.

[0105] Attenuation pixels are set to have lower sensitivity than other pixels but are sparse. Attenuation pixel H0 is, for example, a pixel formed so that the brightness value it receives is 1 / 4 of that of pixels L0 to LF, and attenuation pixel M0 is, for example, a pixel formed so that the brightness value it receives is 1 / 2 of that of pixels L0 to LF. Note that the system may not have attenuation pixels H0, or it may not have attenuation pixels M0. Furthermore, it may have three or more attenuation pixels, for example, pixels whose brightness value is 1 / 8 before attenuation pixel H0.

[0106] As shown in Figure 9, the same processing as for pixel L0 is performed for the light-reducing pixels H0 and M0. The second area of ​​the register can be, for example, 6 bits, with the most significant bit indicating the comparison result with a predetermined value in light-reducing pixel H0, the second most significant bit indicating the comparison result with a predetermined value in light-reducing pixel M0, and the lower 4 bits can be the same as the second area in the first embodiment described above.

[0107] The processing circuit 18 can obtain the brightness value (number of photons) by left-shifting the value in the first region of the register corresponding to the pixel LF by up to 6 bits, referencing the value in the second region.

[0108] By using a light-receiving pixel H0 with 1 / 4 the light-receiving performance as the pixel for the initial imaging of the target, the photon counting process can be stopped when photons corresponding to 262080 to 131072 counts are detected. Therefore, by using a light-receiving pixel, it becomes possible to achieve more efficient power consumption reduction processing in high-brightness regions (simple calculations show a reduction in the number of photon counts to 1 / 64).

[0109] Similarly, by using a light-receiving pixel M0 with half the light-receiving performance as the pixel following the light-receiving pixel H0, the photon counting process can be stopped when a photon corresponding to 131040 to 65536 counts is detected. Therefore, in simple calculations, the number of photon counts can be reduced to 1 / 32.

[0110] Figure 10 schematically shows an example of a light-reducing pixel and other pixels according to one embodiment. For example, each pixel 100 is formed with four microlenses, in a non-limiting example. Other examples include a configuration with fewer or more microlenses, such as two, nine, or sixteen.

[0111] Pixels L0 through LF are formed as pixels that utilize all of these microlenses.

[0112] The light-reducing pixel M0 may be formed, for example, by shielding a region covering half of these microlenses. The light-reducing pixel H0 may be formed, for example, by shielding a region covering three-quarters of these microlenses.

[0113] However, the formation of attenuation pixels is not limited to this, and for example, they may be formed as pixels equipped with an ND filter or the like on the incident surface side of the photoelectric conversion region.

[0114] (Third embodiment)

[0115] In the embodiments described above, the pixel is shown in a form where color differences are not taken into account by color filters or the like. However, the same processing can be achieved even when color filters, such as the three primary colors of RGB, are appropriately provided.

[0116] Figure 11 shows an example of the relationship between register values ​​and output values ​​according to one embodiment. Pixels G0, G2, G4, G6, G8, and GA are equipped with green filters and are pixels for counting photons corresponding to green light; pixels R1, R5, and R9 are equipped with red filters and are pixels for counting photons corresponding to red light; and pixels B3, B7, and BB are equipped with blue filters and are pixels for counting photons corresponding to blue light.

[0117] After the counting of the output from pixel G0 is complete, counter 14 transfers the value of the first area of ​​the register to the first area of ​​the register corresponding to pixel G2. After the counting of the output from pixel R1 is complete, counter 14 transfers the value of the first area of ​​the register to the first area of ​​the register corresponding to pixel R5. After the counting of the output from pixel G2 is complete, counter 14 transfers the value of the first area of ​​the register to the first area of ​​the register corresponding to pixel G4. After the counting of the output from pixel B3 is complete, counter 14 transfers the value of the first area of ​​the register to the first area of ​​the register corresponding to pixel B7.

[0118] In this way, counter 14 transfers the value from the first area of ​​the register to the first area corresponding to the next pixel of the same color. This transfer enables data acquisition and photon counting for each color. In this example, twice as many green pixels are placed as red and blue pixels. In this case, four green pixels may be placed, just like the other pixels, and for example, the value of pixel G0 is transferred to pixel G4, the value of pixel G2 is transferred to pixel G6, and so on, and finally they are added together.

[0119] The counter 14 may, for example, transfer the value of the second region to the first region of the register of pixel R1. The control circuit 16 may, for example, stop at least one process related to photon counting for pixels from pixel R1 onward if bit b11 is 1 in pixel G0. In this case, for example, the final processing circuit 18 may shift the count value at pixel G0 to the left by a shift amount according to the value in the second region to obtain the intensities of red, green, and blue light, respectively. The same applies when the count value at pixel R1 or pixel B3 exceeds a predetermined value.

[0120] As another example, as shown in Figure 12, the counter 14 may transfer the value of the second region for each color in addition to the first region. For example, if the count value in pixel G0 exceeds a predetermined value, the value of the second region of pixel G2 is set to 1000. In this case, since there is no 1 in the second region of pixel R1, the counter 14 performs photon counting from the output of pixel R1 and transfers this result to the register corresponding to pixel R5. The same applies to blue.

[0121] In this case, the control circuit 16 can control the stopping of processing for each color. The processing circuit 18 can obtain the brightness value including the overflow for each color by referring to the register corresponding to the pixel that detected the last photon for each color and left-shifting the value in the first region based on the value in the second region.

[0122] In the configuration shown in Figure 11, processing can be stopped quickly in the event of an overflow. On the other hand, in the configuration shown in Figure 12, even if one color overflows, the reproduction of other colors can be achieved more accurately.

[0123] Although the above uses the three primary colors RGB, it is not limited to these. For example, it may further include pixels that receive white light, or pixels that receive light of other colors.

[0124] If the system further includes pixels that receive white light, when the number of photons in the white light-receiving pixels exceeds a predetermined number, a bit can be set in the second region. Based on this set bit, it is possible to control the stopping of subsequent photon counting processes in the pixels and to perform a bit shift in the processing circuit 18 for the final result.

[0125] Pixels that receive white light may be set as attenuating pixels. For example, they may be set to have 1 / 3 the sensitivity of pixels of other colors, or 1 / 4 the sensitivity of pixels of other colors. If the sensitivity of pixels that receive white light is 1 / 4 the sensitivity of pixels of other colors, then, as in the example in Figure 11, the green pixels can also be configured to add the count value every 4 pixels, as described above.

[0126] Furthermore, even when a color filter is provided, it is of course possible to provide attenuation pixels for each pixel that receives light of each color, as in the second embodiment described above, and the same effects as in the second embodiment with attenuation pixels can be achieved.

[0127] (Fourth Embodiment)

[0128] In each of the above configurations, the processing circuit 18 converted the value in the first region by left bit shifting it based on the value in the second region as the final output. However, the processing circuit 18 may perform further processing on the value in the first region before converting it to an output value.

[0129] For example, let C be the count value (the value in the first area of ​​the register), T be the total exposure time, and A be the time until a predetermined value is exceeded. In this case, the expected count value Cp is given by the following formula.

[0130] The processing circuit 18 may convert the value in the first area of ​​the register that was finally obtained based on equation (1) to obtain the output value.

[0131] Furthermore, dark current can also be taken into consideration. For example, optical black pixels can be placed, and the count value Cd of these optical black pixels can be subtracted from C.

[0132] The optical black may be placed, for example, at least one of the beginning or end of the scan direction (second direction).

[0133] Furthermore, it is also possible to use nonlinear count values, for example. If the count values ​​are nonlinear, the processing circuit 18 can correct this nonlinearity before obtaining the output value. It is also possible to obtain the output value by performing various calculations, such as correction for noisy pixels that emit abnormal pulses and sensitivity correction for color filters.

[0134] (Fifth embodiment)

[0135] Figure 13 is a schematic diagram showing an example of a pixel peripheral circuit according to one embodiment. A pixel unit is defined as a unit including a pixel 100, a pixel control circuit 12, a counter 14, and a control circuit 16. SPAD 110 is an example of a photoelectric light-receiving element provided in the pixel 100.

[0136] The signal output from SPAD 110 is waveform-formed by pixel control circuit 12 and output as pulses to counter 14. Counter 14 counts the pulses output from pixel control circuit 12 and writes them to a register. Control circuit 16 can control SPAD 110, pixel control circuit 12, and / or counter 14 by referring to the register. Control circuit 16 may also control the optical system related to SPAD 110. These processes are as described in the embodiments described above.

[0137] Counter 14 can be connected to counter 14 of other pixel units via read line 140. As shown in the example in Figure 4, for example, counter 14 of pixel unit LE can output the count value, added by referencing a register, to counter 14 of pixel unit LF via read line 140. In this way, counter 14 can transfer the value of a register to other pixel units.

[0138] The readout line 140 may be provided as a single line for consecutive pixels along the second direction, or it may be provided as a multiple line.

[0139] As another example, as shown in Figure 14, the counters 14 may be directly connected to each other between the pixel units. In this configuration, a multiplexer (not shown) may be placed between the counters.

[0140] In this case, as shown in Figure 15, if a color filter is provided, the counters 14 may be directly connected to each other for each color. For example, the pixel unit (G) is a pixel unit that receives green light, and the wiring may be arranged so that the counters 14 of this pixel unit (G) are connected to each other. Even when a color filter is provided, as explained in the above embodiment, the connection of the registers is not limited to the configuration in Figure 15, and various wiring methods suitable for different situations can be used.

[0141] Figure 16 shows an example of the configuration of the solid-state imaging device 1 in this disclosure. For example, the solid-state imaging device 1 rotates a polyhedron 20, which is provided inside or outside the device, at a predetermined speed. The pixel array 10 receives light reflected from the surface of the polyhedron 20 and counts photons. By rotating the polyhedron 20 at an appropriate angular velocity, the pixel array 10 can sequentially acquire information in a predetermined area of ​​a target using pixels that are appropriately arranged in a second direction, for example, as shown in Figure 3.

[0142] <4. Examples of Application>

[0143] The solid-state imaging device 1 according to each embodiment described above can be mounted as an application in various electronic devices. For example, the electronic device may be a device that generates an image using weak light reflected from an object in a dark environment, weak light transmitted through an object, or weak light emitted from an object.

[0144] Furthermore, as one example without limitation, the electronic device may include, for example, a solid-state imaging device 1 and a light-emitting unit, and may operate as a device that primarily acquires reflected light from light emitted from the light-emitting unit onto a target. For example, by equipping the pixels with infrared filters, it is possible to have a light-emitting unit that emits infrared light, and to function as a solid-state imaging unit that receives the infrared light emitted from this light-emitting unit.

[0145] The technology disclosed herein can be applied to a variety of products. For example, the technology disclosed herein may be implemented as a device mounted on any type of mobile vehicle, such as automobiles, electric vehicles, hybrid electric vehicles, motorcycles, bicycles, personal mobility devices, airplanes, drones, ships, robots, construction machinery, or agricultural machinery (tractors).

[0146] Figure 17 is a block diagram showing a schematic configuration example of a vehicle control system 7000, which is an example of a mobile control system to which the technology described herein may be applied. The vehicle control system 7000 comprises a plurality of electronic control units connected via a communication network 7010. In the example shown in Figure 17, the vehicle control system 7000 comprises a drive system control unit 7100, a body system control unit 7200, a battery control unit 7300, an external information detection unit 7400, an internal information detection unit 7500, and an integrated control unit 7600. The communication network 7010 connecting these plurality of control units may be an in-vehicle communication network compliant with any standard such as CAN (Controller Area Network), LIN (Local Interconnect Network), LAN (Local Area Network), or FlexRay (registered trademark).

[0147] Each control unit comprises a microcomputer that performs calculations according to various programs, a storage unit that stores programs executed by the microcomputer or parameters used in various calculations, and a drive circuit that drives various controlled devices. Each control unit is equipped with a network interface for communication with other control units via the communication network 7010, and a communication interface for communication with devices or sensors inside or outside the vehicle via wired or wireless communication. Figure 17 illustrates the functional configuration of the integrated control unit 7600, which includes a microcomputer 7610, a general-purpose communication interface 7620, a dedicated communication interface 7630, a positioning unit 7640, a beacon receiver 7650, an in-vehicle equipment interface 7660, an audio / image output unit 7670, an in-vehicle network interface 7680, and a storage unit 7690. Other control units similarly include a microcomputer, a communication interface, and a storage unit.

[0148] The drivetrain control unit 7100 controls the operation of devices related to the vehicle's drivetrain according to various programs. For example, the drivetrain control unit 7100 functions as a control device for generating driving force for the vehicle, such as an internal combustion engine or drive motor; a driving force transmission mechanism for transmitting driving force to the wheels; a steering mechanism for adjusting the steering angle of the vehicle; and a braking device for generating braking force for the vehicle. The drivetrain control unit 7100 may also function as a control device such as ABS (Antilock Brake System) or ESC (Electronic Stability Control).

[0149] A vehicle state detection unit 7110 is connected to the drivetrain control unit 7100. The vehicle state detection unit 7110 includes, for example, a gyro sensor for detecting the angular velocity of the vehicle's axial rotational motion, an acceleration sensor for detecting the vehicle's acceleration, or at least one of the sensors for detecting the amount of accelerator pedal operation, brake pedal operation, steering wheel steering angle, engine speed, or wheel rotation speed. The drivetrain control unit 7100 performs calculations using signals input from the vehicle state detection unit 7110 to control the internal combustion engine, drive motor, electric power steering system, brake system, etc.

[0150] The body system control unit 7200 controls the operation of various devices mounted on the vehicle body according to various programs. For example, the body system control unit 7200 functions as a control device for a keyless entry system, a smart key system, a power window system, or various lamps such as headlights, reverse lights, brake lights, turn signals, or fog lights. In this case, the body system control unit 7200 may receive radio waves transmitted from a portable device that replaces a key or signals from various switches. The body system control unit 7200 receives these radio waves or signals and controls the vehicle's door lock system, power window system, lamps, etc.

[0151] The battery control unit 7300 controls the secondary battery 7310, which is the power source for the drive motor, according to various programs. For example, the battery control unit 7300 receives information such as battery temperature, battery output voltage, or remaining battery capacity from the battery device equipped with the secondary battery 7310. The battery control unit 7300 uses these signals to perform calculations and controls the temperature of the secondary battery 7310 or the cooling device provided in the battery device.

[0152] The external information detection unit 7400 detects information from outside the vehicle equipped with the vehicle control system 7000. For example, the external information detection unit 7400 is connected to at least one of the imaging unit 7410 and the external information detection unit 7420. The imaging unit 7410 includes at least one of the following: a Time of Flight (ToF) camera, a stereo camera, a monocular camera, an infrared camera, and other cameras. The external information detection unit 7420 includes at least one of the following: an environmental sensor for detecting the current weather or climate, or an ambient information detection sensor for detecting other vehicles, obstacles, or pedestrians around the vehicle equipped with the vehicle control system 7000.

[0153] The environmental sensor may be at least one of the following: a raindrop sensor for detecting rain, a fog sensor for detecting fog, a sunshine sensor for detecting the degree of sunlight, and a snow sensor for detecting snowfall. The ambient information detection sensor may be at least one of the following: an ultrasonic sensor, a radar device, and a LIDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) device. These imaging unit 7410 and external information detection unit 7420 may be provided as independent sensors or devices, or as a device in which multiple sensors or devices are integrated.

[0154] Here, Figure 18 shows examples of the installation locations of the imaging unit 7410 and the external information detection unit 7420. The imaging units 7910, 7912, 7914, 7916, and 7918 are installed, for example, at least one of the following locations on the vehicle 7900: the front nose, side mirrors, rear bumper, tailgate, and the upper part of the windshield inside the passenger compartment. The imaging unit 7910 installed on the front nose and the imaging unit 7918 installed on the upper part of the windshield inside the passenger compartment mainly acquire images of the front of the vehicle 7900. The imaging units 7912 and 7914 installed on the side mirrors mainly acquire images of the sides of the vehicle 7900. The imaging unit 7916 installed on the rear bumper or tailgate mainly acquires images of the rear of the vehicle 7900. The imaging unit 7918, located at the top of the windshield inside the vehicle, is primarily used for detecting preceding vehicles, pedestrians, obstacles, traffic lights, traffic signs, or lanes.

[0155] Figure 18 shows an example of the imaging range of each imaging unit 7910, 7912, 7914, and 7916. Imaging range a shows the imaging range of imaging unit 7910 located on the front nose, imaging ranges b and c show the imaging ranges of imaging units 7912 and 7914 located on the side mirrors, respectively, and imaging range d shows the imaging range of imaging unit 7916 located on the rear bumper or back door. For example, by superimposing the image data captured by imaging units 7910, 7912, 7914, and 7916, an overhead view image of vehicle 7900 can be obtained.

[0156] The external information detection units 7920, 7922, 7924, 7926, 7928, and 7930, which are installed on the front, rear, sides, corners, and the upper part of the windshield inside the vehicle 7900, may be, for example, ultrasonic sensors or radar devices. The external information detection units 7920, 7926, and 7930, which are installed on the front nose, rear bumper, back door, and the upper part of the windshield inside the vehicle 7900, may be, for example, LIDAR devices. These external information detection units 7920 to 7930 are mainly used for detecting preceding vehicles, pedestrians, or obstacles.

[0157] Returning to Figure 17, the explanation continues. The external information detection unit 7400 causes the imaging unit 7410 to capture images of the area outside the vehicle and receives the captured image data. The external information detection unit 7400 also receives detection information from the connected external information detection unit 7420. If the external information detection unit 7420 is an ultrasonic sensor, radar device, or LIDAR device, the external information detection unit 7400 emits ultrasonic waves or electromagnetic waves and receives information on the received reflected waves. Based on the received information, the external information detection unit 7400 may perform object detection processing such as detecting people, vehicles, obstacles, signs, or characters on the road surface, or distance detection processing. Based on the received information, the external information detection unit 7400 may perform environmental recognition processing to recognize rainfall, fog, or road surface conditions. Based on the received information, the external information detection unit 7400 may calculate the distance to an object outside the vehicle.

[0158] Furthermore, the external information detection unit 7400 may perform image recognition processing or distance detection processing to recognize people, vehicles, obstacles, signs, or characters on the road surface based on the received image data. The external information detection unit 7400 may perform distortion correction or alignment processing on the received image data, and may also synthesize image data captured by different imaging units 7410 to generate an overhead view image or a panoramic image. The external information detection unit 7400 may also perform viewpoint transformation processing using image data captured by different imaging units 7410.

[0159] The in-vehicle information detection unit 7500 detects information inside the vehicle. The in-vehicle information detection unit 7500 is connected to, for example, a driver status detection unit 7510 that detects the driver's state. The driver status detection unit 7510 may include a camera that images the driver, a biosensor that detects the driver's biometric information, or a microphone that collects sounds inside the vehicle. The biosensor is installed, for example, on the seat or steering wheel and detects the biometric information of a passenger sitting in the seat or a driver holding the steering wheel. Based on the detection information input from the driver status detection unit 7510, the in-vehicle information detection unit 7500 may calculate the driver's level of fatigue or concentration, or determine whether the driver is dozing off. The in-vehicle information detection unit 7500 may perform processing such as noise cancellation on the collected audio signals.

[0160] The integrated control unit 7600 controls the overall operation of the vehicle control system 7000 according to various programs. An input unit 7800 is connected to the integrated control unit 7600. The input unit 7800 is implemented by a device that can be operated by the occupant, such as a touch panel, buttons, a microphone, a switch, or a lever. The integrated control unit 7600 may also receive data obtained by voice recognition of audio input via the microphone. The input unit 7800 may be, for example, a remote control device using infrared or other radio waves, or an external device such as a mobile phone or PDA (Personal Digital Assistant) that supports the operation of the vehicle control system 7000. The input unit 7800 may also be, for example, a camera, in which case the occupant can input information through gestures. Alternatively, data obtained by detecting the movement of a wearable device worn by the occupant may be input. Furthermore, the input unit 7800 may include, for example, an input control circuit that generates an input signal based on information input by a passenger or the like using the input unit 7800 and outputs it to the integrated control unit 7600. By operating this input unit 7800, passengers or the like can input various data to the vehicle control system 7000 or instruct it to perform processing operations.

[0161] The memory unit 7690 may include a ROM (Read Only Memory) for storing various programs executed by a microcomputer, and a RAM (Random Access Memory) for storing various parameters, calculation results, or sensor values. The memory unit 7690 may also be implemented using magnetic storage devices such as an HDD (Hard Disk Drive), semiconductor storage devices, optical storage devices, or magneto-optical storage devices.

[0162] The general-purpose communication interface 7620 is a general-purpose communication interface that mediates communication between various devices present in the external environment 7750. The general-purpose communication interface 7620 may implement cellular communication protocols such as GSM (Global System of Mobile communications), WiMAX (registered trademark), LTE (Registered trademark) (Long Term Evolution), or LTE-A (LTE-Advanced), or other wireless communication protocols such as wireless LAN (also known as Wi-Fi (registered trademark)) or Bluetooth (registered trademark). The general-purpose communication interface 7620 may connect, for example, to devices (e.g., application servers or control servers) located on an external network (e.g., the Internet, a cloud network, or a carrier-specific network) via a base station or access point. Furthermore, the general-purpose communication interface 7620 may connect to terminals located near the vehicle (for example, terminals belonging to the driver, pedestrians, or shops, or MTC (Machine Type Communication) terminals) using, for example, P2P (Peer To Peer) technology.

[0163] The Dedicated Communication Interface 7630 is a communication interface that supports communication protocols developed for use in vehicles. The Dedicated Communication Interface 7630 may implement standard protocols such as WAVE (Wireless Access in Vehicle Environment), DSRC (Dedicated Short Range Communications), or cellular communication protocols, which are combinations of lower-layer IEEE 802.11p and upper-layer IEEE 1609. The Dedicated Communication Interface 7630 typically performs V2X communication, a concept that includes one or more of the following: vehicle-to-vehicle communication, vehicle-to-infrastructure communication, vehicle-to-home communication, and vehicle-to-pedestrian communication.

[0164] The positioning unit 7640 performs positioning by receiving GNSS signals from GNSS (Global Navigation Satellite System) satellites (for example, GPS signals from GPS (Global Positioning System) satellites) and generates location information including the vehicle's latitude, longitude, and altitude. The positioning unit 7640 may also determine its current location by exchanging signals with a wireless access point, or it may acquire location information from a terminal such as a mobile phone, PHS, or smartphone that has positioning capabilities.

[0165] The beacon receiver 7650 receives radio waves or electromagnetic waves transmitted from, for example, a radio station installed on a road, and obtains information such as the current location, traffic congestion, road closures, or travel time. The functions of the beacon receiver 7650 may also be included in the dedicated communication interface 7630 described above.

[0166] The in-vehicle equipment interface 7660 is a communication interface that mediates connections between the microcomputer 7610 and various in-vehicle equipment 7760 located inside the vehicle. The in-vehicle equipment interface 7660 may establish a wireless connection using wireless communication protocols such as Wi-Fi, Bluetooth (registered trademark), NFC (Near Field Communication), or WUSB (Wireless USB). Furthermore, the in-vehicle equipment interface 7660 may establish wired connections such as USB (Universal Serial Bus), HDMI (Registered Trademark) (High-Definition Multimedia Interface), or MHL (Mobile High-Definition Link) via connection terminals (and cables if necessary) not shown. The in-vehicle equipment 7760 may include, for example, at least one of the following: a mobile device or wearable device owned by a passenger, or an information device brought into or installed in the vehicle. The in-vehicle equipment 7760 may also include a navigation device that performs route searching to any destination. The in-vehicle equipment interface 7660 exchanges control signals or data signals with these in-vehicle equipment 7760s.

[0167] The in-vehicle network interface 7680 is an interface that mediates communication between the microcomputer 7610 and the communication network 7010. The in-vehicle network interface 7680 transmits and receives signals and other data in accordance with a predetermined protocol supported by the communication network 7010.

[0168] The microcomputer 7610 of the integrated control unit 7600 controls the vehicle control system 7000 according to various programs based on information acquired via at least one of the following: general-purpose communication I / F 7620, dedicated communication I / F 7630, positioning unit 7640, beacon receiver 7650, in-vehicle equipment I / F 7660, and in-vehicle network I / F 7680. For example, the microcomputer 7610 may calculate control target values ​​for the drive force generator, steering mechanism, or braking device based on acquired in-vehicle and external information, and output control commands to the drivetrain control unit 7100. For example, the microcomputer 7610 may perform coordinated control aimed at realizing ADAS (Advanced Driver Assistance System) functions, including collision avoidance or impact mitigation, following based on distance between vehicles, maintaining vehicle speed, collision warning, or lane departure warning. Furthermore, the microcomputer 7610 may perform cooperative control for purposes such as autonomous driving, where the vehicle drives autonomously without driver intervention, by controlling the drive force generating device, steering mechanism, or braking device, etc., based on the acquired information about the vehicle's surroundings.

[0169] The microcomputer 7610 may generate three-dimensional distance information between the vehicle and surrounding structures, people, and other objects based on information acquired via at least one of the general-purpose communication I / F 7620, dedicated communication I / F 7630, positioning unit 7640, beacon receiver 7650, in-vehicle equipment I / F 7660, and in-vehicle network I / F 7680, and create local map information including surrounding information of the vehicle's current location. Furthermore, the microcomputer 7610 may predict dangers such as vehicle collision, proximity of pedestrians, or entry into a closed road based on the acquired information, and generate warning signals. These warning signals may, for example, be signals to generate a warning sound or illuminate a warning lamp.

[0170] The audio-image output unit 7670 transmits at least one of audio and image output signals to an output device capable of visually or audibly notifying information to the vehicle's occupants or to those outside the vehicle. In the example in Figure 17, the output devices are exemplified as an audio speaker 7710, a display unit 7720, and an instrument panel 7730. The display unit 7720 may include, for example, at least one of an onboard display and a head-up display. The display unit 7720 may have an AR (Augmented Reality) display function. The output devices may be other devices other than these, such as headphones, wearable devices such as glasses-type displays worn by occupants, projectors, or lamps. If the output device is a display device, the display device visually displays the results obtained from various processes performed by the microcomputer 7610 or information received from other control units in various formats such as text, images, tables, graphs, etc. Furthermore, if the output device is an audio output device, the audio output device converts the audio signal, consisting of the reproduced audio data or sound data, into an analog signal and outputs it audibly.

[0171] In the example shown in Figure 17, at least two control units connected via the communication network 7010 may be integrated into a single control unit. Alternatively, individual control units may be composed of multiple control units. Furthermore, the vehicle control system 7000 may include other control units not shown. Also, in the above description, some or all of the functions performed by one control unit may be assigned to other control units. In other words, as long as information is transmitted and received via the communication network 7010, predetermined calculation processing may be performed by any of the control units. Similarly, a sensor or device connected to one control unit may be connected to another control unit, and multiple control units may transmit and receive detection information to and from each other via the communication network 7010.

[0172] In the vehicle control system 7000 described above, the solid-state imaging device 1 according to this embodiment, as described with reference to Figures 1 to 16, can be applied to a part of the external information detection unit 7400 or the internal information detection unit 7500 of the application example shown in Figure 17.

[0173] Furthermore, at least some of the components of the solid-state imaging device 1 described using Figures 1 to 16 may be implemented in a module (for example, an integrated circuit module consisting of a single die) for the external information detection unit 7400 or the internal information detection unit 7500 shown in Figure 17. Alternatively, the solid-state imaging device 1 described using Figures 1 to 16 may be implemented by multiple control units of the external information detection unit 7400 or the internal information detection unit 7500 shown in Figure 17.

[0174] The embodiments described above may also take the following forms.

[0175] (1) A solid-state imaging device comprising: a plurality of pixels arranged in an array in a first direction and a second direction intersecting the first direction, and acquiring data of the same object by multiple exposure, shifting in the second direction at unit time intervals; a register provided for each of the plurality of pixels, storing the data acquired by each of the plurality of pixels; a control circuit for controlling imaging at the pixels and data acquisition from the pixels; and an output circuit for converting and outputting the values ​​stored in the registers, wherein the registers copy the values ​​of the registers of the pixels that acquired information of the same object before the unit time interval; the control circuit controls imaging and data acquisition at the pixel corresponding to the register when the value of the register is not greater than or equal to a predetermined value; controls stopping at least one of imaging or data acquisition at the pixel corresponding to the register when the value of the register is greater than or equal to the predetermined value; and the output circuit converts and outputs the values ​​stored in the registers by a predetermined calculation according to the timing when the value of the register becomes greater than or equal to a predetermined value.

[0176] (2) The solid-state imaging apparatus according to (1), wherein the pixel is a photon counting element.

[0177] (3) The solid-state imaging apparatus according to (2), wherein the register operates as a counter for counting photons incident on the pixel, and the counter adds a count value based on the output from the pixel to the stored value and stores it.

[0178] (4) The solid-state imaging apparatus according to (3), wherein the register comprises a first area for storing count values ​​and a second area for indicating that the stored value is greater than or equal to a predetermined value, and when the value in the first area exceeds the predetermined value, a value indicating that the value stored in the first area is greater than or equal to a predetermined value is set in the second area.

[0179] (5) The solid-state imaging apparatus according to (4), wherein the output circuit sets the value of the second region based on the timing at which the value of the first region exceeds the predetermined value.

[0180] (6) The solid-state imaging apparatus according to (5), wherein the output circuit outputs a value stored in the first region multiplied by a constant according to the value stored in the second region.

[0181] (7) The solid-state imaging apparatus according to (5) or (6), wherein the output circuit shifts the value stored in the first region to the left according to the value stored in the second region and outputs it.

[0182] (8) A solid-state imaging device according to any one of (5) to (7), wherein a value to be stored in the second region is set based on the elapsed time from when the target is first photographed until the first region exceeds a predetermined value.

[0183] (9) A solid-state imaging device according to any one of (1) to (8), wherein the leading pixel in the second direction for imaging the target is set to have a lower sensitivity than the other pixels in the second direction to which that pixel belongs.

[0184] (10) The solid-state imaging device according to (9), wherein the second pixel from the front in the second direction for photographing the target is set to have a higher sensitivity than the first pixel, and a lower sensitivity than the other pixels in the second direction to which the first pixel and the second pixel belong.

[0185] (11) The solid-state imaging device according to any one of (1) to (10), wherein each of the plurality of pixels is equipped with a color filter selected from a plurality of colors according to the arrangement of the pixels, data relating to light transmitted through the color filter is acquired, and processing relating to the value of the register is performed for each pixel equipped with the same color color filter.

[0186] (12) The solid-state imaging device according to any one of (1) to (10), wherein each of the plurality of pixels is equipped with a color filter selected from a plurality of colors according to the arrangement of the pixels, data relating to light transmitted through the color filter is acquired, and the processing relating to the value of the register is performed for each pixel that is capturing the same object regardless of the color transmitted by the color filter.

[0187] (13) Electronic device comprising: a plurality of pixels arranged in an array in a first direction and a second direction intersecting the first direction, and acquiring data of the same object by multiple exposure, shifting in the second direction at unit time intervals; a register provided for each of the plurality of pixels, storing the data acquired by each of the plurality of pixels; and a control circuit for controlling imaging at the pixels and acquiring data from the pixels, wherein the register copies the value of the register of the pixel that acquired information of the same object before the unit time interval; the control circuit controls imaging and data acquisition at the pixel corresponding to the register when the value of the register is not equal to or greater than a predetermined value; and controls stopping at least one of imaging or data acquisition at the pixel corresponding to the register when the value of the register is equal to or greater than the predetermined value.

[0188] (14) The electronic device according to (13), wherein the plurality of pixels, the register, and the control circuit can have the configurations described in (2) to (12).

[0189] The aspects of this disclosure are not limited to the embodiments described above, but include various conceivable variations, and the effects of this disclosure are not limited to those described above. The components in each embodiment may be appropriately combined and applied. That is, various additions, modifications, and partial deletions are possible, as long as they do not deviate from the conceptual idea and spirit of this disclosure derived from the claims and their equivalents.

[0190] 1: Solid-state imaging device, 10: Pixel array, 100: Pixel, 100a, 100b, 100c, 100d, 100e1, 100f, 100g, 100h: Pixel, L0, L1, L2, L3, L4, L5, L6, L7, L8, L9, LA, LB, LC, LD, LE, LF: Pixel, G0, R1, G2, B3, G4, R5, G6, B7, G8, R9, GA, BB: Pixel, H0, M0: Light-reducing pixels, 110: SPAD, 12: Pixel control circuit, 14: Counter, 140: Readout line, 16: Control circuit, 18: Processing circuit, 20: Multifaceted mirror

Claims

1. A solid-state imaging device comprising: a plurality of pixels arranged in an array in a first direction and a second direction intersecting the first direction, and acquiring data of the same object by shifting in the second direction by multiple exposures at unit time intervals; a register provided for each of the plurality of pixels and storing the data acquired by each of the plurality of pixels; a control circuit for controlling imaging at the pixels and data acquisition from the pixels; and an output circuit for converting and outputting the values ​​stored in the registers, wherein the registers copy the values ​​of the registers of the pixels that acquired information of the same object in the unit time intervals prior to the current time interval; the control circuit controls imaging and data acquisition at the pixel corresponding to the register when the value of the register is not greater than or equal to a predetermined value; controls stopping at least one of imaging or data acquisition at the pixel corresponding to the register when the value of the register is greater than or equal to the predetermined value; and the output circuit converts and outputs the values ​​stored in the registers by a predetermined calculation according to the timing when the value of the register becomes greater than or equal to a predetermined value.

2. The solid-state imaging apparatus according to claim 1, wherein the pixel is a photon counting element.

3. The solid-state imaging apparatus according to claim 2, wherein the register operates as a counter for counting photons incident on the pixel, and the counter adds a count value based on the output from the pixel to a stored value and stores it.

4. The solid-state imaging apparatus according to claim 3, wherein the register comprises a first area for storing count values ​​and a second area for indicating that the stored value is greater than or equal to a predetermined value, and when the value in the first area exceeds the predetermined value, a value indicating that the value stored in the first area is greater than or equal to a predetermined value is set in the second area.

5. The solid-state imaging apparatus according to claim 4, wherein the output circuit sets the value of the second region based on the timing at which the value of the first region exceeds the predetermined value.

6. The solid-state imaging apparatus according to claim 5, wherein the output circuit outputs a value stored in the first region multiplied by a constant according to the value stored in the second region.

7. The solid-state imaging apparatus according to claim 5, wherein the output circuit shifts the value stored in the first region to the left according to the value stored in the second region and outputs it.

8. The solid-state imaging apparatus according to claim 5, wherein a value to be stored in the second region is set based on the elapsed time from when the target is first photographed until the first region exceeds a predetermined value.

9. The solid-state imaging device according to claim 1, wherein the leading pixel in the second direction for photographing the target is set to have a lower sensitivity than the other pixels in the second direction to which that pixel belongs.

10. The solid-state imaging device according to claim 9, wherein the second pixel from the front in the second direction for imaging the target is set to have higher sensitivity than the first pixel, and lower sensitivity than the other pixels in the second direction to which the first pixel and the second pixel belong.

11. The solid-state imaging apparatus according to claim 1, wherein each of the plurality of pixels is equipped with a color filter selected from a plurality of colors based on the arrangement of the pixels, data relating to light transmitted through the color filter is acquired, and processing relating to the value of the register is performed for each pixel equipped with the same color color filter.

12. The solid-state imaging device according to claim 1, wherein each of the plurality of pixels is equipped with a color filter selected from a plurality of colors based on the arrangement of the pixels, data relating to light transmitted through the color filter is acquired, and the processing relating to the value of the register is performed for each pixel that is capturing the same object regardless of the color transmitted through the color filter.

13. An electronic device comprising: a plurality of pixels arranged in an array in a first direction and a second direction intersecting the first direction, and acquiring data of the same object by multiple exposure, shifting in the second direction at unit time intervals; a register provided for each of the plurality of pixels, storing the data acquired by each of the plurality of pixels; and a control circuit that controls imaging at the pixels and data acquisition from the pixels, wherein the register copies the value of the register of the pixel that acquired information of the same object before the unit time interval; the control circuit controls imaging and data acquisition at the pixel corresponding to the register when the value of the register is not equal to or greater than a predetermined value; and controls stopping at least one of imaging or data acquisition at the pixel corresponding to the register when the value of the register is equal to or greater than the predetermined value.

Images