Solid-state imaging device and electronic apparatus

Abstract

[Problem] To effectively utilize resources. [Solution] This solid-state imaging device comprises a pixel array, a readout circuit, a counting circuit, a storage circuit, a control circuit, and a histogram processing circuit. In the pixel array, pixels provided with light-receiving elements for detecting the incidence of photons are disposed in an array. The readout circuit reads out detection signals from the pixels at a sampling interval. The counting circuit counts the number of the detection signals, read out from the readout circuit, at each sampling interval in units of macro pixels including the pixels. The storage circuit stores the counted pixel values for each sampling interval in each of the macro pixels in the storage circuit in units of bins having a width defined on the basis of the sampling interval. The control circuit dynamically determines the number of bins allocated to each of the plurality of macro pixels. The histogram processing circuit generates a histogram for each of the macro pixels from values stored in the bins allocated to the macro pixels.

Description

Solid-state imaging device and electronic device

[0001] The present disclosure relates to a solid-state imaging device and an electronic device.

[0002] In various technical fields, for example, in vehicle driving and the like, ranging techniques for measuring the distance to an object have been widely studied. Direct Time of Flight (dToF) is one of such techniques, and it is a method of irradiating an object with infrared light or the like, directly receiving the infrared light or the like reflected by the object by a light receiving element, and measuring the distance to the object based on the time from irradiation to reception.

[0003] In dToF, it is necessary to store the data for each predetermined time width acquired by a single photon avalanche photodiode (SPAD) or the like as a histogram to appropriately acquire the round-trip time from irradiation to reception. Generally, the number of bins of the histogram is determined for each device, and the maximum number of distances in one light receiving pixel or one macro pixel is determined by this number of bins.

[0004] However, when the number of bins is fixed, even when there is an object at a short distance that does not require the maximum distance depending on this number of bins, the number of bins for forming the histogram does not change, so there are often many unused bins. As a result, there is a problem that resources are not effectively utilized in both the power consumption for acquiring the histogram for all the number of bins and the time for acquiring a sufficient amount of data.

[0005] Japanese Unexamined Patent Application Publication No. 2020-091117

[0006] Therefore, one of the non-limiting problems to be solved by the embodiments of the present disclosure is to realize an appropriate resource allocation and effectively utilize the resources. The problems to be solved by the embodiments of the present disclosure can also be, as some further non-limiting examples, the problems corresponding to the effects described in the embodiments. That is, the problems corresponding to any at least one of the effects described in the description of the embodiments of the present disclosure can be the problems to be solved in the present disclosure.

[0007] According to one embodiment, the solid-state imaging device comprises a pixel array, a readout circuit, a counting circuit, a storage circuit, a control circuit, and a histogram processing circuit. The pixel array has pixels, each equipped with a photodetector for detecting the incidence of photons, arranged in an array. The readout circuit reads out detection signals from the pixels at a sampling period. The counting circuit counts the number of detection signals read out from the readout circuit at each sampling period, in units of macropixels containing one or more of the pixels. The storage circuit stores the pixel values ​​for each macropixel at each sampling period, as counted by the counting circuit, in units of bins having a width defined based on the sampling period. The control circuit dynamically determines the number of bins to be assigned to each of the multiple macropixels. The histogram processing circuit generates a histogram for each macropixel from the values ​​stored in the bins assigned to that macropixel.

[0008] The histogram processing circuit may calculate the distance from the histogram to the object being imaged for each macropixel.

[0009] The control circuit may determine the number of bins to be assigned to each macropixel based on the distance to the object captured for each macropixel.

[0010] The control circuit may assign a larger number of bins to each macropixel as the distance to the imaged object increases, and a smaller number of bins as the distance to the imaged object decreases.

[0011] The control circuit may group the macropixels according to the size of the number of bins determined for each macropixel, determine the exposure time for each group of macropixels, and control one or more macropixels belonging to the same group to be exposed at the same time.

[0012] The histogram processing circuit may detect one or more peaks present in the histogram for each macropixel and acquire position information for each peak; the control circuit may dynamically define the width of the bins around the peaks of the histogram of the macropixel based on the position of each peak; and the counting circuit may store the count value for each bin within the range to which a bin is assigned for each peak of the macropixel.

[0013] According to one embodiment, the electronic device comprises a solid-state imaging device as described above, and a light-emitting device that irradiates a target with light within a range detectable by the pixels. The light emitted by the light-emitting device is reflected by the target and captured by the solid-state imaging device to generate a depth image.

[0014] According to one embodiment, the electronic device comprises a solid-state imaging device described in relation to the above group, and a light-emitting device that irradiates light within a range detectable by the pixels. The solid-state imaging device outputs data relating to the group to the light-emitting device, and the light-emitting device irradiates light for each group at a timing based on the exposure time.

[0015] According to one embodiment, the imaging method involves a control circuit setting the number of bins to be assigned to each macropixel having one or more pixels in long-distance mode; a counting circuit counting the detection signals output based on the light sensed by each pixel in the plurality of macropixels to which bins are assigned and storing them in a memory circuit; a histogram processing circuit calculating the distance to the object captured for each macropixel based on the value for each bin stored in the memory circuit; and the control circuit determining the number of bins to be assigned to each macropixel based on the calculated distance.

[0016] The control circuit may, if the distance calculated at the macropixel is shorter than the distance that can be measured in the long-distance mode by a predetermined distance or more, reduce the number of bins assigned to the macropixel to less than the number of bins assigned in the long-distance mode.

[0017] According to one embodiment, the solid-state imaging device comprises a pixel array, a readout circuit, a counting circuit, a storage circuit, a control circuit, and a histogram processing circuit. The pixel array has pixels, each equipped with a photodetector for detecting the incidence of photons, arranged in an array. The readout circuit reads out detection signals from the pixels at a sampling period. The counting circuit counts the number of detection signals read out from the readout circuit at each sampling period, in units of macropixels containing one or more of the pixels. The storage circuit stores the pixel values ​​for each macropixel at each sampling period, as counted by the counting circuit, in units of bins having a width defined based on the sampling period. The control circuit dynamically determines the number of bins to be assigned to a plurality of macropixels according to the number of peaks acquired for each macropixel. The histogram processing circuit generates a histogram for each macropixel from the values ​​stored in the bins assigned to that macropixel. Furthermore, the control circuit sets the number of peaks and the number of bins to be assigned to each of the macro pixels at the number of peaks; the histogram processing circuit detects a number of peak positions corresponding to the number of peaks; the control circuit sets parameters for imaging a predetermined range including the peak positions; and the histogram processing circuit obtains the distance to the target for each of the macro pixels from the histogram in the predetermined range including the peak positions.

[0018] The solid-state imaging device described in the preceding paragraph has features that are applicable to any of the features of the solid-state imaging device described above, to the extent that they do not contradict each other.

[0019] According to one embodiment, the imaging method involves a control circuit setting the number of bins to be assigned to each macropixel having one or more pixels in 1-peak mode; a counting circuit counting detection signals output based on the light sensed by each pixel in the plurality of macropixels to which bins are assigned and storing them in a memory circuit; a histogram processing circuit detecting the range of bins in which the peak of the count value exists for each macropixel based on the bin-by-bin values ​​stored in the memory circuit; the control circuit dynamically changing the width of the bins based on the range of bins in which each detected peak exists, and determining the timing for counting in the counting circuit.

[0020] A schematic diagram showing an example of a distance measuring system according to one embodiment. A schematic block diagram showing an example of a solid-state imaging device according to one embodiment. A schematic diagram showing an example of a histogram according to one embodiment. A schematic diagram showing an example of a memory area related to a histogram according to one embodiment. A flowchart showing an example of imaging processing according to one embodiment. A schematic diagram showing an example of a histogram according to one embodiment. A schematic diagram showing an example of a memory area related to a histogram according to one embodiment. A flowchart showing an example of imaging processing according to one embodiment. A schematic diagram showing an example of a memory area related to a histogram according to one embodiment. A schematic diagram showing an example of a memory area related to a histogram according to one embodiment. A block diagram showing an example of the schematic configuration of a vehicle control system. An explanatory diagram showing an example of the installation location of the external information detection unit and the imaging unit.

[0021] The embodiments of this disclosure will now be described with reference to the drawings. The drawings are for illustrative purposes only, and the shape, size, or size ratio of each component in the actual device does not need to be exactly as shown in the drawings. Furthermore, the drawings are simplified, so any other components necessary for implementation should be appropriately provided in addition to those shown in the drawings.

[0022] This disclosure describes various aspects in the following order: 1. Overview of the distance measuring system (distance measuring device) 2. Histogram processing 3. Application examples

[0023] <1. Overview of the Distance Measurement System (Distance Measurement Device)>

[0024] Figure 1 is a schematic diagram showing an example of a distance measuring system according to one embodiment. The distance measuring system 1 is configured to include, for example, a light-emitting device 2 and a solid-state imaging device 3. The distance measuring system 1 can also be implemented as an electronic device, with the light-emitting device 2 and the solid-state imaging device 3 fixed and mounted in the same housing, rather than being arranged separately.

[0025] The light-emitting device 2 irradiates the target with light having a predetermined bandwidth. The light-emitting device 2 can change parameters such as the timing of light irradiation based on feedback from the solid-state imaging device 3.

[0026] The solid-state imaging device 3 receives the light emitted by the light-emitting device 2 that is reflected from the object, and calculates the distance to the object based on the timing of the reception of the reflected light. For example, the solid-state imaging device 3 receives data related to the timing of light irradiation from the light-emitting device 2, exposes the object so that it can receive light at an appropriate reception timing based on this timing, and measures the distance to the object by receiving the reflected light emitted by the light-emitting device 2.

[0027] The solid-state imaging device 3 receives reflected light (photons) in a target area corresponding to each light-receiving pixel, or each macropixel containing one or more light-receiving pixels, at a predetermined sampling period using, for example, an avalanche photodiode such as a SPAD. The solid-state imaging device 3 counts the received photons to generate a histogram and calculates the distance to the target corresponding to each light-receiving pixel or macropixel from this histogram.

[0028] Figure 2 is a schematic block diagram showing an example of a solid-state imaging device 3 according to one embodiment. The solid-state imaging device 3 includes a pixel array 30, a control circuit 31, a memory circuit 32, a readout circuit 33, a counting circuit 34, a histogram processing circuit 35, and an input / output interface 36. The solid-state imaging device 3 receives reflected light from the light-emitting device 2 at an appropriate timing to measure the distance to the target. The solid-state imaging device 3 can also generate a distance image, for example.

[0029] The pixel array 30 is a region in which light-receiving pixels, such as SPADs, are arranged in a two-dimensional array. The pixel array 30 is formed such that light is focused onto each light-receiving pixel by an optical system arranged separately from or integrated with the pixel array 30, and each light-receiving pixel receives reflected light from an appropriate region.

[0030] Light-receiving pixels arranged in a two-dimensional array may be driven individually, or multiple light-receiving pixels may be driven as macropixels that are exposed at the same time. When driven as macropixels, the outputs from light-receiving pixels belonging to the same macropixel can be added together and considered as reflected light from a single region.

[0031] A light-receiving pixel outputs a signal corresponding to the intensity of the light it receives. For example, a light-receiving pixel detects incident light on a photon-by-photon basis and outputs an avalanche current (detection signal).

[0032] A macropixel can be defined, for example, as a set of 3 x 3 pixels, but is not limited to this. It can also be defined as a set of fewer pixels, such as 1 pixel, 2 x 2 pixels, or more than 10 x 10 pixels. This definition may be determined by the device or may be variable. While a lower number of pixels in a macropixel allows for the acquisition of higher resolution images, it can also lead to problems such as a higher noise ratio. Therefore, it is desirable to define it appropriately according to the environment, application, etc.

[0033] The control circuit 31 controls the light-receiving pixels arranged in the pixel array 30. The control circuit 31 is not limited to this and may also control other processes in the solid-state imaging device 3. The control circuit 31 can dynamically perform controls related to the histogram of each macropixel, for example, controls related to the number of bins for generating the histogram for each macropixel.

[0034] The memory circuit 32 stores data necessary for executing the processing of the solid-state imaging device 3. The memory circuit 32 may be configured to include various forms of Random Access Memory (RAM), Read Only Memory (ROM), registers, buffers, or caches as desired. Furthermore, the memory circuit 32 may be configured in a way that allows for the division of the memory area depending on the application.

[0035] The memory circuit 32 may be configured to reserve an area for storing, for example, histogram data in bin units, which is necessary to measure the maximum distance that can be measured for each macropixel. The width of the bin may be determined by the sampling period.

[0036] Furthermore, areas for histograms of multiple macropixels may be reserved. In either case, these areas can be managed by forming banks in the memory circuit 32, for example. As an example that is not limited, one bank may be set up to store histograms capable of measuring the maximum distance of three macropixels. Of course, one bank may be set up to store areas for one histogram, two histograms, or four or more histograms.

[0037] In other words, the memory circuit 32 may be configured to allocate a number of bins that can store data up to the maximum measurement distance according to the sampling period, in an appropriate manner for each macropixel, or for each frame, including the time allocated for counting and signal processing, so that data can be acquired for each macropixel.

[0038] The readout circuit 33 reads out the detection signal output from the light-receiving pixels of the pixel array 30. The readout circuit 33 reads out the detection signal output from the pixel array 30 at a predetermined sampling period, converts it to a digital signal, and outputs it. The readout circuit 33 can use a readout circuit for a known SPAD.

[0039] The reading method of the reading circuit 33 may be a scan method that scans all effective light-receiving pixels to acquire a signal, or an event detection method that reads out a detection signal at the timing when light is received at each light-receiving pixel.

[0040] The counting circuit 34 counts the digital signal from each light-receiving pixel read by the readout circuit 33 in units of the sampling period, adds them up for each macropixel, and stores them in the memory circuit 32. For example, the counting circuit 34 counts the number of detection signals output from pixels and read out by the readout circuit 33 in units of macropixels for each sampling period, and stores them in the area of ​​a bin corresponding to the detection timing. As a result, the memory circuit 32 stores the number of detection signals in bins that form a histogram for each macropixel.

[0041] The histogram processing circuit 35 generates a histogram from the bin values ​​added by the counting circuit 34 on a macropixel basis, and performs various operations on this histogram. For example, the histogram processing circuit 35 can calculate the distance to the target based on the timing of the light received on each macropixel of the pixel array 30 from the histogram. Alternatively, the histogram processing circuit 35 may generate a distance image across the pixel array 30 using the distances obtained on a macropixel basis.

[0042] The control circuit 31 and the histogram processing circuit 35 may have at least some of their functions executed by a general-purpose processing circuit, such as a Central Processing Unit (CPU) or the like. Also, not limited to the control circuit 31 and the histogram processing circuit 35, for other configurations, at least a part of the processing may be executed by a general-purpose processing circuit. In this case, it may be a form in which information processing by software can be specifically realized by a processing circuit of hardware. Also, information related to software may be stored in the storage circuit 32.

[0043] The input / output I / F 36 is an interface that connects the inside and outside of the solid-state imaging device 3 and executes transfer of data and the like. The solid-state imaging device 3 can, for example, transmit and receive at least either commands or data to and from the light-emitting device 2 via the input / output I / F 36. When the light-emitting device 2 and the solid-state imaging device 3 are provided as the distance measurement system 1, the input / output I / F 36 may be realized by wiring such as a bus or a high-speed interface such as Mobile Industry Processor Interface (MIPI (registered trademark)) in the transmission and reception of data and the like between the light-emitting device 2 and the solid-state imaging device 3.

[0044] Also, the solid-state imaging device 3 may be formed of a plurality of chips. For example, a part of the pixel array 30, the readout circuit 33, the counting circuit 34, and the storage circuit 32 may be formed as one chip, and a part of the control circuit 31, the histogram processing circuit 35, and the storage circuit 32 may be formed on other chips, and each may transmit and receive data via a selector or an interface.

[0045] <2. Processing of Histogram>

[0046] Next, the processing of the histogram in the present disclosure will be described. In the present disclosure, for example, the area of bins for generating a histogram assigned when measuring the maximum distance in one macro pixel is utilized.

[0047] (First Embodiment)

[0048] FIG. 3 is a diagram schematically showing an example of a histogram according to an embodiment. In this figure, bins in the area for generating the histogram secured in the memory circuit 32 and the number of photons for each bin are shown, and for three macro pixels, namely macro pixel MP1, macro pixel MP2, and macro pixel MP3, the bins for the maximum measurable distance are shown. In the example of management by a bank, it can be assumed that an area of a histogram for measuring the maximum distance of three macro pixels is allocated to one bank.

[0049] Note that the number of bins and the number of histograms stored in one memory area among a plurality of areas for forming a histogram are shown as an example and are not limited to this example. For example, in the initial state, an area for the number of bins for forming a histogram for measuring the maximum distance corresponding to one macro pixel may be secured as one memory area.

[0050] As described above, a region of a predetermined size for generating a histogram is secured in the memory circuit 32. The solid-state imaging device 3 stores the number of photons detected by the light-receiving pixels belonging to each macro pixel in bins having a width defined by a time width in each macro pixel in the region secured in the memory circuit 32. The histogram processing circuit 35 executes processing related to the histogram by referring to this region of the memory circuit 32.

[0051] When the sensor is a scanning type, the solid-state imaging device 3 can execute processing from exposure to counting by dividing the scanning in one frame into a plurality of sub-frames, for example, and can acquire data for one or a plurality of macro pixels in each sub-frame. The solid-state imaging device 3 repeatedly executes the sub-frame processing until the acquisition of data for all the macro pixels included in the pixel array 30 is completed, and this may be regarded as the scanning processing for one frame. FIG. 3 is a diagram showing an example of the state of one bank obtained as a result of the processing of one sub-frame.

[0052] After the capture of a frame (or subframe) is complete, the histogram processing circuit 35 extracts, for example, the bin where the photon count is most frequent from the histogram of the region corresponding to each macropixel, and calculates the distance from the time corresponding to this bin to the target region acquired by that macropixel. The histogram processing circuit 35 can store the calculated distance for each macropixel in a predetermined area of ​​the memory circuit 32. Note that the histogram processing circuit 35 is not limited to the method using the mode, and can use any method (including statistical methods) to calculate distance from the histogram.

[0053] The histogram processing circuit 35 can generate calculated data, including histogram-related data for each macropixel, and combine packets for all macropixels to generate output data. For example, the histogram processing circuit 35 can include data on the number of bins used for each macropixel in the payload.

[0054] For example, the histogram processing circuit 35 can include in its payload information for each bank that includes data such as the number of macropixels contained in that bank, identifiers for the contained macropixels, and the number of histogram bins for each of the contained macropixels.

[0055] In this embodiment, the control circuit 31 sets various parameters for the next imaging timing (frame) based on the distance calculated by the histogram processing circuit 35.

[0056] In Figure 3, for example, by referring to the histograms corresponding to macropixels MP1 and MP3, a peak in the histogram exists in a range relatively close to the maximum distance. On the other hand, in the histogram corresponding to macropixel MP2, a peak exists in a range relatively close to the distance. The histogram processing circuit 35 calculates the distance to each macropixel from the histograms corresponding to macropixels MP1, MP2, and MP3, and stores it in the memory information storage area of ​​the memory circuit 32.

[0057] The control circuit 31 determines the number of histogram bins to be allocated for each macropixel in the next frame by referring to the memory circuit 32's area for storing distance information. In other words, the control circuit 31 determines the number of bins to be allocated for each macropixel in the next frame based on the distance to the object captured for each macropixel.

[0058] Figure 4 shows an example of the region that forms the histogram after determining the number of bins to be assigned to each macropixel in the next frame in the example of Figure 3. Macropixels MP1 and MP3 are assigned the same range of region as shown in Figure 3, while macropixel MP2, which had an object at a relatively close distance, is assigned a narrower range of region. Furthermore, if the region assigned to macropixel MP4 is a narrow range of region, the histogram processing circuit 35 can be controlled to include the region that generates the histogram corresponding to macropixel MP4 in this memory region.

[0059] In other words, the histogram processing circuit 35 can control the number of bins assigned to each macropixel, with a larger number of bins assigned to the object being imaged for each macropixel as the distance to the object is longer, and a smaller number of bins assigned to the object being imaged for each macropixel as the distance to the object is shorter.

[0060] The histogram processing circuit 35 may, for example, control the allocation of the number of bins to the counting circuit 34 and / or the histogram processing circuit 35. The counting circuit 34 and / or the histogram processing circuit 35 perform processing such as adding the detection signals output by the light-receiving pixels for each macropixel and storing them in the memory area corresponding to the bins, according to the allocated number of bins.

[0061] Furthermore, the histogram processing circuit 35 can output signals for controlling the exposure of light-receiving pixels in the pixel array 30. For example, in the case of Figure 4, the histogram processing circuit 35 can output control signals to the pixel array 30 to shorten the exposure time for macro pixels MP2 and MP4. Based on the control signals, the pixel array 30 may shorten the exposure time for light-receiving pixels belonging to macro pixels MP2 and MP4 and control imaging in the next frame.

[0062] Figure 5 is a flowchart showing an example of processing by a solid-state imaging device 3 according to one embodiment. The mode in which a histogram for the maximum measurable distance is used for each macropixel is referred to as the long-distance mode.

[0063] First, the histogram processing circuit 35 manages the memory circuit 32 in the far-distance mode as its initial state (S100). This management may be carried out by banks as described above. However, it is not limited to banks; the method of managing the memory area is not particularly limited as long as it is a method that can appropriately manage a predetermined number of bins logically or physically. For example, as shown in Figure 3, the histogram processing circuit 35 allocates the number of bins necessary to form a histogram in far-distance mode for three macropixels in one bank.

[0064] The solid-state imaging device 3 measures the distance between each macropixel using the light-receiving pixels belonging to the pixel array 30, the readout circuit 33, the counting circuit 34, and the histogram processing circuit 35, and acquires a distance image (S102). The data related to the acquired distance image may be stored in, for example, the memory circuit 32, or output to the outside via the input / output I / F 36. If stored in the memory circuit 32, it can be stored in, for example, at least a portion of the memory circuit 32 in a chip provided in the solid-state imaging device 3, which is different from the pixel array 30, etc.

[0065] The histogram processing circuit 35 stores the distance for each macropixel in an appropriate area such as a register in the memory circuit 32 (S104). For example, the histogram processing circuit 35 can generate packets for each bank described above, storing the distance calculated for the macropixels that contain histograms in that bank, associated with the macropixel identifier.

[0066] Before performing imaging processing for subsequent frames, the control circuit 31 determines the number of bins to allocate to each macropixel by referring to the distance for each macropixel stored in the processing of step S104 (S106). For example, for a histogram like the one in Figure 3, the control circuit 31 allocates an area to store the histogram for the next frame, as shown in Figure 4. For example, if the number of bins required to measure distances shorter than those in the long-distance mode is sufficient, the control circuit 31 can allocate fewer bins to the macropixel.

[0067] Furthermore, the control circuit 31 can also output a control signal to the pixel array 30 at the imaging timing of the next frame, requesting that the exposure time be shortened for macro pixels whose histogram bin allocation has decreased.

[0068] The solid-state imaging device 3 generates a histogram with the time related to the number of photons detected at each macropixel on the horizontal axis, within the range of the number of bins allocated in step S106, and obtains a distance image by calculating the distance using this histogram (S108).

[0069] The histogram processing circuit 35 stores the distance for each macropixel calculated in the above processing in an appropriate area such as a register in the memory circuit 32, associating it with the identifier representing the macropixel and the number of bins assigned to the macropixel (S110).

[0070] The solid-state imaging device 3 determines whether or not to initialize the number of bins assigned to each macropixel (S112). The solid-state imaging device 3 may, for example, determine whether to initialize the number of bins based on whether a predetermined number of frames have been processed since the processing in S100.

[0071] If the number of bins is not initialized (S112: NO), the solid-state imaging device 3 repeatedly executes the process from step S106 until the imaging is complete. If the number of bins is initialized (S112: YES), the solid-state imaging device 3 repeatedly executes the process from step S100. Completion of imaging may be requested, for example, by the user or another information processing device, or by the solid-state imaging device 3 or at least one of the components in the ranging system 1 or the light-emitting device 2.

[0072] As described above, by dynamically changing the bin area for generating the histogram frame by frame and effectively utilizing memory areas (e.g., banks), resource utilization efficiency can be improved. In addition to effectively utilizing memory resources, it is also possible to dynamically reduce the number of bins allocated to macro pixels, thereby shortening the exposure time for macro pixels to which fewer bins are allocated, and thus reducing overall time costs and power consumption.

[0073] (Second Embodiment)

[0074] In the first embodiment described above, we explained how to change the allocation of the number of bins, but in order to make more effective use of resources, it is also possible to swap the macro pixels allocated to each memory area (e.g., bank).

[0075] Figure 6 shows an example of a histogram acquired in two subframes within the first frame. According to the same process as in the previously described embodiment, the control circuit 31 controls the number of bins for macropixels MP2, MP4, MP5, and MP6 in the next frame without changing the number of bins for macropixels MP1 and MP3. With this process, in the next frame, for example, the first subframe can be assigned in the same way as in Figure 4, but in this embodiment, the macropixels assigned to each subframe can be changed.

[0076] The control circuit 31, for example, groups the macropixels. For example, the control circuit 31 groups macropixels MP1 and MP3 into the first group, macropixels MP2, MP4, MP5 and MP6 into the second group, and so on. According to this grouping, the control circuit 31 allocates memory areas, such as banks, and allocates areas to form histograms for each macropixel.

[0077] Figure 7 shows an example of memory areas for acquiring histograms in two subframes within a frame in the second frame following Figure 6. The first subframe is allocated memory to store the histograms of macropixels MP1 and MP3 of the first group, which are still set to long-range mode. Although not shown in Figure 6, memory areas can also be combined for other macropixels, such as macropixel MP7.

[0078] On the other hand, the second subframe uses a memory area reserved for the second group of macropixels MP2, MP4, MP5, and MP6, which do not continue in long-range mode.

[0079] As described above, the control circuit 31 can divide the memory area for generating histograms by group. In the first subframe of the second frame, for example, if the measurement is not taken for the same amount of time after the light-emitting device 2 has illuminated, accurate distance measurement may not be possible. For this reason, it is desirable that macro pixels MP1, etc., be assigned the same number of bins as in the first frame for distance measurement, and that the exposure time is also the same as in the first frame.

[0080] On the other hand, in the second subframe of the second frame, it is highly likely that it is sufficient to receive light for a shorter period than in the first subframe, for example, after the light-emitting device 2 has irradiated. As a result, the exposure time of the second subframe can be shortened.

[0081] In view of the above, the control circuit 31 can determine the number of bins of macro pixels for each subframe and control the exposure time of the macro pixels that receive light in each subframe. For example, the control circuit 31 may set a shorter exposure time for the second subframe than for the first frame.

[0082] In this way, the control circuit 31 can group macropixels according to the size of the next frame's bin number determined for each macropixel, and determine the exposure time and exposure timing for each group. The control circuit 31 can also transmit signals to the light-emitting device 2 to control the irradiation time for each subframe.

[0083] In particular, when measuring distance using a scanning method, if the light-emitting elements in the light-emitting device 2 that provide illumination and the light-receiving pixels in the solid-state imaging device 3 that receive light are associated, the control of the light emission of the light-emitting elements in the light-emitting device 2 corresponding to each macropixel in the solid-state imaging device 3 for each subframe can be determined more precisely. Specifically, the control circuit 31 can determine the timing and duration of illumination from the light-emitting elements corresponding to macropixels MP1, MP3, and MP7 in the first subframe, and the timing and duration of illumination from the light-emitting elements corresponding to macropixels MP2, MP4, MP5, and MP6 in the second subframe.

[0084] The control circuit 31 may determine the irradiation timing and irradiation time for each determined light-emitting element (region) and notify the light-emitting device 2. The light-emitting device 2 can control the irradiation based on the irradiation timing and irradiation time for each received light-emitting element (region).

[0085] Furthermore, the control circuit 31 can also set the exposure intensity. The longer the measurement distance, the higher the intensity of the light emission required. For this reason, when the measurement distance becomes shorter, the control circuit 31 may send a signal to the light emission device 2 to control the reduction of the irradiation intensity.

[0086] Figure 8 is a flowchart showing an example of the imaging process according to this embodiment. Processes denoted by the same reference numerals as in Figure 5 perform essentially the same process. In this embodiment, an additional process is added between the process in step S106 and the process in step S108.

[0087] In step S106, the control circuit 31 determines the number of bins for each macropixel and allocates a memory area for generating a histogram. After that, the control circuit 31 further groups the macropixels according to the number of bins (S200). For example, as shown in Figure 7, the macropixels are classified into two groups: one group of macropixels that continue in long-distance mode and whose number of bins does not change, and the other group.

[0088] Furthermore, the grouping is not limited to this; the control circuit 31 may, after assigning a number of bins to the macropixels, rearrange them in the order of the assigned bins (for example, in ascending or descending order), and then group them according to this order so as not to exceed the memory capacity of the bank.

[0089] For example, in the case of Figure 6, the control circuit 31 sorts the macropixels in ascending order from the bin number assignment in the second frame into macropixel MP2, macropixel MP4, macropixel MP5, macropixel MP6, macropixel MP1, and macropixel MP3. Then, the control circuit 31 adds up the bin numbers in order and groups them within the limits that do not exceed the bank capacity. That is, if the control circuit 31 determines that the sum of the bank numbers allocated to macropixels MP2, MP4, MP5, and MP6 does not exceed the bank capacity, but adding the bin number of the next macropixel MP1 would exceed the bank capacity, then it classifies macropixels MP2, MP4, MP5, and MP6 as one group. Similarly, the control circuit 31 classifies macropixel MP1 and macropixel MP3 as one group.

[0090] Based on these results, the control circuit 31 calculates the exposure time for a group including macropixels MP2, MP4, MP5, and MP6 from the number of bins allocated to each macropixel, and similarly calculates the exposure time for the other group. As an example, the control circuit 31 can set the exposure time for macropixels MP2, MP4, MP5, and MP6 to receive photons for a duration corresponding to the number of bins allocated to each macropixel, which is the maximum number of bins allocated to each macropixel—in this case, the number of bins allocated to macropixel MP6. The exposure time can be calculated as the time from the timing of irradiation in the light-emitting device 2.

[0091] The control circuit 31 transmits information regarding exposure time, exposure intensity, and the position of the light-emitting element to be exposed to the light-emitting device 2 as needed. As a result, the light-emitting device 2 and the solid-state imaging device 3 work together to measure the distance to the target by irradiating it with appropriate light and receiving reflected light from the target.

[0092] Although the term "grouping" was used for explanatory purposes, as mentioned above, it can be rephrased as sorting by the number of bins to which macro pixels are allocated, and then assigning banks in the sorted order.

[0093] As described above, this embodiment makes it possible to reduce the exposure time for the entire frame. As a result, in addition to reducing the memory capacity according to the aforementioned embodiment, it is also possible to reduce power consumption by reducing the time required for light reception. It is also possible to reduce the irradiation time in the light-emitting device 2, thereby achieving further reduction in power consumption.

[0094] In the first and second embodiments, for example, as shown in Figures 4 and 7, the memory capacity allocated to macro pixels with shorter distances is reduced according to the measured distance. Various methods can be used to reduce this memory capacity.

[0095] The control circuit 31 may, for example, set one or more threshold distances for the distances that can be measured in the histogram in long-range mode, and determine the number of bins for macro pixels in the next frame depending on whether the distance is greater than or equal to the threshold. In this case, grouping can be performed for each number of bins classified by the threshold.

[0096] The control circuit 31 may, for example, allocate a number of bins in the next frame that allows the distance measured for each macropixel to be sufficiently measured in the long-range mode, depending on the distance measured. Specifically, the control circuit 31 may allocate a number of bins in the next frame that allows the measurement of a constant multiple of the measured distance (e.g., 1.5 times), or a number of bins that allows the measurement of distances up to a constant distance away from the measured distance, depending on the measured distance.

[0097] Furthermore, while Figure 7 shows macropixels with the same number of bins being classified into the same bank, this is not the only way to do so. For example, when sorting and grouping by the number of bins, if a bank has a fractional number of bins, macropixels that can be sufficiently measured with that fractional number of bins may be assigned to the same bank. For example, macropixels may be sorted in descending order of the number of bins, and macropixels may be assigned to each bank in a way that minimizes the number of fractional bins.

[0098] (Third embodiment)

[0099] In the embodiments described above, the case where there is one peak for each macropixel was explained. In contrast, for example, when there is a transparent or translucent object such as glass between the object and the object, it is more desirable to be able to measure the distance from multiple peaks. In this embodiment, a method for achieving more accurate distance measurement by appropriately allocating memory even when there are multiple peaks will be described.

[0100] Figure 9 is a flowchart illustrating an example of the imaging process according to this embodiment. The solid-state imaging device 3 can, for example, generate histograms of 6 macropixels, 3 macropixels, or 2 macropixels in one memory area (bank) within one subframe, depending on the situation. However, it is not limited to this, and any configuration that can generate histograms for multiple types of macropixel counts is acceptable.

[0101] First, the solid-state imaging device 3 is set to a single-peak mode, for example, a mode that generates a coarse histogram for six macropixels and extracts one peak (S300). This setting may be performed by the control circuit 31. The control circuit 31 can be set, for example, to acquire the histogram of the maximum distance for six macropixels by appropriately setting the bin width.

[0102] Figure 10 shows an example of generating a coarse histogram for six macropixels. The control circuit 31 controls the bin width so that a histogram for the six macropixels can be generated with coarse accuracy as shown in the figure above. The control circuit 31 also controls the exposure timing, etc., so that the histograms for the six macropixels are generated at the same time. In this case, the control circuit 31 may also notify the light-emitting device 2 of data or send control signals so that the light-emitting elements in the appropriate areas emit light at the appropriate timing and for the appropriate irradiation time.

[0103] Returning to Figure 9, the solid-state imaging device 3 receives light under the conditions set by the control circuit 31, generates a histogram for each macropixel, and obtains the peak position for each macropixel (S302). Peak position detection can be performed, for example, by the histogram processing circuit 35. The histogram processing circuit 35 can determine, for example, that the position indicated by the square brackets in the upper diagram of Figure 10 is the peak position. The histogram processing circuit 35 can, for example, define the range in which the peak exists as a predetermined width, and obtain time information or distance information within the predetermined width including the peak position by calculating the peak position using any statistical means.

[0104] The solid-state imaging device 3 acquires a distance image from a histogram acquired with finer temporal resolution (i.e., distance resolution) based on the information at the peak positions acquired by the histogram processing circuit 35 (S304). For example, the control circuit 31 first controls the exposure timing and other parameters for each macropixel so that each macropixel can be imaged within a predetermined width region including the peak position acquired by the histogram processing circuit 35.

[0105] The solid-state imaging device 3 can acquire a histogram with finer resolution for a predetermined width region including peaks by imaging using parameters set by the control circuit 31. The control circuit 31 may be controlled by distance in addition to exposure timing, or the same parameters may be used for exposure timing and exposure time to control the timing of reading and adding in the readout circuit 33 and the counting circuit 34.

[0106] Using a histogram based on the information captured in this state (for example, the lower figure in Figure 10), the histogram processing circuit 35 can generate a distance image in one-peak mode, where one peak is acquired from each macropixel and the distance is calculated.

[0107] Next, the control circuit 31 sets the 2-peak mode (S306). Note that the processing from step S306 can be, for example, the imaging in the frame following the processing up to step S304. However, it is also possible to perform the processing in step S304 as the imaging in the frame following step S302.

[0108] Figure 11 shows a histogram when controlled in two-peak mode, that is, a mode in which two peaks are acquired for each macropixel. The solid-state imaging device 3 is set to a mode in which the positions of two peaks can be acquired by generating a histogram of three macropixels in one memory area (bank), for example, as shown in the upper part of Figure 11.

[0109] Returning to Figure 9, the solid-state imaging device 3 acquires a histogram with coarse temporal resolution in the two-peak mode and obtains the peak positions from this histogram (S308). The result is, for example, the histogram in the upper part of Figure 11, which shows peaks P1 and P2 for macropixel MP1, peaks P1 and P2 for macropixel MP2, and peak P2 for peak P1 for macropixel MP3.

[0110] Next, similar to step S304, the control circuit 31 sets appropriate imaging parameters and / or histogram acquisition parameters based on the position of each peak acquired by the histogram processing circuit 35 to acquire a detailed histogram at each peak (for example, the lower figure in Figure 11). The histogram processing circuit 35 then uses the histogram information acquired by these parameters to perform appropriate statistical processing to obtain a distance image of the distance to the target at each macropixel (S310).

[0111] If, in the process of step S308, only one peak is detected in a certain macropixel, the solid-state imaging device 3 can set parameters to acquire detailed time information (distance information) for the range containing the detected single peak in that macropixel. Alternatively, if only one peak is detected, the solid-state imaging device 3 may generate a detailed histogram using the same parameters as for the peaks P1 of other macropixels, assuming that there is no peak P2 for the corresponding macropixel.

[0112] Subsequently, the solid-state imaging device 3 may similarly acquire a histogram in 3-peak mode. An example of a histogram in 3-peak mode is shown in Figure 12. In 3-peak mode as well, the solid-state imaging device 3 can similarly acquire the position of each peak and generate a higher-resolution histogram for each peak within a predetermined width, thereby generating a distance image in which the distance from the three peaks to the target at each macropixel is obtained.

[0113] In this way, the histogram processing circuit 35 detects the peaks present in the histogram for each macropixel and acquires the position information of each peak. The control circuit 31 then dynamically changes the bin width around each peak based on the peak's position, and the readout circuit 33 and counting circuit 34 acquire the count value for each peak of each macropixel in the memory area (bank) where a histogram is generated. By generating a histogram from these count values, the histogram processing circuit 35 can calculate the distance to the target with greater accuracy at each macropixel.

[0114] As described above, by dynamically changing the number of macro pixels per bank (the number of histogram bins assigned to each macro pixel) and the bin width setting at the peak, similar to the embodiment described above, it is possible to obtain more accurate distance images even when there are multiple peaks.

[0115] Furthermore, the third embodiment can be used in combination with the embodiments described above. For example, the solid-state imaging device 3 acquires the number of peaks for each macropixel in 3-peak mode in the first frame and acquires highly accurate distance information for each macropixel in the same manner as described above.

[0116] The control circuit 31 sets the number of bins to be allocated to each macropixel using the method of the first embodiment, based on the acquired distance information. In this case, the control circuit 31 can also further group the macropixels using the method of the second embodiment.

[0117] The solid-state imaging device 3 can generate a distance image that makes efficient use of resources by repeating the processing described in the previous paragraph. Furthermore, the solid-state imaging device 3 can repeatedly perform processing from imaging in 3-peak mode every predetermined number of frames.

[0118] According to this embodiment, it is possible to measure distances with high accuracy in 3-peak mode, for example, even when there are transparent objects, and to generate distance images that make effective use of resources as in the first or second embodiment based on said distances.

[0119] <3. Examples of Application>

[0120] The technology disclosed herein can be applied to a variety of products. For example, the technology disclosed herein may be implemented as a device mounted on any type of mobile vehicle, such as automobiles, electric vehicles, hybrid electric vehicles, motorcycles, bicycles, personal mobility devices, airplanes, drones, ships, robots, construction machinery, or agricultural machinery (tractors).

[0121] Figure 13 is a block diagram showing a schematic configuration example of a vehicle control system 7000, which is an example of a mobile control system to which the technology described herein may be applied. The vehicle control system 7000 comprises a plurality of electronic control units connected via a communication network 7010. In the example shown in Figure 13, the vehicle control system 7000 comprises a drive system control unit 7100, a body system control unit 7200, a battery control unit 7300, an external information detection unit 7400, an internal information detection unit 7500, and an integrated control unit 7600. The communication network 7010 connecting these plurality of control units may be an in-vehicle communication network compliant with any standard such as CAN (Controller Area Network), LIN (Local Interconnect Network), LAN (Local Area Network), or FlexRay (registered trademark).

[0122] Each control unit comprises a microcomputer that performs calculations according to various programs, a storage unit that stores programs executed by the microcomputer or parameters used in various calculations, and a drive circuit that drives various controlled devices. Each control unit is equipped with a network interface for communication with other control units via the communication network 7010, and a communication interface for communication with devices or sensors inside or outside the vehicle via wired or wireless communication. Figure 13 illustrates the functional configuration of the integrated control unit 7600, which includes a microcomputer 7610, a general-purpose communication interface 7620, a dedicated communication interface 7630, a positioning unit 7640, a beacon receiver 7650, an in-vehicle equipment interface 7660, an audio / image output unit 7670, an in-vehicle network interface 7680, and a storage unit 7690. Other control units similarly include a microcomputer, a communication interface, and a storage unit.

[0123] The drivetrain control unit 7100 controls the operation of devices related to the vehicle's drivetrain according to various programs. For example, the drivetrain control unit 7100 functions as a control device for generating driving force for the vehicle, such as an internal combustion engine or drive motor; a driving force transmission mechanism for transmitting driving force to the wheels; a steering mechanism for adjusting the steering angle of the vehicle; and a braking device for generating braking force for the vehicle. The drivetrain control unit 7100 may also function as a control device such as ABS (Antilock Brake System) or ESC (Electronic Stability Control).

[0124] A vehicle state detection unit 7110 is connected to the drivetrain control unit 7100. The vehicle state detection unit 7110 includes, for example, a gyro sensor for detecting the angular velocity of the vehicle's axial rotational motion, an acceleration sensor for detecting the vehicle's acceleration, or at least one of the sensors for detecting the amount of accelerator pedal operation, brake pedal operation, steering wheel steering angle, engine speed, or wheel rotation speed. The drivetrain control unit 7100 performs calculations using signals input from the vehicle state detection unit 7110 to control the internal combustion engine, drive motor, electric power steering system, brake system, etc.

[0125] The body system control unit 7200 controls the operation of various devices mounted on the vehicle body according to various programs. For example, the body system control unit 7200 functions as a control device for a keyless entry system, a smart key system, a power window system, or various lamps such as headlights, reverse lights, brake lights, turn signals, or fog lights. In this case, the body system control unit 7200 may receive radio waves transmitted from a portable device that replaces a key or signals from various switches. The body system control unit 7200 receives these radio waves or signals and controls the vehicle's door lock system, power window system, lamps, etc.

[0126] The battery control unit 7300 controls the secondary battery 7310, which is the power source for the drive motor, according to various programs. For example, the battery control unit 7300 receives information such as battery temperature, battery output voltage, or remaining battery capacity from the battery device equipped with the secondary battery 7310. The battery control unit 7300 uses these signals to perform calculations and controls the temperature of the secondary battery 7310 or the cooling device provided in the battery device.

[0127] The external information detection unit 7400 detects information from outside the vehicle equipped with the vehicle control system 7000. For example, the external information detection unit 7400 is connected to at least one of the imaging unit 7410 and the external information detection unit 7420. The imaging unit 7410 includes at least one of the following: a Time of Flight (ToF) camera, a stereo camera, a monocular camera, an infrared camera, and other cameras. The external information detection unit 7420 includes at least one of the following: an environmental sensor for detecting the current weather or climate, or an ambient information detection sensor for detecting other vehicles, obstacles, or pedestrians around the vehicle equipped with the vehicle control system 7000.

[0128] The environmental sensor may be at least one of the following: a raindrop sensor for detecting rain, a fog sensor for detecting fog, a sunshine sensor for detecting the degree of sunlight, and a snow sensor for detecting snowfall. The ambient information detection sensor may be at least one of the following: an ultrasonic sensor, a radar device, and a LIDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) device. These imaging unit 7410 and external information detection unit 7420 may be provided as independent sensors or devices, or as a device in which multiple sensors or devices are integrated.

[0129] Here, Figure 14 shows examples of the installation locations of the imaging unit 7410 and the external information detection unit 7420. The imaging units 7910, 7912, 7914, 7916, and 7918 are installed, for example, at least one of the following locations on the vehicle 7900: the front nose, side mirrors, rear bumper, tailgate, and the upper part of the windshield inside the passenger compartment. The imaging unit 7910 installed on the front nose and the imaging unit 7918 installed on the upper part of the windshield inside the passenger compartment mainly acquire images of the front of the vehicle 7900. The imaging units 7912 and 7914 installed on the side mirrors mainly acquire images of the sides of the vehicle 7900. The imaging unit 7916 installed on the rear bumper or tailgate mainly acquires images of the rear of the vehicle 7900. The imaging unit 7918, located at the top of the windshield inside the vehicle, is primarily used for detecting preceding vehicles, pedestrians, obstacles, traffic lights, traffic signs, or lanes.

[0130] Figure 14 shows an example of the imaging range of each imaging unit 7910, 7912, 7914, and 7916. Imaging range a shows the imaging range of imaging unit 7910 located on the front nose, imaging ranges b and c show the imaging ranges of imaging units 7912 and 7914 located on the side mirrors, respectively, and imaging range d shows the imaging range of imaging unit 7916 located on the rear bumper or back door. For example, by superimposing the image data captured by imaging units 7910, 7912, 7914, and 7916, an overhead view image of vehicle 7900 can be obtained.

[0131] The external information detection units 7920, 7922, 7924, 7926, 7928, and 7930, which are installed on the front, rear, sides, corners, and the upper part of the windshield inside the vehicle 7900, may be, for example, ultrasonic sensors or radar devices. The external information detection units 7920, 7926, and 7930, which are installed on the front nose, rear bumper, back door, and the upper part of the windshield inside the vehicle 7900, may be, for example, LIDAR devices. These external information detection units 7920 to 7930 are mainly used for detecting preceding vehicles, pedestrians, or obstacles.

[0132] Returning to Figure 13, the explanation continues. The external information detection unit 7400 causes the imaging unit 7410 to capture images of the area outside the vehicle and receives the captured image data. The external information detection unit 7400 also receives detection information from the connected external information detection unit 7420. If the external information detection unit 7420 is an ultrasonic sensor, radar device, or LIDAR device, the external information detection unit 7400 emits ultrasonic waves or electromagnetic waves and receives information on the received reflected waves. Based on the received information, the external information detection unit 7400 may perform object detection processing such as detecting people, vehicles, obstacles, signs, or characters on the road surface, or distance detection processing. Based on the received information, the external information detection unit 7400 may perform environmental recognition processing to recognize rainfall, fog, or road surface conditions. Based on the received information, the external information detection unit 7400 may calculate the distance to an object outside the vehicle.

[0133] Furthermore, the external information detection unit 7400 may perform image recognition processing or distance detection processing to recognize people, vehicles, obstacles, signs, or characters on the road surface based on the received image data. The external information detection unit 7400 may perform distortion correction or alignment processing on the received image data, and may also synthesize image data captured by different imaging units 7410 to generate an overhead view image or a panoramic image. The external information detection unit 7400 may also perform viewpoint transformation processing using image data captured by different imaging units 7410.

[0134] The in-vehicle information detection unit 7500 detects information inside the vehicle. The in-vehicle information detection unit 7500 is connected to, for example, a driver status detection unit 7510 that detects the driver's state. The driver status detection unit 7510 may include a camera that images the driver, a biosensor that detects the driver's biometric information, or a microphone that collects sounds inside the vehicle. The biosensor is installed, for example, on the seat or steering wheel and detects the biometric information of a passenger sitting in the seat or a driver holding the steering wheel. Based on the detection information input from the driver status detection unit 7510, the in-vehicle information detection unit 7500 may calculate the driver's level of fatigue or concentration, or determine whether the driver is dozing off. The in-vehicle information detection unit 7500 may perform processing such as noise cancellation on the collected audio signals.

[0135] The integrated control unit 7600 controls the overall operation of the vehicle control system 7000 according to various programs. An input unit 7800 is connected to the integrated control unit 7600. The input unit 7800 is implemented by a device that can be operated by the occupant, such as a touch panel, buttons, a microphone, a switch, or a lever. The integrated control unit 7600 may also receive data obtained by voice recognition of audio input via the microphone. The input unit 7800 may be, for example, a remote control device using infrared or other radio waves, or an external device such as a mobile phone or PDA (Personal Digital Assistant) that supports the operation of the vehicle control system 7000. The input unit 7800 may also be, for example, a camera, in which case the occupant can input information through gestures. Alternatively, data obtained by detecting the movement of a wearable device worn by the occupant may be input. Furthermore, the input unit 7800 may include, for example, an input control circuit that generates an input signal based on information input by a passenger or the like using the input unit 7800 and outputs it to the integrated control unit 7600. By operating this input unit 7800, passengers or the like can input various data to the vehicle control system 7000 or instruct it to perform processing operations.

[0136] The memory unit 7690 may include a ROM (Read Only Memory) for storing various programs executed by a microcomputer, and a RAM (Random Access Memory) for storing various parameters, calculation results, or sensor values. The memory unit 7690 may also be implemented using magnetic storage devices such as an HDD (Hard Disk Drive), semiconductor storage devices, optical storage devices, or magneto-optical storage devices.

[0137] The general-purpose communication interface 7620 is a general-purpose communication interface that mediates communication between various devices present in the external environment 7750. The general-purpose communication interface 7620 may implement cellular communication protocols such as GSM (Global System of Mobile communications), WiMAX (registered trademark), LTE (Registered trademark) (Long Term Evolution), or LTE-A (LTE-Advanced), or other wireless communication protocols such as wireless LAN (also known as Wi-Fi (registered trademark)) or Bluetooth (registered trademark). The general-purpose communication interface 7620 may connect, for example, to devices (e.g., application servers or control servers) located on an external network (e.g., the Internet, a cloud network, or a carrier-specific network) via a base station or access point. Furthermore, the general-purpose communication interface 7620 may connect to terminals located near the vehicle (for example, terminals belonging to the driver, pedestrians, or shops, or MTC (Machine Type Communication) terminals) using, for example, P2P (Peer To Peer) technology.

[0138] The Dedicated Communication Interface 7630 is a communication interface that supports communication protocols developed for use in vehicles. The Dedicated Communication Interface 7630 may implement standard protocols such as WAVE (Wireless Access in Vehicle Environment), DSRC (Dedicated Short Range Communications), or cellular communication protocols, which are combinations of lower-layer IEEE 802.11p and upper-layer IEEE 1609. The Dedicated Communication Interface 7630 typically performs V2X communication, a concept that includes one or more of the following: vehicle-to-vehicle communication, vehicle-to-infrastructure communication, vehicle-to-home communication, and vehicle-to-pedestrian communication.

[0139] The positioning unit 7640 performs positioning by receiving GNSS signals from GNSS (Global Navigation Satellite System) satellites (for example, GPS signals from GPS (Global Positioning System) satellites) and generates location information including the vehicle's latitude, longitude, and altitude. The positioning unit 7640 may also determine its current location by exchanging signals with a wireless access point, or it may acquire location information from a terminal such as a mobile phone, PHS, or smartphone that has positioning capabilities.

[0140] The beacon receiver 7650 receives radio waves or electromagnetic waves transmitted from, for example, a radio station installed on a road, and obtains information such as the current location, traffic congestion, road closures, or travel time. The functions of the beacon receiver 7650 may also be included in the dedicated communication interface 7630 described above.

[0141] The in-vehicle equipment interface 7660 is a communication interface that mediates connections between the microcomputer 7610 and various in-vehicle equipment 7760 located inside the vehicle. The in-vehicle equipment interface 7660 may establish a wireless connection using wireless communication protocols such as Wi-Fi, Bluetooth (registered trademark), NFC (Near Field Communication), or WUSB (Wireless USB). Furthermore, the in-vehicle equipment interface 7660 may establish wired connections such as USB (Universal Serial Bus), HDMI (Registered Trademark) (High-Definition Multimedia Interface), or MHL (Mobile High-Definition Link) via connection terminals (and cables if necessary) not shown. The in-vehicle equipment 7760 may include, for example, at least one of the following: a mobile device or wearable device owned by a passenger, or an information device brought into or installed in the vehicle. The in-vehicle equipment 7760 may also include a navigation device that performs route searching to any destination. The in-vehicle equipment interface 7660 exchanges control signals or data signals with these in-vehicle equipment 7760s.

[0142] The in-vehicle network interface 7680 is an interface that mediates communication between the microcomputer 7610 and the communication network 7010. The in-vehicle network interface 7680 transmits and receives signals and other data in accordance with a predetermined protocol supported by the communication network 7010.

[0143] The microcomputer 7610 of the integrated control unit 7600 controls the vehicle control system 7000 according to various programs based on information acquired via at least one of the following: general-purpose communication I / F 7620, dedicated communication I / F 7630, positioning unit 7640, beacon receiver 7650, in-vehicle equipment I / F 7660, and in-vehicle network I / F 7680. For example, the microcomputer 7610 may calculate control target values ​​for the drive force generator, steering mechanism, or braking device based on acquired in-vehicle and external information, and output control commands to the drivetrain control unit 7100. For example, the microcomputer 7610 may perform coordinated control aimed at realizing ADAS (Advanced Driver Assistance System) functions, including collision avoidance or impact mitigation, following based on distance between vehicles, maintaining vehicle speed, collision warning, or lane departure warning. Furthermore, the microcomputer 7610 may perform cooperative control for purposes such as autonomous driving, where the vehicle drives autonomously without driver intervention, by controlling the drive force generating device, steering mechanism, or braking device, etc., based on the acquired information about the vehicle's surroundings.

[0144] The microcomputer 7610 may generate three-dimensional distance information between the vehicle and surrounding structures, people, and other objects based on information acquired via at least one of the general-purpose communication I / F 7620, dedicated communication I / F 7630, positioning unit 7640, beacon receiver 7650, in-vehicle equipment I / F 7660, and in-vehicle network I / F 7680, and create local map information including surrounding information of the vehicle's current location. Furthermore, the microcomputer 7610 may predict dangers such as vehicle collision, proximity of pedestrians, or entry into a closed road based on the acquired information, and generate warning signals. These warning signals may, for example, be signals to generate a warning sound or illuminate a warning lamp.

[0145] The audio-image output unit 7670 transmits at least one of audio and image output signals to an output device capable of visually or audibly notifying information to the vehicle's occupants or to those outside the vehicle. In the example in Figure 13, the output devices are exemplified as an audio speaker 7710, a display unit 7720, and an instrument panel 7730. The display unit 7720 may include, for example, at least one of an onboard display and a head-up display. The display unit 7720 may have an AR (Augmented Reality) display function. The output devices may be other devices other than these, such as headphones, wearable devices such as glasses-type displays worn by occupants, projectors, or lamps. If the output device is a display device, the display device visually displays the results obtained from various processes performed by the microcomputer 7610 or information received from other control units in various formats such as text, images, tables, graphs, etc. Furthermore, if the output device is an audio output device, the audio output device converts the audio signal, consisting of the reproduced audio data or sound data, into an analog signal and outputs it audibly.

[0146] In the example shown in Figure 13, at least two control units connected via the communication network 7010 may be integrated into a single control unit. Alternatively, individual control units may be composed of multiple control units. Furthermore, the vehicle control system 7000 may include other control units not shown. Also, in the above description, some or all of the functions performed by one control unit may be assigned to other control units. In other words, as long as information is transmitted and received via the communication network 7010, predetermined calculation processing may be performed by any of the control units. Similarly, a sensor or device connected to one control unit may be connected to another control unit, and multiple control units may transmit and receive detection information to and from each other via the communication network 7010.

[0147] Furthermore, the computer programs for realizing each function of the solid-state imaging device 3 or the distance measuring system 1 according to this embodiment, as described with reference to Figures 1 to 12, can be implemented in any of the control units, etc. A computer-readable recording medium containing such a computer program can also be provided. Examples of recording media include magnetic disks, optical disks, magneto-optical disks, and flash memory. Alternatively, the computer programs may be distributed, for example, via a network, without using a recording medium.

[0148] In the vehicle control system 7000 described above, the solid-state imaging device 3 or distance measuring system 1 according to this embodiment, as described with reference to Figures 1 to 12, can be applied to the external information detection unit 7400 of the application example shown in Figure 13.

[0149] Furthermore, at least some components of the solid-state imaging device 3 or the distance measuring system 1 described using Figures 1 to 12 may be implemented in a module for the external information detection unit 7400 shown in Figure 13 (for example, an integrated circuit module consisting of a single die). Alternatively, the solid-state imaging device 3 or the distance measuring system 1 described using Figures 1 to 12 may be implemented by multiple control units of the external information detection unit 7400 shown in Figure 13.

[0150] The embodiments described above may also take the following forms.

[0151] (1) A solid-state imaging device comprising: a pixel array in which pixels, each equipped with a photodetector for detecting the incidence of photons, are arranged in an array; a readout circuit for reading detection signals from the pixels at a sampling period; a counting circuit for counting the number of detection signals read out from the readout circuit at each sampling period in units of macropixels including one or more of the pixels; a storage circuit for storing the pixel values ​​at each sampling period for each macropixel counted by the counting circuit in a storage circuit in units of bins having a width defined based on the sampling period; a control circuit for dynamically determining the number of bins to be assigned to each of the macropixels; and a histogram processing circuit for generating a histogram for each macropixel from the values ​​stored in the bins assigned to that macropixel.

[0152] (2) The solid-state imaging apparatus according to (1), wherein the histogram processing circuit calculates the distance from the histogram to the object being imaged for each macropixel.

[0153] (3) The solid-state imaging apparatus according to (2), wherein the control circuit determines the number of bins to be assigned to each macropixel based on the distance to the object captured for each macropixel.

[0154] (4) The solid-state imaging apparatus according to (3), wherein the control circuit is configured to assign a larger number of bins to each macropixel as the distance to the imaged object increases, and assign a smaller number of bins as the distance to the imaged object decreases.

[0155] (5) The solid-state imaging apparatus according to any one of (1) to (4), wherein the control circuit groups the macro pixels according to the size of the number of bins determined for each macro pixel, determines the exposure time of the pixels belonging to each group, and controls one or more macro pixels belonging to the same group to be exposed at the same time.

[0156] (6) The solid-state imaging apparatus according to any one of (2) to (5), wherein the histogram processing circuit detects one or more peaks present in the histogram for each macropixel and acquires position information for each peak, the control circuit dynamically defines the width of the bins around the peaks of the histogram of the macropixel based on the position of each peak, and the counting circuit stores the count value for each bin within the range to which a bin is assigned for each peak of the macropixel.

[0157] (7) An electronic device comprising: a solid-state imaging device as described in any of (2) to (6); and a light-emitting device that irradiates a target with light within a range detectable by the pixels, wherein the solid-state imaging device captures the light reflected from the target by the light emitted by the light-emitting device to generate a depth image.

[0158] (8) An electronic device comprising: a solid-state imaging device as described in (5); and a light-emitting device that irradiates light within a range detectable by the pixels, wherein the solid-state imaging device outputs data relating to the group to the light-emitting device, and the light-emitting device irradiates light for each group at a timing based on the exposure time.

[0159] (9) An imaging method comprising: a control circuit setting the number of bins to be assigned to each macropixel having one or more pixels in long-distance mode; a counting circuit counting detection signals output based on the light sensed by each pixel in the plurality of macropixels to which bins are assigned and storing them in a memory circuit; a histogram processing circuit calculating the distance to the object photographed for each macropixel based on the value for each bin stored in the memory circuit; and the control circuit determining the number of bins to be assigned to each macropixel based on the calculated distance.

[0160] (10) The imaging method according to (9), wherein the control circuit reduces the number of bins assigned to the macropixel to less than the number of bins assigned to the far-distance mode when the distance calculated in the macropixel is shorter than a predetermined distance or more than the distance that can be measured in the far-distance mode.

[0161] (A1) A pixel array in which pixels equipped with photodetectors for detecting the incidence of photons are arranged in an array; a readout circuit for reading detection signals from the pixels at a sampling period; a counting circuit for counting the number of detection signals read out from the readout circuit at each sampling period in units of macropixels including one or more of the pixels; a storage circuit for storing the pixel values ​​at each sampling period for each macropixel counted by the counting circuit in a storage circuit in units of bins having a width defined based on the sampling period; a control circuit for dynamically determining the number of bins to be assigned to a plurality of macropixels according to the number of peaks acquired for each macropixel; a histogram processing circuit for generating a histogram for each macropixel from the values ​​stored in the bins assigned to the macropixel; wherein the control circuit sets the number of peaks and the number of bins to be assigned to each of the macropixels at the number of peaks; the histogram processing circuit detects a number of peak positions corresponding to the number of peaks; and the control circuit sets parameters for imaging a predetermined range including the peak positions. The histogram processing circuit obtains the distance to each of the macro pixels from the histogram in a predetermined range including the peak position, in a solid-state imaging device.

[0162] (A2) The solid-state imaging device described in (A1) has features that are applicable to a non-contradictory extent from among the features described in (1) to (8).

[0163] (A3) An imaging method comprising: a control circuit setting the number of bins to be assigned to each macropixel having one or more pixels in 1-peak mode; a counting circuit counting detection signals output based on the light sensed by each pixel in the plurality of macropixels to which bins are assigned and storing them in a memory circuit; a histogram processing circuit detecting the range of bins in which the peak of the count value exists for each macropixel based on the bin-by-bin values ​​stored in the memory circuit; and the control circuit dynamically changing the width of the bins based on the range of bins in which each detected peak exists, and determining the timing for counting in the counting circuit.

[0164] The aspects of this disclosure are not limited to the embodiments described above, but include various conceivable variations, and the effects of this disclosure are not limited to those described above. The components in each embodiment may be appropriately combined and applied. That is, various additions, modifications, and partial deletions are possible, as long as they do not deviate from the conceptual idea and spirit of this disclosure derived from the claims and their equivalents.

[0165] 1: Distancing system, 2: Light-emitting device, 3: Solid-state imaging device, 30: Pixel array, 31: Control circuit, 32: Memory circuit, 33: Readout circuit, 34: Counting circuit, 35: Histogram processing circuit, 36: Input / Output I / F, MP1, MP2, MP3, MP4, MP5, MP6, MP7: Macro pixels, P1, P2, P3: Peak

Claims

1. A solid-state imaging device comprising: a pixel array in which pixels, each equipped with a photodetector for detecting the incidence of photons, are arranged in an array; a readout circuit for reading detection signals from the pixels at a sampling period; a counting circuit for counting the number of detection signals read out from the readout circuit at each sampling period in units of macropixels including one or more of the pixels; a storage circuit for storing the pixel values ​​at each sampling period for each macropixel counted by the counting circuit in a storage circuit in units of bins having a width defined based on the sampling period; a control circuit for dynamically determining the number of bins to be assigned to each of the macropixels; and a histogram processing circuit for generating a histogram for each macropixel from the values ​​stored in the bins assigned to that macropixel.

2. The solid-state imaging apparatus according to claim 1, wherein the histogram processing circuit calculates the distance from the histogram to the object being imaged for each macropixel.

3. The solid-state imaging apparatus according to claim 2, wherein the control circuit determines the number of bins to be assigned to each macropixel based on the distance to the object imaged for each macropixel.

4. The solid-state imaging apparatus according to claim 3, wherein the control circuit assigns a larger number of bins to each macropixel as the distance to the imaged object increases, and assigns a smaller number of bins as the distance to the imaged object decreases.

5. The control circuit groups the macropixels according to the size of the number of bins determined for each macropixel, determines the exposure time for each group of macropixels, and controls one or more macropixels belonging to the same group to be exposed at the same time, as described in claim 1.

6. The solid-state imaging apparatus according to claim 2, wherein the histogram processing circuit detects one or more peaks present in the histogram for each macropixel and acquires position information for each peak; the control circuit dynamically defines the width of the bins around the peaks of the histogram of the macropixel based on the position of each peak; and the counting circuit stores the count value for each bin within the range to which a bin is assigned for each peak of the macropixel.

7. An electronic device comprising: a solid-state imaging device according to claim 2; a light-emitting device that irradiates a target with light within a range detectable by the pixels, wherein the solid-state imaging device captures the light reflected from the target by the light emitted by the light-emitting device to generate a depth image.

8. An electronic device comprising: a solid-state imaging device according to claim 5; and a light-emitting device that irradiates light within a range detectable by the pixels, wherein the solid-state imaging device outputs data relating to the group to the light-emitting device, and the light-emitting device irradiates light for each group at a timing based on the exposure time.

9. An imaging method comprising: a control circuit setting the number of bins to be assigned to each macropixel having one or more pixels in long-distance mode; a counting circuit counting detection signals output based on the light sensed by each pixel in the plurality of macropixels to which bins are assigned and storing them in a memory circuit; a histogram processing circuit calculating the distance to the object captured for each macropixel based on the bin values ​​stored in the memory circuit; and the control circuit determining the number of bins to be assigned to each macropixel based on the calculated distance.

10. The imaging method according to claim 9, wherein the control circuit reduces the number of bins assigned to the macropixel to less than the number of bins assigned in the far-distance mode when the distance calculated in the macropixel is shorter than a predetermined distance or more than the distance that can be measured in the far-distance mode.

Images