Solid-state imaging device

Abstract

[Problem] To acquire a high-resolution image in an event-based vision sensor (EVS). [Solution] This solid-state imaging device comprises a pixel array, a drive unit, and a signal processing circuit. In the pixel array, light-receiving pixels are arranged in a two-dimensional array along a first direction and a second direction intersecting the first direction. The drive unit shifts pixels belonging to the pixel array, in a plane including the first direction and the second direction, in units of distance smaller than a light-receiving pixel or in units of light-receiving pixels. The signal processing circuit processes signals output from a plurality of light-receiving pixels. At least some of the light-receiving pixels arranged in the pixel array are first pixels for detecting an event. The drive unit shifts the light-receiving pixels belonging to the pixel array to a plurality of predetermined positions. The signal processing circuit detects an event, for each first pixel, by comparing current data with past data at the same position as the current position.

Description

Solid-state imaging device

[0001] The present disclosure relates to a solid-state imaging device.

[0002] An event-based vision sensor (EVS) is a sensor that performs data output by asynchronously detecting luminance changes in pixels and outputting information such as the coordinates and time of the changed data. To increase the resolution of an EVS, there is a problem that sensitivity decreases when pixels are miniaturized. For general pixels that receive light for each color such as RGB, there is a method of increasing resolution by shifting the pixels during imaging. However, when imaging is performed with shifted pixels in an EVS, a problem arises in that events caused by the shifted pixels are detected regardless of changes in the subject.

[0003] Japanese Unexamined Patent Publication No. 2012-138807

[0004] Accordingly, one non-limiting problem to be solved by the embodiments of the present disclosure is to acquire high-resolution images in an EVS. The problems to be solved by the embodiments of the present disclosure may be, as several non-limiting examples, problems corresponding to the effects described in the embodiments. That is, a problem corresponding to at least any one of the effects described in the description of the embodiments of the present disclosure can be set as the problem to be solved by the present disclosure.

[0005] A solid-state imaging device includes a pixel array, a drive unit, and a signal processing circuit. In the pixel array, light-receiving pixels are arranged in a two-dimensional array along a first direction and a second direction intersecting the first direction. The drive unit shifts the pixels belonging to the pixel array in a plane including the first direction and the second direction, in units of a distance smaller than the light-receiving pixels or in units of the light-receiving pixels. The signal processing circuit processes signals output from the plurality of light-receiving pixels. At least a part of the light-receiving pixels arranged in the pixel array are first pixels that detect events. The drive unit shifts the light-receiving pixels belonging to the pixel array to a plurality of predetermined positions. The signal processing circuit detects an event by comparing, for each of the first pixels, current data and past data at the same position as the current position.

[0006] Each of the first pixels may be provided with multiple memories.

[0007] The memory may store data for each predetermined position of the corresponding first pixel.

[0008] If an event is detected in the corresponding first pixel, the memory corresponding to the location where the event was detected may be reset.

[0009] The signal processing circuit may output the event information acquired by the first pixel with increased resolution, based on the event information detected at each predetermined position in the first pixel.

[0010] The plurality of light-receiving pixels may include a second pixel in which at least a portion of the pixels other than the first pixel acquires brightness information of the object being photographed.

[0011] The signal processing circuit may output the event information acquired by the first pixel with increased resolution, based on the event information detected at each predetermined position in the first pixel, and may also output the grayscale information with increased resolution.

[0012] The first pixel and the second pixel may have different resolutions, the drive unit may shift the pixels belonging to the pixel array on a pixel-by-pixel basis of the pixel with the higher resolution among the first and second pixels, or it may increase the resolution of the pixel with the lower resolution among the first and second pixels.

[0013] The plurality of pixels may be binned, and the first pixel may have the plurality of memories for each binned unit.

[0014] The number of the aforementioned multiple memories may be the same as the number of binned pixels, and each memory may be a memory provided for each pixel.

[0015] The memory used by the aforementioned plurality of memories may be changed in synchronization with the shift performed by the drive unit.

[0016] The signal processing circuit may output event information with an offset added to the coordinates according to the shift amount by the drive unit.

[0017] A schematic block diagram showing an example of a solid-state imaging device according to one embodiment. A schematic diagram showing an example of pixel position according to one embodiment. A schematic diagram showing an example of pixel position according to one embodiment. A schematic diagram showing an example of pixel position according to one embodiment. A schematic diagram showing an example of pixel position according to one embodiment. A timing chart showing an example of memory usage according to one embodiment. A flowchart showing an example of processing by a solid-state imaging device according to one embodiment. A schematic diagram showing an example of pixel arrangement in a solid-state imaging device according to one embodiment. A schematic diagram showing an example of pixel arrangement in a solid-state imaging device according to one embodiment. A schematic diagram showing an example of pixel arrangement in a solid-state imaging device according to one embodiment. A schematic circuit diagram showing an example of memory and subtractor arrangement according to one embodiment. A schematic circuit diagram showing an example of memory and subtractor arrangement according to one embodiment. A schematic diagram showing a monitoring system according to one embodiment.

[0018] The embodiments of this disclosure will now be described with reference to the drawings. The drawings are for illustrative purposes only, and the shape, size, or size ratio of each component in the actual device does not need to be exactly as shown in the drawings. Furthermore, the drawings are simplified, so any other components necessary for implementation should be appropriately provided in addition to those shown in the drawings.

[0019] Figure 1 is a schematic block diagram showing a part of a solid-state imaging device according to one embodiment. The solid-state imaging device 1 comprises at least a pixel array 10, a control circuit 12, a drive unit 14, and a signal processing circuit 16. Other elements for operating the solid-state imaging device 1, such as a power supply, wiring, and optical system, are not shown but may be provided as appropriate.

[0020] Solid-state imaging device 1 is an imaging device that includes at least one pixel related to EVS (hereinafter referred to as the first pixel) which is an event-detectable pixel, and is capable of outputting event detection information at a resolution higher than the resolution of this first pixel.

[0021] The pixel array 10 is a region in the solid-state imaging device 1 that receives incident light. The pixel array 10 is provided with a plurality of pixels 100. The solid-state imaging device 1 can detect events if at least a portion of the pixels 100 arranged in the pixel array 10 are formed by first pixels. The pixels 100 are arranged in a two-dimensional array in the pixel array 10 along a first direction (e.g., line direction) and a second direction intersecting the first direction (e.g., column direction).

[0022] The control circuit 12 is a circuit for controlling at least some of the processes involved in the control of the solid-state imaging device 1, from light reception to image generation and output. In addition, the control circuit 12 may also be a circuit that performs other controls necessary for the solid-state imaging device 1. For example, the control circuit 12 may control the light reception timing in the pixel array 10, the timing of signal output from the pixel array 10 to the signal processing circuit 16, and the processing of the pixel array 10 by the drive unit 14.

[0023] The drive unit 14 shifts the pixels 100 belonging to the pixel array 10 in pixel units or units smaller than pixels, within a plane including the first and second directions. The drive unit 14 is, for example, an actuator, and converts the electrical signals input from the control circuit 12 into a force that shifts the pixels 100 belonging to the pixel array 10, thereby shifting the pixel array 10. The control circuit 12 periodically shifts multiple predetermined positions of the pixels 100 belonging to the pixel array 10, for example.

[0024] The drive unit 14 may be configured to shift the pixel array 10 itself, or, as an alternative example, to shift the pixels 100 belonging to the pixel array 10 within the area reserved as the pixel array 10.

[0025] The signal processing circuit 16 is a circuit that generates and outputs an image based on the signal output from the pixel 100 belonging to the pixel 100. For example, the signal processing circuit 16 generates and outputs an image signal by applying analog-to-digital conversion (AD conversion), high-resolution processing, linear matrix processing, etc., to the signal output from each pixel 100.

[0026] The signal processing circuit 16 can generate an image with a higher resolution than the resolution of the pixels 100 arranged in the pixel array 10, based on the brightness information acquired at each predetermined position shifted by the drive unit 14. The operation of the drive unit 14 and the signal processing circuit 16 also enables higher resolution event detection information acquired at each first pixel. The signal processing circuit 16 can, for example, store a signal value for each first pixel at each imaging timing, and detect events by comparing the currently acquired data with data acquired at the same predetermined position in the past.

[0027] In this disclosure, the separation of the signal processing circuit 16 is not strictly defined. For example, some of the functions of the signal processing circuit 16 described above may be performed in the circuit of the pixel 100 or in the circuit included in the pixel array 10.

[0028] The process described above as a general flow will be explained in more detail below using several embodiments.

[0029] (First Embodiment)

[0030] Figures 2 to 5 schematically show an example of the relative positions of pixels according to one embodiment. In other words, these figures show each predetermined position of the pixel array 10 shifted by the drive unit 14. These figures show four predetermined positions, but the number of predetermined positions is not limited to this, and the amount of shift and the direction of shift may differ according to the number of predetermined positions.

[0031] Figure 2 shows a first predetermined position as an example. At the first predetermined position, the pixels 100 belonging to the pixel array 10 are arranged in a two-dimensional array along the first and second directions within the pixel array 10, as shown in the figure. Figures 3 to 5 show the second to fourth predetermined positions as relative positions to this position.

[0032] Figure 3 shows a second predetermined position as an example. In Figures 3 to 5, the dotted lines indicate the positions of each pixel 100 at the first predetermined position shown in Figure 2, and the arrows shown at the bottom of the figures indicate the direction of movement from the previous predetermined position. The drive unit 14 shifts each pixel 100 to the second predetermined position, which is shifted by 1 / 2 pixel in the first direction relative to the first predetermined position. However, it is not limited to this, and the second predetermined position can be a position shifted by (Δx2, Δy2) relative to the first predetermined position.

[0033] Figure 4 shows an example of a third predetermined position. The drive unit 14 shifts each pixel 100 to a third predetermined position which is shifted by 1 / 2 pixel in the second direction from the second predetermined position shown in Figure 3. However, it is not limited to this, and the third predetermined position can be a position shifted by (Δx3, Δy3) relative to the first predetermined position.

[0034] Figure 5 shows an example of a fourth predetermined position. The drive unit 14 shifts each pixel 100 to the fourth predetermined position, which is shifted by -1 / 2 pixels in the first direction from the third predetermined position shown in Figure 4. However, it is not limited to this, and the fourth predetermined position can be a position shifted by (Δx4, Δy4) relative to the first predetermined position.

[0035] After Figure 5, the drive unit 14 shifts each pixel 100 to a first predetermined position which is shifted by -1 / 2 pixels in the second direction from the fourth predetermined position.

[0036] As an example, as shown above, the drive unit 14 may shift the pixels 100 belonging to the pixel array 10 from a first predetermined position to a fourth predetermined position at unit time intervals. Each pixel 100 outputs an analog signal obtained by photoelectric conversion of the light incident at each predetermined position. That is, the unit time for which the drive unit 14 performs the shift drive may be the same as the unit time for imaging (detection at the first pixel) at the pixel 100.

[0037] The signal processing circuit 16 generates and outputs an image with a resolution higher than the physical resolution of the pixel 100, based on the analog signal output from the pixel 100 at each predetermined position. This high-resolution image generation can be achieved using a common method of generating high-resolution images by shifting pixels.

[0038] As described above, the signal processing circuit 16 detects an event for the first pixel by comparing information acquired at the same predetermined location in the past with the information acquired now. For this reason, each first pixel may be provided with multiple memories. For example, by providing four memories for each first pixel, past data can be stored corresponding to each predetermined location.

[0039] The memory may be provided as part of the pixel circuitry associated with the pixel 100, or it may be provided in another area. As described above, it is desirable that the memory be provided in a number equal to or greater than the number of positions shifted by the drive unit 14.

[0040] In the initial state, for each first pixel, the solid-state imaging device 1 stores data acquired at each predetermined position for event detection, for example, luminance information or gradation information, in the memory corresponding to the predetermined position for the first pixel.

[0041] After that, for each first pixel, the solid-state imaging device 1 compares the data acquired at each predetermined position for event detection with the luminance information stored in the memory corresponding to the position, and determines whether an event has occurred at the predetermined position of the first pixel.

[0042] The solid-state imaging device 1 can detect that an event has occurred, for example, when the difference between the past luminance value and the current luminance value is larger than a predetermined value. In this case, future event detection can be performed by updating (resetting) the value in the memory with the current luminance value.

[0043] When an event is detected, the signal processing circuit 16 can output a high-resolution event detection result by taking the shift amount at the predetermined position into consideration. The signal processing circuit 16 can output the event detection result as an image, or can acquire and output the result as an expression using a distance smaller than the pixel size, for example, as the number of pixels including a decimal point.

[0044] The above processing will be described in more detail including the shift operation of the drive unit 14.

[0045] FIG. 6 is a timing chart showing an example of the use timing of the memory in a certain first pixel according to an embodiment. The first phase, the second phase, the third phase, and the fourth phase respectively represent states of shifting to the first predetermined position, the second predetermined position, the third predetermined position, and the fourth predetermined position. The drive unit 14 starts from the first predetermined position as a reference position, shifts the pixels so as to return to the first predetermined position via the second predetermined position, the third predetermined position, and the fourth predetermined position, and defines the respective states as the first phase, the second phase, the third phase, and the fourth phase.

[0046] The driving unit 14 shifts the pixel 100 from the first predetermined position to the fourth predetermined position in a unit time τ. For example, the driving unit 14 shifts the pixel 100 located at the first predetermined position in the first phase to the second predetermined position in the second phase after the unit time τ. Similarly, the driving unit 14 shifts each predetermined position every unit time τ.

[0047] In the first phase, the first memory corresponding to the first pixel is used. In the initial stage, the first memory stores the luminance value (gradation value) acquired at the first pixel located at the first predetermined position. Similarly, in the second phase after the unit time τ, the luminance value acquired at the second predetermined position is stored in the second memory; in the third phase after the subsequent unit time τ, the luminance value acquired at the third predetermined position is stored in the third memory; and in the fourth phase after the subsequent unit time τ, the luminance value acquired at the fourth predetermined position is stored in the fourth memory.

[0048] As described above, a plurality of memories may be provided such that the memory to be used is changed in synchronization with the shift by the driving unit 14.

[0049] In each phase, the solid-state imaging device 1 obtains a difference by subtracting, with a subtractor, the gradation value based on the light intensity currently acquired by the first pixel and the past gradation value at the same corresponding position stored in the corresponding memory, and when this difference is larger than a predetermined value, it can detect that an event has occurred at the relevant position of the first pixel. In addition, the solid-state imaging device 1 can also determine that a bright transition or a dark transition has occurred by considering the sign of the output of the subtractor.

[0050] Assume that an event occurs in the region corresponding to the third predetermined position of the first pixel at time t = T, that is, a change in the luminance value of the imaging target occurs. In this case, the solid-state imaging device 1 detects that an event has occurred in the first pixel, resets the third memory used in the third phase, and stores the luminance value acquired at time t = T into the third memory.

[0051] A reset may be achieved, for example, by inputting the AZ signal to the subtractor and the corresponding memory to discharge them. When the memory is discharged, the new grayscale value at the time the event was detected is written to the memory, making it possible to respond appropriately to subsequent events.

[0052] The signal processing circuit 16 may, for example, output that an event occurred at the third predetermined position (Δx3, Δy3) for the first pixel located at position (x, y). In other words, the signal processing circuit 16 can output both the position information of the first pixel and the amount of shift from the reference position, that is, it can also output offset information corresponding to the amount of shift.

[0053] Alternatively, the signal processing circuit 16 may, as another example, generate an event detection image with a resolution higher than the physical resolution of the multiple first pixels belonging to the pixel array 10, based on the event detection information at the third predetermined position of the first pixel and the output from other first pixels.

[0054] Figure 7 is a flowchart illustrating an example of processing by a solid-state imaging device 1 according to one embodiment. First, the solid-state imaging device 1 sets the pixels 100 to a reference position using a drive unit 14 (S100). The reference position may be, for example, a first predetermined position. As an example, assume that multiple pixels 100 are arranged at their respective first predetermined positions.

[0055] The solid-state imaging device 1 activates a subtractor corresponding to a first predetermined position for each first pixel (S102). The subtractor is, for example, a pixel circuit within each first pixel, a circuit associated with each first pixel, or a circuit incorporated in the signal processing circuit 16 that corresponds to each first pixel, and is a circuit for detecting the occurrence of an event in the first pixel by comparing the currently acquired grayscale value with past grayscale values.

[0056] The solid-state imaging device 1 activates a memory (first memory) corresponding to a first predetermined position for each first pixel (S104). By activating the first memory, it is possible to compare the grayscale value acquired by the first pixel corresponding to the first predetermined position acquired from this timing onward with the grayscale value acquired by the first pixel corresponding to the first predetermined position in the past.

[0057] If no event occurs, the process proceeds to S106. If an event occurs, the system is reset, the contents of the first memory are rewritten with the acquired gradation value, and the process proceeds to S106.

[0058] The solid-state imaging device 1 deactivates the corresponding memory after imaging at the first predetermined position is completed (S106). By deactivating the first memory, the grayscale values ​​captured at the first predetermined position can be retained.

[0059] The solid-state imaging device 1 deactivates the first memory and then deactivates the corresponding subtractor (S108). By deactivating the subtractor at the first predetermined position, the processing at the first predetermined position, i.e., the processing of the first phase, is completed.

[0060] The solid-state imaging device 1 shifts the pixel 100 after the first phase of processing is completed (S110). When the first phase of processing is completed, the signal processing circuit 16 drives the pixel 100 to shift from the first predetermined position to the second predetermined position.

[0061] After shifting pixel 100, the solid-state imaging device 1 starts the second phase of processing. That is, the solid-state imaging device 1 performs the processing at the second predetermined position, which is the second phase of processing corresponding to each first pixel (S102-S108). The second memory is used in the second phase. After this, the third and fourth phases of processing are performed, and after the fourth phase processing is completed, the device moves on to the first phase of processing.

[0062] In this way, by providing multiple memory locations corresponding to each first pixel, it becomes possible to achieve event detection that exceeds physical limitations through repeated processing.

[0063] (Second Embodiment)

[0064] Next, we will explain the arrangement of the first pixels with some examples. For example, the pixel array 10 may be provided with multiple first pixels and multiple second pixels as pixels 100. The second pixels may be pixels that acquire multiple colors of light for acquiring a general image, pixels that acquire brightness values ​​related to grayscale, or pixels that acquire images such as infrared, but are pixels used for purposes other than those used for event detection.

[0065] As an example not limited to this disclosure, we will explain using a second pixel that receives red (R), green (G), and blue (B) light respectively, and a first pixel that performs event detection. Of course, this does not exclude other color combinations, combinations of three or more colors, infrared, ultraviolet, and pixels for other purposes such as pixels that acquire only brightness values, in which case at least a portion of the second pixel is included.

[0066] Furthermore, the color arrangement is shown as an example only and does not exclude other arrangements. In addition, the pixel size ratio is also shown as an example only and may include pixels of different sizes, for example, to obtain high dynamic range (HDR).

[0067] Figure 8 shows an example of the arrangement of pixels 100 according to one embodiment. A first pixel E, and second pixels R, G, and B are shown. The first pixel E is a pixel that detects events, the second pixel R is a pixel that receives red light, the second pixel G is a pixel that receives green light, and the second pixel B is a pixel that receives blue light.

[0068] The drive unit 14 may, for example, use the position in Figure 8 as the first predetermined position, a position shifted by one pixel in the first direction as the second predetermined position, a position shifted by one pixel in both the first and second directions as the third predetermined position, and a position shifted by one pixel in the second direction as the fourth predetermined position, and shift the pixels 100 (first pixel E, second pixel R, second pixel G, and second pixel B) accordingly.

[0069] Alternatively, the drive unit 14 may, as another example, shift the pixel 100 by designating a position shifted by 1 / 2 pixel in the first direction as the second predetermined position, a position shifted by 1 / 2 pixel in both the first and second directions as the third predetermined position, and a position shifted by 1 / 2 pixel in the second direction as the fourth predetermined position.

[0070] The signal processing circuit 16 enhances the resolution of the output from the first pixel E, and also enhances the resolution of the color image based on the outputs from the second pixel R, the second pixel G, and the second pixel B, and outputs it.

[0071] This configuration allows for event detection in a high-resolution state while simultaneously enhancing the resolution of the image acquired from the second pixel. In other words, it enables the enhancement of resolution for color images, luminance images, infrared images, etc.

[0072] (Third embodiment)

[0073] In the embodiments described above, each of the pixels 100 is shown to function as a first and second pixel, but the invention is not limited thereto. The first and second pixels may be formed by pixels that are at least partially binned.

[0074] Figure 9 shows an example of the arrangement of pixels 100 according to one embodiment. The first pixel E, the second pixel R, and the second pixel B are pixels formed by binning 1 × 2 pixels, and the second pixel G may be a pixel formed by binning 2 × 2 pixels. In other words, pixel 100 can be formed by using a portion of the pixels corresponding to R and B from among the pixels corresponding to the binned R, G, and B as the first pixel E.

[0075] The first pixel E may also be a pixel in which the light-receiving area is formed as a single unit.

[0076] The first pixel E can, for example, have multiple memories for each binned unit. That is, a first pixel E with a size of 1 × 2 pixels can have multiple memories. The first pixel E can also use each of the memories provided in a 1 × 1 pixel pixel circuit as multiple memories.

[0077] The drive unit 14 may, for example, shift the pixel 100 by setting the position shown in Figure 9 as the first predetermined position, the position shifted by one pixel before binning in the first direction as the second predetermined position, the position shifted by one pixel before binning in both the first and second directions as the third predetermined position, and the position shifted by one pixel before binning in the second direction as the fourth predetermined position.

[0078] The signal processing circuit 16 enhances the resolution of the output from the first pixel E and, based on the binned outputs from the second pixel R, second pixel G, and second pixel B, enhances the resolution of the color image and outputs it.

[0079] This configuration allows for the acquisition of a color image using binned 4x4 pixel units, and the formation of 2x2 event detection pixels within these 4x4 pixel units. By increasing the resolution, it becomes possible to achieve equivalent resolution for the event detection unit and the color image formation unit.

[0080] (Fourth Embodiment)

[0081] Figure 10 shows an example of the arrangement of pixels 100 according to one embodiment. As shown in this figure, pixels may be binned, and the first pixel E may be formed as a 1 × 2 pixel in the range of the second pixel G and the second pixel B. In this case, it is possible to increase the resolution of event occurrence information detected from the first pixel without changing the resolution of the image formed from the second pixel.

[0082] In this embodiment as well, similar to the embodiment described above, the drive unit 14 shifts the pixels 100, and the signal processing circuit 16 recombines them to achieve high-resolution event detection.

[0083] Thus, the processing of the present disclosure can be applied even when the first pixel and the second pixel have different resolutions. In this configuration, the drive unit 14 can, for example, shift pixels on a pixel-by-pixel basis among the first and second pixels, which have higher resolution, to increase the resolution of the lower-resolution pixels.

[0084] (Fifth embodiment)

[0085] Figure 11 shows an example of the arrangement of pixels 100 according to one embodiment. As shown in this figure, the image may be binned and the first pixel E may be formed to be 2 × 2 pixels in size so that it is located in the center of the binned pixels.

[0086] In this embodiment as well, similar to the embodiment described above, the drive unit 14 shifts the pixels 100, and the signal processing circuit 16 recombines them to achieve high-resolution event detection.

[0087] (Sixth Embodiment)

[0088] Next, we will explain some examples of the arrangement of memory and subtractors. Subtractors may be provided corresponding to each memory. Note that the photoelectric conversion region is shown in a simplified form in the circuit diagram; this is for illustrative purposes to make the explanation easier to understand and does not preclude the possibility of a more complex pixel circuit.

[0089] Figure 12 is a schematic circuit diagram showing an example of the arrangement of a memory and subtractor according to one embodiment. The first pixel E includes, for example, a photodetector P and a current-voltage conversion circuit (hereinafter referred to as IV converter 102).

[0090] The photodetector P converts incident light into photoelectric power and outputs a current corresponding to the light intensity. The IV converter 102 is a circuit that converts the current output from the photodetector P into a voltage and outputs it.

[0091] The first memory 104A, the second memory 104B, the third memory 104C, and the fourth memory 104D are memories that store the grayscale values ​​acquired in the first, second, third, and fourth phases, respectively. These memories may be composed of, for example, capacitors.

[0092] The first subtractor 106A, the second subtractor 106B, the third subtractor 106C, and the fourth subtractor 106D are circuits that subtract the grayscale values ​​acquired in the first, second, third, and fourth phases, respectively, from the grayscale values ​​stored in the corresponding memory and output the result.

[0093] Each memory and subtractor is activated based on a signal from the control circuit 12. The control circuit 12 controls the drive unit 14 and activates one of the memory and subtractor combinations in synchronization with the position in which the drive unit 14 shifts the pixels 100, thereby generating and outputting the difference between the current grayscale value at each predetermined position and the stored past grayscale value.

[0094] The comparator 108 determines whether the difference output from the subtractor is greater than a predetermined threshold. If the difference output from the subtractor is greater than the predetermined threshold in the positive direction, the comparator 108 detects an event in which the grayscale value becomes brighter, and if the difference output from the subtractor is greater than the predetermined threshold in the negative direction, it detects an event in which the grayscale value becomes darker.

[0095] The detector 110 can determine, based on the output from the comparator 108, which predetermined location has an event occurring in either the positive or negative direction. The signal processing circuit 16 obtains the location of the event based on the information output from the detector 110, and based on this obtained information, can obtain event detection information at a resolution higher than the actual resolution of the first pixel E.

[0096] Furthermore, if the detector 110 detects the occurrence of an event, it can perform a reset operation to overwrite the memory corresponding to the predetermined location where the event occurred with the currently acquired grayscale value.

[0097] The solid-state imaging device 1 can be configured in such a way that multiple memories and subtractors are arranged for each first pixel E.

[0098] (Seventh Embodiment)

[0099] Figure 13 is a schematic circuit diagram showing an example of the arrangement of memory, subtractor, comparator, and determination device according to one embodiment. When binning pixel 100, it is also possible to bin the first pixel E. As an example that is not limited, each of the first pixels E may be equipped with an IV converter, memory, subtractor, comparator, and determination device for each of the photodetectors formed thereon.

[0100] In Figure 13, the first photodetector PA and first converter 102A, the second photodetector PB and second converter 102B, the third photodetector PC and third converter 102C, and the fourth photodetector PD and fourth converter 102D are arranged at the first, second, third, and fourth predetermined positions, but the design is not limited to these arrangements. For example, the first photodetector PA to the third photodetector PC may be treated as an integrated photodetector, and one IV converter 102 may be provided for this integrated photodetector.

[0101] The first photodetector PA, the first converter 102A, the first memory 104A, the first comparator 108A, and the first judge 110A may be formed as pixel circuits for each of the first pixels EA. Similarly, the pixel circuit of the first pixel EB may include a second photodetector PB, a second converter 102B, a second memory 104B, a second comparator 108B, and a second classifier 110B; the pixel circuit of the first pixel EC may include a third photodetector PC, a third converter 102C, a third memory 104CCD, a third comparator 108C, and a third classifier 110C; and the pixel circuit of the first pixel ED may include a fourth photodetector PD, a fourth converter 102D, a fourth memory 104D, a fourth comparator 108D, and a fourth classifier 110D.

[0102] When a first binned pixel E is formed by providing a converter, subtractor, memory, comparator, and classifier for each pixel, the subtractor, memory, comparator, and classifier corresponding to each pre-binning pixel can be used as event detectors corresponding to their respective predetermined positions (phases).

[0103] By implementing it in this way, when binning is performed, it becomes possible to reduce the area of ​​memory and other components by ensuring uniformity as a pixel circuit.

[0104] (Examples of application)

[0105] The solid-state imaging device described above can be used, for example, in video see-through applications in head-mounted displays or in deblurring applications in mobile devices. Furthermore, this solid-state imaging device can also be used for event detection in automotive applications.

[0106] (modified version)

[0107] Figure 14 is a schematic diagram showing a monitoring system using a modified example of the solid-state imaging device in this disclosure. Monitoring system 2 comprises a monitoring device. The monitoring device is an example of the solid-state imaging device in this disclosure that eliminates the shift operation by the drive unit 14 and has eight memories for each event detection pixel or group of event detection pixels.

[0108] The monitoring device rotates the solid-state imaging device by 45 degrees for each phase and performs imaging in each phase. Each event detection pixel is equipped with a corresponding memory for each phase, and moving objects can be detected by comparing them with past information stored in this memory.

[0109] The operation of the memory, subtractor, comparator, etc., can be the same as in the previously described embodiment. Also, as in the previously described embodiment, in addition to the event detection pixels, there are pixels for acquiring other general color images and infrared images.

[0110] This configuration makes it possible to detect the occurrence of events in each direction without having to prepare a large memory space.

[0111] The embodiments described above may also take the following forms.

[0112] (1) A solid-state imaging device comprising: a pixel array in which light-receiving pixels are arranged in a two-dimensional array along a first direction and a second direction intersecting the first direction; a drive unit that shifts pixels belonging to the pixel array in units of distance smaller than the light-receiving pixels or in units of the light-receiving pixels within a plane including the first direction and the second direction; and a signal processing circuit that processes signals output from the plurality of light-receiving pixels, wherein at least a portion of the light-receiving pixels arranged in the pixel array are first pixels that detect events; the drive unit shifts the light-receiving pixels belonging to the pixel array to a plurality of predetermined positions; and the signal processing circuit detects events by comparing current data with past data at the same position as the current position for each of the first pixels.

[0113] (2) The solid-state imaging device according to (1), wherein each of the first pixels is provided with a plurality of memories.

[0114] (3) The solid-state imaging apparatus according to (2), wherein the memory stores data for each predetermined position of the corresponding first pixel.

[0115] (4) The solid-state imaging device according to (3), which resets the memory corresponding to the position where an event is detected when an event is detected in the corresponding first pixel.

[0116] (5) The solid-state imaging apparatus according to any one of (1) to (4), wherein the signal processing circuit outputs the event information acquired by the first pixel with high resolution based on the event information detected at each predetermined position in the first pixel.

[0117] (6) The solid-state imaging device according to any one of (1) to (4), wherein at least a portion of the pixels other than the first pixel are second pixels that acquire brightness information of the object to be photographed.

[0118] (7) The solid-state imaging apparatus according to (6), wherein the signal processing circuit outputs the event information acquired by the first pixel with high resolution and the grayscale information with high resolution, based on the event information detected at each predetermined position in the first pixel.

[0119] (8) The solid-state imaging apparatus according to (6), wherein the first pixel and the second pixel have different resolutions, and the drive unit shifts the pixels belonging to the pixel array on a pixel-by-pixel basis of the pixels with higher resolution among the first and second pixels, and increases the resolution of the pixels with lower resolution among the first and second pixels.

[0120] (9) The solid-state imaging apparatus according to any one of (2) to (4), wherein the plurality of pixels are binned, and the first pixel has the plurality of memories for each binned unit.

[0121] (10) The solid-state imaging apparatus according to (9), wherein the number of memories is the same as the number of binned pixels, and each memory is a memory provided for each pixel.

[0122] (11) The solid-state imaging apparatus according to any one of (2) to (4), wherein the memory used is changed in synchronization with the shift by the drive unit.

[0123] (12) The solid-state imaging apparatus according to (11), wherein the signal processing circuit outputs an offset to the coordinates of the event information to be output according to the amount of shift by the drive unit.

[0124] The aspects of this disclosure are not limited to the embodiments described above, but include various conceivable variations, and the effects of this disclosure are not limited to those described above. The components in each embodiment may be appropriately combined and applied. That is, various additions, modifications, and partial deletions are possible, as long as they do not deviate from the conceptual idea and spirit of this disclosure derived from the claims and their equivalents.

[0125] 1: Solid-state imaging device, 10: Pixel array, 100: Pixel, 102: IV converter, 102A: First converter, 102B: Second converter, 102C: Third converter, 102D: Fourth converter, 104A: First memory, 104B: Second memory, 104C: Third memory, 104D: Fourth memory, 106A: First subtractor, 106B: Second subtractor, 106C: Third subtractor, 106D: Fourth subtractor, 108: Comparator, 108A: First comparator, 108B: Second comparator, 108C: Third comparator, 108D: Fourth comparator, 110: Classifier, 110A: First classifier, 110B: Second classifier, 110C: Third detector, 110D: Fourth detector, E: First pixel, acquires high-resolution image in EVS, R, G, B: Second pixel, P: Photodetector, PA: First photodetector, PB: Second photodetector, PC: Third photodetector, PD: Fourth photodetector, 12: Control circuit, 14: Drive unit, 16: Signal processing circuit, 2: Monitoring system

Claims

1. A solid-state imaging device comprising: a pixel array in which light-receiving pixels are arranged in a two-dimensional array along a first direction and a second direction intersecting the first direction; a drive unit that shifts pixels belonging to the pixel array in units of distance smaller than the light-receiving pixels or in units of the light-receiving pixels within a plane including the first direction and the second direction; and a signal processing circuit that processes signals output from the plurality of light-receiving pixels, wherein at least a portion of the light-receiving pixels arranged in the pixel array are first pixels that detect events; the drive unit shifts the light-receiving pixels belonging to the pixel array to a plurality of predetermined positions; and the signal processing circuit detects events by comparing current data with past data at the same position as the current position for each of the first pixels.

2. The solid-state imaging apparatus according to claim 1, comprising a plurality of memories for each of the first pixels.

3. The solid-state imaging apparatus according to claim 2, wherein the memory stores data for each predetermined position of the corresponding first pixel.

4. The solid-state imaging device according to claim 3, wherein when an event is detected in a corresponding first pixel, the memory corresponding to the position where the event was detected is reset.

5. The solid-state imaging apparatus according to claim 1, wherein the signal processing circuit outputs the event information acquired by the first pixel with high resolution based on the event information detected at each predetermined position in the first pixel.

6. The solid-state imaging device according to claim 1, wherein at least a portion of the pixels other than the first pixel are second pixels that acquire brightness information of the object to be photographed.

7. The solid-state imaging apparatus according to claim 6, wherein the signal processing circuit outputs the event information acquired by the first pixel with increased resolution based on the event information detected at each predetermined position in the first pixel, and also outputs the grayscale information with increased resolution.

8. The solid-state imaging apparatus according to claim 6, wherein the first pixel and the second pixel have different resolutions, and the drive unit shifts the pixels belonging to the pixel array on a pixel-by-pixel basis of the pixels with higher resolution among the first and second pixels, and increases the resolution of the pixels with lower resolution among the first and second pixels.

9. The solid-state imaging apparatus according to claim 2, wherein the plurality of pixels are binned, and the first pixel comprises the plurality of memories for each binned unit.

10. The solid-state imaging apparatus according to claim 9, wherein the number of memories is the same as the number of binned pixels, and each memory is a memory provided for each pixel.

11. The solid-state imaging apparatus according to claim 2, wherein the memory used is changed in synchronization with the shift by the drive unit.

12. The solid-state imaging apparatus according to claim 11, wherein the signal processing circuit outputs an offset to the coordinates of the event information to be output according to the amount of shift by the drive unit.

Images