Sensor device and method for operating a sensor device

Abstract

A sensor device (10) comprises a plurality of pixels (51) each configured to receive light from a scene and to perform photoelectric conversion to generate an electrical signal, event detection circuitry (20) that is configured to detect as event data intensity changes above a predetermined threshold of the light received by each of a first subset of the pixels (51), pixel signal generating circuitry (30) that is configured to generate a pixel signal indicating intensity values of the received light for each pixel (51) of a second subset of the pixels (51), and a control unit (40) that is configured to control the event detection circuitry (20) and the pixel signal generating circuitry (30). Here, a pixel (51) of the first subset of pixels (51) and a pixel (51) of the second subset of pixels (51) that receive light from the same location of the scene are associated with each other, and the control unit (40) is configured to change the pixel signal generation process for at least one pixel (51) of the second subset of pixels (51) when event data have been detected for the pixel (51) of the first subset of pixels (51) that is associated with said at least one pixel (51) of the second subset of pixels (51).

Description

[0001] 73957

[0002] 1

[0003] SENSOR DEVICE AND METHOD FOR OPERATING A SENSOR DEVICE

[0004] FIELD OF THE INVENTION

[0005] The present technology relates to a sensor device and a method for operating a sensor device, in particular, to a sensor device and a method for operating a sensor device that allow synchronization of image data obtained by a conventional image sensor and events detected by an event-based vision sensor, EVS.

[0006] BACKGROUND

[0007] Usual digital video cameras that are e.g. used as stand-alone devices or integrated into other electronic devices such as mobile phones include sensor devices that capture images on a frame basis. Here, changing light conditions e.g. due to motion within a captured scene or due to a varying illumination of the scene lead often to a degradation of the in principle obtainable image quality.

[0008] It is therefore desirable to improve the imaging capacities of sensor devices such as to improve the image quality of captured images.

[0009] SUMMARY OF INVENTION

[0010] To this end, a sensor device is provided that comprises a plurality of pixels each configured to receive light from a scene and to perform photoelectric conversion to generate an electrical signal, event detection circuitry that is configured to detect as event data intensity changes above a predetermined threshold of the light received by each of a first subset of the pixels, pixel signal generating circuitry that is configured to generate a pixel signal indicating intensity values of the received light for each pixel of a second subset of the pixels, and a control unit that is configured to control the event detection circuitry and the pixel signal generating circuitry. Here, a pixel of the first subset of pixels and a pixel of the second subset of pixels that receive light from the same location of the scene are associated with each other, and the control unit is configured to change the pixel signal generation process for at least one pixel of the second subset of pixels when event data have been detected for the pixel of the first subset of pixels that is associated with said at least one pixel of the second subset of pixels.

[0011] Further, a method for operating a sensor device is provided, the method comprising receiving light from a scene and performing photoelectric conversion with a plurality of pixels of the sensor device to generate an electrical signal; detecting, by event detection circuitry of the sensor device, as event data intensity changes above a predetermined threshold of the light received by each of a first subset of the pixels; generating, by pixel signal generating circuitry of the sensor device, a pixel signal indicating intensity values of the received light for each pixel of a second subset of the pixels, wherein a pixel of the first subset of pixels and a pixel of the second subset of pixels that receive light from the same location of the scene are associated with each other; and controlling the event detection circuitry and the pixel signal generating circuitry to change the pixel signal generation process for at least one pixel of the second subset of pixels when event data have been detected for the pixel of the first subset of pixels that is73957

[0012] 2

[0013] associated with said at least one pixel of the second subset of pixels.

[0014] Thus, conventional video camera pixels (second subset of pixels), which generate pixel signals representing the intensity of the received light are supplemented with event detection pixels / EVS pixels (first subset of pixels) that are sensitive to intensity changes above a certain threshold. Once an event is detected in an EVS pixel, the pixel signal generation process in an associated video camera pixel (i.e. a video camera pixel seeing the same spot of a scene, i.e. that receives the same intensity as the EVS pixel) is changed. The detection of an event indicates a change in the observed light intensity (or illumination or brightness). Hence, event detection can be used to trigger a change of the pixel signal generation process such as to adapt it to the new observed light conditions. By doing this on a pixel (or row) basis, it is possible to adapt pixel signal generation to the momentarily present light conditions. This ensures that image generation is performed pixel-wise with optimal conditions regarding signal-to-noise or saturation. Accordingly, the image quality can be improved.

[0015] BRIEF DESCRIPTION OF DRAWINGS

[0016] Fig. 1 is a schematic diagram of a sensor device.

[0017] Fig. 2 is a schematic block diagram of a sensor section.

[0018] Fig. 3 is a schematic block diagram of a pixel array section.

[0019] Fig. 4 is a schematic circuit diagram of a pixel block.

[0020] Fig. 5 is a schematic block diagram illustrating of an event detecting section.

[0021] Fig. 6 is a schematic circuit diagram of a current-voltage converting section.

[0022] Fig. 7 is a schematic circuit diagram of a subtraction section and a quantization section.

[0023] Fig. 8 is a schematic timing chart of an example operation of the sensor section.

[0024] Fig. 9 is a schematic diagram of a frame data generation method based on event data.

[0025] Fig. 10 is a schematic block diagram of another quantization section.

[0026] Fig. 11 is a schematic diagram of another event detecting section.

[0027] Fig. 12 is a schematic block diagram of another pixel array section.

[0028] Fig. 13 is a schematic circuit diagram of another pixel block.3

[0029] Fig. 14 is a schematic block diagram of a scan-type imaging device.

[0030] Fig. 15 is a schematic block diagram of a sensor device and its function.

[0031] Figs. 16A and 16B illustrate schematically another function of the sensor device.

[0032] Figs. 17A and 17B illustrate schematically temporal uncertainties in view of event detection pixel resetting.

[0033] Figs. 18A and 18B illustrate schematically motion blur detection based on event detection pixel resetting.

[0034] Fig. 19 shows different schematic block diagrams of a sensor device.

[0035] Fig. 20 illustrates schematically a process flow of a method for operating a sensor device.

[0036] Fig. 21 is a schematic block diagram of a smart phone.

[0037] Fig. 22 is a schematic block diagram of a vehicle control system.

[0038] Fig. 23 is a diagram of assistance in explaining an example of installation positions of an outside-vehicle information detecting section and an imaging section.

[0039] Figs. 24A and 24B are schematic illustrations of a mobile device and a head mounted display comprising a sensor device

[0040] DETAILED DESCRIPTION

[0041] The present disclosure is directed to an improvement of the quality of images captured by a combination of traditional, frame-based sensors, such as active pixel sensors, APS, and event based or dynamic vision sensors, EVS / DVS, i.e. of hybrid sensors. In particular, the problem is addressed how to reduce the influences of changing light intensities in a scene observed by the frame-based sensor based on the EVS / DVS sensor. The solutions to this problem discussed below are applicable to all sensor types that allow capturing a scene by generating an event data stream together with a full intensity frame sequence. However, in order to ease the description and also in order to cover an important application example, the following description is focused without prejudice on a specific implementation of a hybrid sensor including both, APS pixels and EVS / DVS pixels.

[0042] Thus, first a possible implementation of a hybrid sensor will be described. This is of course purely exemplary. It is to be understood that a differently implemented hybrid sensor could be used.

[0043] Fig. 1 is a diagram illustrating a configuration example of a sensor device 10, which is in the example of Fig. 1 constituted by a sensor chip.4

[0044] The sensor device 10 is a single-chip semiconductor chip and includes a sensor die (substrate) 11, which serves as a plurality of dies (substrates), and a logic die 12 that are stacked. Note that, the sensor device 10 can also include only a single die or three or more stacked dies.

[0045] In the sensor device 10 of Fig. 1, the sensor die 11 includes (a circuit serving as) a sensor section 21, and the logic die 12 includes a logic section 22. Note that, the sensor section 21 can be partly formed on the logic die 12. Further, the logic section 22 can be partly formed on the sensor die 11.

[0046] The sensor section 21 includes pixels configured to perform photoelectric conversion on incident light to generate electrical signals, and generates event data indicating the occurrence of events that are changes in the electrical signal of the pixels. The sensor section 21 supplies the event data to the logic section 22. That is, the sensor section 21 performs imaging of performing, in the pixels, photoelectric conversion on incident light to generate electrical signals, similarly to a synchronous image sensor, for example. The sensor section 21, however, also generates event data indicating the occurrence of events that are changes in the electrical signal of the pixels in addition to generating image data in a frame format (frame data). The sensor section 21 outputs, to the logic section 22, the event data obtained by the imaging.

[0047] Here, the synchronous image sensor is an image sensor configured to perform imaging in synchronization with a vertical synchronization signal and output frame data that is image data in a frame format. The event detection functions of the sensor section 21 can be regarded as asynchronous (an asynchronous image sensor) in contrast to the synchronous image sensor, since as far as event detection is concerned the sensor section 21 does not operate in synchronization with a vertical synchronization signal when outputting event data.

[0048] Note that, the sensor section 21 generates and outputs, other than event data, frame data, similarly to the synchronous image sensor. In addition, the sensor section 21 can output, together with event data, electrical signals of pixels in which events have occurred, as pixel signals that are pixel values of the pixels in frame data.

[0049] The logic section 22 controls the sensor section 21 as needed. Further, the logic section 22 performs various types of data processing, such as data processing of generating frame data on the basis of event data from the sensor section 21 and image processing on frame data from the sensor section 21 or frame data generated on the basis of the event data from the sensor section 21, and outputs data processing results obtained by performing the various types of data processing on the event data and the frame data.

[0050] Fig. 2 is a block diagram illustrating a configuration example of the sensor section 21 of Fig. 1.

[0051] The sensor section 21 includes a pixel array section 31, a driving section 32, an arbiter 33, an AD (Analog to Digital) conversion section 34, and an output section 35.

[0052] The pixel array section 31 preferably includes a plurality of pixels 51 (Fig. 3) arrayed in a two-5

[0053] dimensional lattice pattern. The pixel array section 31 detects, in a case where a change larger than a predetermined threshold (including a change equal to or larger than the threshold as needed) has occurred in (a voltage corresponding to) a photocurrent that is an electrical signal generated by photoelectric conversion in the pixel 51, the change in the photocurrent as an event. In a case of detecting an event, the pixel array section 31 outputs, to the arbiter 33, a request for requesting the output of event data indicating the occurrence of the event. Then, in a case of receiving a response indicating event data output permission from the arbiter 33, the pixel array section 31 outputs the event data to the driving section 32 and the output section 35. In addition, the pixel array section 31 may output an electrical signal of the pixel 51 in which the event has been detected to the AD conversion section 34, as a pixel signal.

[0054] The driving section 32 supplies control signals to the pixel array section 31 to drive the pixel array section 31. For example, the driving section 32 drives the pixel 51 regarding which the pixel array section 31 has output event data, so that the pixel 51 in question supplies (outputs) a pixel signal to the AD conversion section 34. But the driving section 32 may also drive the pixels 51 in a synchronous manner to generate frame data.

[0055] The arbiter 33 arbitrates the requests for requesting the output of event data from the pixel array section 31, and returns responses indicating event data output permission or prohibition to the pixel array section 31.

[0056] The AD conversion section 34 includes, for example, a single-slope ADC (AD converter) (not illustrated) in each column of pixel blocks 41 (Fig. 3) described later, for example. The AD conversion section 34 performs, with the ADC in each column, AD conversion on pixel signals of the pixels 51 of the pixel blocks 41 in the column, and supplies the resultant to the output section 35. Note that, the AD conversion section 34 can perform CDS (Correlated Double Sampling) together with pixel signal AD conversion.

[0057] The output section 35 performs necessary processing on the pixel signals from the AD conversion section 34 and the event data from the pixel array section 31 and supplies the resultant to the logic section 22 (Fig.

[0058] 1).

[0059] Here, a change in the photocurrent generated in the pixel 51 can be recognized as a change in the amount of light entering the pixel 51, so that it can also be said that an event is a change in light amount (a change in light amount larger than the threshold) in the pixel 51.

[0060] Event data indicating the occurrence of an event at least includes location information (coordinates or the like) indicating the location of a pixel block in which a change in light amount, which is the event, has occurred. Besides, the event data can also include the polarity (positive or negative) of the change in light amount.

[0061] With regard to the series of event data that is output from the pixel array section 31 at timings at which events have occurred, it can be said that, as long as the event data interval is the same as the event occurrence interval, the event data implicitly includes time point information indicating (relative) time6

[0062] points at which the events have occurred. However, for example, when the event data is stored in a memory and the event data interval is no longer the same as the event occurrence interval, the time point information implicitly included in the event data is lost. Thus, the output section 35 includes, in event data, time point information indicating (relative) time points at which events have occurred, such as timestamps, before the event data interval is changed from the event occurrence interval. The processing of including time point information in event data can be performed in any block other than the output section 35 as long as the processing is performed before time point information implicitly included in event data is lost.

[0063] Fig. 3 is a block diagram illustrating a configuration example of the pixel array section 31 of Fig. 2.

[0064] The pixel array section 31 may include a plurality of pixel blocks 41. Each pixel block 41 may include the IxJ pixels 51 that are one or more pixels arrayed in I rows and J columns (I and J are integers), an event detecting section 52, and a pixel signal generating section 53. The one or more pixels 51 in the pixel block 41 share the event detecting section 52 and the pixel signal generating section 53. Further, in each column of the pixel blocks 41, a VSL (Vertical Signal Line) for connecting the pixel blocks 41 to the ADC of the AD conversion section 34 is wired.

[0065] The pixel 51 receives light incident from an object and performs photoelectric conversion to generate a photocurrent serving as an electrical signal. The pixel 51 supplies the photocurrent to the event detecting section 52 under the control of the driving section 32.

[0066] The event detecting section 52 detects, as an event, a change larger than the predetermined threshold in photocurrent from each of the pixels 51, under the control of the driving section 32. In a case of detecting an event, the event detecting section 52 may supply, to the arbiter 33 (Fig. 2), a request for requesting the output of event data indicating the occurrence of the event. Then, when receiving a response indicating event data output permission to the request from the arbiter 33, the event detecting section 52 outputs the event data to the driving section 32 and the output section 35.

[0067] The pixel signal generating section 53 generates, in the case where the event detecting section 52 has detected an event, a voltage corresponding to a photocurrent from the pixel 51 as a pixel signal, and supplies the voltage to the AD conversion section 34 through the VSL, under the control of the driving section 32.

[0068] Here, detecting a change larger than the predetermined threshold in photocurrent as an event can also be recognized as detecting, as an event, absence of change larger than the predetermined threshold in photocurrent. The pixel signal generating section 53 can generate a pixel signal in the case where absence of change larger than the predetermined threshold in photocurrent has been detected as an event as well as in the case where a change larger than the predetermined threshold in photocurrent has been detected as an event.

[0069] Further, the event detecting section 52 and the pixel signal generating section 53 may operate7

[0070] independently from each other. This means that while event data are generated by the event detecting section 52, pixel signals are generated by the pixel signal generating section 53 in a synchronous manner to generate frame data. The event detecting section 52 and the pixel signal generating section 53 may also operate in a time multiplexed manner such that each of the sections receives the photocurrent from the pixels 51 only during different time intervals.

[0071] Fig. 4 is a circuit diagram illustrating a configuration example of the pixel block 41. The configuration of Fig. 4 may be referred to as “shared pixel configuration”.

[0072] The pixel block 41 includes, as described with reference to Fig. 3, the pixels 51, the event detecting section 52, and the pixel signal generating section 53.

[0073] The pixel 51 includes a photoelectric conversion element 61 and transfer transistors 62 and 63.

[0074] The photoelectric conversion element 61 includes, for example, a PD (Photodiode). The photoelectric conversion element 61 receives incident light and performs photoelectric conversion to generate charges.

[0075] The transfer transistor 62 includes, for example, an N (Negative)-type MOS (Metal-Oxide-Semiconductor) FET (Field Effect Transistor). The transfer transistor 62 of the n-th pixel 51 of the IxJ pixels 51 in the pixel block 41 is turned on or off in response to a control signal OFGn supplied from the driving section 32 (Fig. 2). When the transfer transistor 62 is turned on, charges generated in the photoelectric conversion element 61 are transferred (supplied) to the event detecting section 52, as a photocurrent.

[0076] The transfer transistor 63 includes, for example, an N-type MOSFET. The transfer transistor 63 of the n-th pixel 51 of the IxJ pixels 51 in the pixel block 41 is turned on or off in response to a control signal TRGn supplied from the driving section 32. When the transfer transistor 63 is turned on, charges generated in the photoelectric conversion element 61 are transferred to an FD 74 of the pixel signal generating section 53.

[0077] The IxJ pixels 51 in the pixel block 41 are connected to the event detecting section 52 of the pixel block 41 through nodes 60. Thus, photocurrents generated in (the photoelectric conversion elements 61 of) the pixels 51 are supplied to the event detecting section 52 through the nodes 60. As a result, the event detecting section 52 receives the sum of photocurrents from all the pixels 51 in the pixel block 41. Thus, the event detecting section 52 detects, as an event, a change in sum of photocurrents supplied from the IxJ pixels 51 in the pixel block 41.

[0078] The pixel signal generating section 53 includes a reset transistor 71, an amplification transistor 72, a selection transistor 73, and the FD (Floating Diffusion) 74.

[0079] The reset transistor 71, the amplification transistor 72, and the selection transistor 73 include, for example, N-type MOSFETs.8

[0080] The reset transistor 71 is turned on or off in response to a control signal RST supplied from the driving section 32 (Fig. 2). When the reset transistor 71 is turned on, the FD 74 is connected to a power supply VDD, and charges accumulated in the FD 74 are thus discharged to the power supply VDD. With this, the FD 74 is reset.

[0081] The amplification transistor 72 has a gate connected to the FD 74, a drain connected to the power supply VDD, and a source connected to the VSL through the selection transistor 73. The amplification transistor 72 is a source follower and outputs a voltage (electrical signal) corresponding to the voltage of the FD 74 supplied to the gate to the VSL through the selection transistor 73.

[0082] The selection transistor 73 is turned on or off in response to a control signal SEL supplied from the driving section 32. When the selection transistor 73 is turned on, a voltage corresponding to the voltage of the FD 74 from the amplification transistor 72 is output to the VSL.

[0083] The FD 74 accumulates charges transferred from the photoelectric conversion elements 61 of the pixels 51 through the transfer transistors 63, and converts the charges to voltages.

[0084] With regard to the pixels 51 and the pixel signal generating section 53, which are configured as described above, the driving section 32 turns on the transfer transistors 62 with control signals OFGn, so that the transfer transistors 62 supply, to the event detecting section 52, photocurrents based on charges generated in the photoelectric conversion elements 61 of the pixels 51. With this, the event detecting section 52 receives a current that is the sum of the photocurrents from all the pixels 51 in the pixel block 41, which might also be only a single pixel.

[0085] When the event detecting section 52 detects, as an event, a change in photocurrent (sum of photocurrents) in the pixel block 41, the driving section 32 may turn off the transfer transistors 62 of all the pixels 51 in the pixel block 41, to thereby stop the supply of the photocurrents to the event detecting section 52. Then, the driving section 32 may sequentially turn on, with the control signals TRGn, the transfer transistors 63 of the pixels 51 in the pixel block 41 in which the event has been detected, so that the transfer transistors 63 transfers charges generated in the photoelectric conversion elements 61 to the FD 74. The FD 74 accumulates the charges transferred from (the photoelectric conversion elements 61 of) the pixels 51. Voltages corresponding to the charges accumulated in the FD 74 are output to the VSL, as pixel signals of the pixels 51, through the amplification transistor 72 and the selection transistor 73.

[0086] As described above, in the sensor section 21 (Fig. 2), in this manner only pixel signals of the pixels 51 in the pixel block 41 in which an event has been detected are sequentially output to the VSL. The pixel signals output to the VSL are supplied to the AD conversion section 34 to be subjected to AD conversion.

[0087] Here, in the pixels 51 in the pixel block 41, the transfer transistors 63 can be turned on not sequentially but simultaneously. In this case, the sum of pixel signals of all the pixels 51 in the pixel block 41 can be output.9

[0088] Further, it is also possible to operate the pixel generating section 53 independently from the event detection by controlling the transfer transistors 63 without regard of the event detection or by entirely omitting the transfer transistors 63. In this case frame data can be generated by the pixel generating section 53 in the conventional, synchronous manner, while the event detection can be performed concurrently. If necessary, photocurrent transfer to the event detecting section 52 and the pixel signal generating section 53 may be time multiplexed such that only one of the two transfer transistors 62, 63 per pixel 51 is open at a given point in time.

[0089] In the pixel array section 31 of Fig. 3, the pixel block 41 includes one or more pixels 51, and the one or more pixels 51 share the event detecting section 52 and the pixel signal generating section 53. Thus, in the case where the pixel block 41 includes a plurality of pixels 51, the numbers of the event detecting sections 52 and the pixel signal generating sections 53 can be reduced as compared to a case where the event detecting section 52 and the pixel signal generating section 53 are provided for each of the pixels 51, with the result that the scale of the pixel array section 31 can be reduced.

[0090] Note that, in the case where the pixel block 41 includes a plurality of pixels 51, the event detecting section 52 can be provided for each of the pixels 51. In the case where the plurality of pixels 51 in the pixel block 41 share the event detecting section 52, events are detected in units of the pixel blocks 41. In the case where the event detecting section 52 is provided for each of the pixels 51, however, events can be detected in units of the pixels 51.

[0091] Yet, even in the case where the plurality of pixels 51 in the pixel block 41 share the single event detecting section 52, events can be detected in units of the pixels 51 when the transfer transistors 62 of the plurality of pixels 51 are temporarily turned on in a time-division manner.

[0092] Further, in a case where there is no need to output pixel signals, the pixel block 41 can be formed without the pixel signal generating section 53. In the case where the pixel block 41 is formed without the pixel signal generating section 53, the sensor section 21 can be formed without the AD conversion section 34 and the transfer transistors 63. In this case, the scale of the sensor section 21 can be reduced. The sensor will then output the address of the pixel (block) in which the event occurred, if necessary with a time stamp.

[0093] Moreover, additional pixels 51 connected to pixel signal generating sections 53, but not to event detecting sections 52 may be provided on the sensor die 11. These pixels 51 may be interleaved with the pixels 51 connected to the event detecting section 52, but may also form a separate pixel array for generating frame data. The separation of the event detection function and the pixel signal generating function may even by implemented as two pixel arrays on different dies that are arranged such as to observe the same scene. Thus basically any pixel arrangement / pixel circuitry might be used that allows obtaining event data with a first subset of pixels 51 and generating of pixel signals with a second subset of pixels 51.

[0094] Fig. 5 is a block diagram illustrating a configuration example of the event detecting section 52 of Fig. 3.10

[0095] The event detecting section 52 includes a current-voltage converting section 81, a buffer 82, a subtraction section 83, a quantization section 84, and a transfer section 85.

[0096] The current-voltage converting section 81 converts (a sum of) photocurrents from the pixels 51 to voltages corresponding to the logarithms of the photocurrents (hereinafter also referred to as a "photovoltage") and supplies the voltages to the buffer 82.

[0097] The buffer 82 buffers photovoltages from the current-voltage converting section 81 and supplies the resultant to the subtraction section 83.

[0098] The subtraction section 83 calculates, at a timing instructed by a row driving signal that is a control signal from the driving section 32, a difference between the current photovoltage and a photovoltage at a timing slightly shifted from the current time, and supplies a difference signal corresponding to the difference to the quantization section 84.

[0099] The quantization section 84 quantizes difference signals from the subtraction section 83 to digital signals and supplies the quantized values of the difference signals to the transfer section 85 as event data.

[0100] The transfer section 85 transfers (outputs), on the basis of event data from the quantization section 84, the event data to the output section 35. That is, the transfer section 85 supplies a request for requesting the output of the event data to the arbiter 33. Then, when receiving a response indicating event data output permission to the request from the arbiter 33, the transfer section 85 outputs the event data to the output section 35.

[0101] Fig. 6 is a circuit diagram illustrating a configuration example of the current-voltage converting section 81 of Fig. 5.

[0102] The current-voltage converting section 81 includes transistors 91 to 93. As the transistors 91 and 93, for example, N-type MOSFETs can be employed. As the transistor 92, for example, a P-type MOSFET can be employed.

[0103] The transistor 91 has a source connected to the gate of the transistor 93, and a photocurrent is supplied from the pixel 51 to the connecting point between the source of the transistor 91 and the gate of the transistor 93. The transistor 91 has a drain connected to the power supply VDD and a gate connected to the drain of the transistor 93.

[0104] The transistor 92 has a source connected to the power supply VDD and a drain connected to the connecting point between the gate of the transistor 91 and the drain of the transistor 93. A predetermined bias voltage Vbias is applied to the gate of the transistor 92. With the bias voltage Vbias, the transistor 92 is turned on or off, and the operation of the current-voltage converting section 81 is turned on or off depending on whether the transistor 92 is turned on or off.73957

[0105] 11

[0106] The source of the transistor 93 is grounded.

[0107] In the current-voltage converting section 81, the transistor 91 has the drain connected on the power supply VDD side and is thus a source follower. The source of the transistor 91, which is the source follower, is connected to the pixels 51 (Fig. 4), so that photocurrents based on charges generated in the photoelectric conversion elements 61 of the pixels 51 flow through the transistor 91 (from the drain to the source). The transistor 91 operates in a subthreshold region, and at the gate of the transistor 91, photovoltages corresponding to the logarithms of the photocurrents flowing through the transistor 91 are generated. As described above, in the current-voltage converting section 81, the transistor 91 converts photocurrents from the pixels 51 to photovoltages corresponding to the logarithms of the photocurrents.

[0108] In the current-voltage converting section 81, the transistor 91 has the gate connected to the connecting point between the drain of the transistor 92 and the drain of the transistor 93, and the photovoltages are output from the connecting point in question.

[0109] Fig. 7 is a circuit diagram illustrating configuration examples of the subtraction section 83 and the quantization section 84 of Fig. 5.

[0110] The subtraction section 83 includes a capacitor 101, an operational amplifier 102, a capacitor 103, and a switch 104. The quantization section 84 includes a comparator 111.

[0111] The capacitor 101 has one end connected to the output terminal of the buffer 82 (Fig. 5) and the other end connected to the input terminal (inverting input terminal) of the operational amplifier 102. Thus, photovoltages are input to the input terminal of the operational amplifier 102 through the capacitor 101.

[0112] The operational amplifier 102 has an output terminal connected to the non-inverting input terminal (+) of the comparator 111.

[0113] The capacitor 103 has one end connected to the input terminal of the operational amplifier 102 and the other end connected to the output terminal of the operational amplifier 102.

[0114] The switch 104 is connected to the capacitor 103 to switch the connections between the ends of the capacitor 103. The switch 104 is turned on or off in response to a row driving signal that is a control signal from the driving section 32, to thereby switch the connections between the ends of the capacitor 103.

[0115] A photovoltage on the buffer 82 (Fig. 5) side of the capacitor 101 when the switch 104 is on is denoted by Vinit, and the capacitance (electrostatic capacitance) of the capacitor 101 is denoted by Cl. The input terminal of the operational amplifier 102 serves as a virtual ground terminal, and a charge Qinit that is accumulated in the capacitor 101 in the case where the switch 104 is on is expressed by Expression (1).73957

[0116] 12

[0117] Qinit = Cl x Vinit (1)

[0118] Further, in the case where the switch 104 is on, the connection between the ends of the capacitor 103 is cut (short-circuited), so that no charge is accumulated in the capacitor 103.

[0119] When a photovoltage on the buffer 82 (Fig. 5) side of the capacitor 101 in the case where the switch 104 has thereafter been turned off is denoted by Vafter, a charge Qafter that is accumulated in the capacitor 101 in the case where the switch 104 is off is expressed by Expression (2).

[0120] Qafter = Cl x Vafter (2)

[0121] When the capacitance of the capacitor 103 is denoted by C2 and the output voltage of the operational amplifier 102 is denoted by Vout, a charge Q2 that is accumulated in the capacitor 103 is expressed by Expression (3).

[0122] Q2 = -C2 x Vout (3)

[0123] Since the total amount of charges in the capacitors 101 and 103 does not change before and after the switch 104 is turned off, Expression (4) is established.

[0124] Qinit = Qafter + Q2 (4)

[0125] When Expression (1) to Expression (3) are substituted for Expression (4), Expression (5) is obtained.

[0126] Vout = - (C1 / C2) x (Vafter - Vinit) (5)

[0127] With Expression (5), the subtraction section 83 subtracts the photovoltage Vinit from the photovoltage Vafter, that is, calculates the difference signal (Vout) corresponding to a difference Vafter - Vinit between the photovoltages Vafter and Vinit. With Expression (5), the subtraction gain of the subtraction section 83 is C1 / C2. Since the maximum gain is normally desired, Cl is preferably set to a large value and C2 is preferably set to a small value. Meanwhile, when C2 is too small, kTC noise increases, resulting in a risk of deteriorated noise characteristics. Thus, the capacitance C2 can only be reduced in a range that achieves acceptable noise. Further, since the pixel blocks 41 each have installed therein the event detecting section 52 including the subtraction section 83, the capacitances Cl and C2 have space constraints. In consideration of these matters, the values of the capacitances Cl and C2 are determined.

[0128] The comparator 111 compares a difference signal from the subtraction section 83 with a predetermined threshold (voltage) Vth (>0) applied to the inverting input terminal (-), thereby quantizing the difference signal. The comparator 111 outputs the quantized value obtained by the quantization to the transfer section 85 as event data.

[0129] For example, in a case where a difference signal is larger than the threshold Vth, the comparator 11173957

[0130] 13

[0131] outputs an H (High) level indicating 1, as event data indicating the occurrence of an event. In a case where a difference signal is not larger than the threshold Vth, the comparator 111 outputs an L (Low) level indicating 0, as event data indicating that no event has occurred.

[0132] The transfer section 85 supplies a request to the arbiter 33 in a case where it is confirmed on the basis of event data from the quantization section 84 that a change in light amount that is an event has occurred, that is, in the case where the difference signal (Vout) is larger than the threshold Vth. When receiving a response indicating event data output permission, the transfer section 85 outputs the event data indicating the occurrence of the event (for example, H level) to the output section 35.

[0133] The output section 35 includes, in event data from the transfer section 85, location / address information regarding (the pixel block 41 including) the pixel 51 in which an event indicated by the event data has occurred and time point information indicating a time point at which the event has occurred, and further, as needed, the polarity of a change in light amount that is the event, i.e. whether the intensity did increase or decrease. The output section 35 outputs the event data.

[0134] As the data format of event data including location information regarding the pixel 51 in which an event has occurred, time point information indicating a time point at which the event has occurred, and the polarity of a change in light amount that is the event, for example, the data format called "AER (Address Event Representation)" can be employed.

[0135] Note that, a gain A of the entire event detecting section 52 is expressed by the following expression where the gain of the current-voltage converting section 81 is denoted by CGiogand the gain of the buffer 82 is 1.

[0136] A = CGiogC 1 / C2 (ZiPhoto_n) (6)

[0137] Here, iPhoto_n denotes a photocurrent of the n-th pixel 51 of the IxJ pixels 51 in the pixel block 41. In Expression (6), E denotes the summation of n that takes integers ranging from 1 to HJ.

[0138] Note that, the pixel 51 can receive any light as incident light with an optical fdter through which predetermined light passes, such as a color fdter. For example, in a case where the pixel 51 receives visible light as incident light, event data indicates the occurrence of changes in pixel value in images including visible objects. Further, for example, in a case where the pixel 51 receives, as incident light, infrared light, millimeter waves, or the like for ranging, event data indicates the occurrence of changes in distances to objects. In addition, for example, in a case where the pixel 51 receives infrared light for temperature measurement, as incident light, event data indicates the occurrence of changes in temperature of objects. In the present embodiment, the pixel 51 is assumed to receive visible light as incident light.

[0139] Fig. 8 is a timing chart illustrating an example of the operation of the sensor section 21 of Fig. 2.

[0140] At Timing TO, the driving section 32 changes all the control signals OFGn from the L level to the H level, thereby turning on the transfer transistors 62 of all the pixels 51 in the pixel block 41. With this, the73957

[0141] 14

[0142] sum of photocurrents from all the pixels 51 in the pixel block 41 is supplied to the event detecting section 52. Here, the control signals TRGn are all at the L level, and hence the transfer transistors 63 of all the pixels 51 are off.

[0143] For example, at Timing Tl, when detecting an event, the event detecting section 52 outputs event data at the H level in response to the detection of the event.

[0144] At Timing T2, the driving section 32 sets all the control signals OFGn to the L level on the basis of the event data at the H level, to stop the supply of the photocurrents from the pixels 51 to the event detecting section 52. Further, the driving section 32 sets the control signal SEL to the H level, and sets the control signal RST to the H level over a certain period of time, to control the FD 74 to discharge the charges to the power supply VDD, thereby resetting the FD 74. The pixel signal generating section 53 outputs, as a reset level, a pixel signal corresponding to the voltage of the FD 74 when the FD 74 has been reset, and the AD conversion section 34 performs AD conversion on the reset level.

[0145] At Timing T3 after the reset level AD conversion, the driving section 32 sets a control signal TRG1 to the H level over a certain period to control the first pixel 51 in the pixel block 41 in which the event has been detected (or which is triggered for other reasons, as e.g. time multiplexed output and / or synchronous readout of frame data) to transfer, to the FD 74, charges generated by photoelectric conversion in (the photoelectric conversion element 61 of) the first pixel 51. The pixel signal generating section 53 outputs, as a signal level, a pixel signal corresponding to the voltage of the FD 74 to which the charges have been transferred from the pixel 51, and the AD conversion section 34 performs AD conversion on the signal level.

[0146] The AD conversion section 34 outputs, to the output section 35, a difference between the signal level and the reset level obtained after the AD conversion, as a pixel signal serving as a pixel value of the image (frame data).

[0147] Here, the processing of obtaining a difference between a signal level and a reset level as a pixel signal serving as a pixel value of an image is called "CDS." CDS can be performed after the AD conversion of a signal level and a reset level, or can be simultaneously performed with the AD conversion of a signal level and a reset level in a case where the AD conversion section 34 performs single-slope AD conversion. In the latter case, AD conversion is performed on the signal level by using the AD conversion result of the reset level as an initial value.

[0148] At Timing T4 after the AD conversion of the pixel signal of the first pixel 51 in the pixel block 41, the driving section 32 sets a control signal TRG2 to the H level over a certain period of time to control the second pixel 51 in the pixel block 41 in which the event has been detected to output a pixel signal.

[0149] In the sensor section 21, similar processing is executed thereafter, so that pixel signals of the pixels 51 in the pixel block 41 in which the event has been detected are sequentially output.When the pixel signals of all the pixels 51 in the pixel block 41 are output, the driving section 32 sets all the control signals OFGn to the H level to turn on the transfer transistors 62 of all the pixels 51 in the pixel block 41.

[0150] Fig. 9 is a diagram illustrating an example of a frame data generation method based on event data.

[0151] The logic section 22 sets a frame interval and a frame width on the basis of an externally input command, for example. Here, the frame interval represents the interval of frames of frame data that is generated on the basis of event data. The frame width represents the time width of event data that is used for generating frame data on a single frame. A frame interval and a frame width that are set by the logic section 22 are also referred to as a "set frame interval" and a "set frame width," respectively.

[0152] The logic section 22 generates, on the basis of the set frame interval, the set frame width, and event data from the sensor section 21, frame data that is image data in a frame format, to thereby convert the event data to the frame data.

[0153] That is, the logic section 22 generates, in each set frame interval, frame data on the basis of event data in the set frame width from the beginning of the set frame interval.

[0154] Here, it is assumed that event data includes time point information ti indicating a time point at which an event has occurred (hereinafter also referred to as an "event time point") and coordinates (x, y) serving as location information regarding (the pixel block 41 including) the pixel 51 in which the event has occurred (hereinafter also referred to as an "event location").

[0155] In Fig. 9, in a three-dimensional space (time and space) with the x axis, the y axis, and the time axis t, points representing event data are plotted on the basis of the event time point t and the event location (coordinates) (x, y) included in the event data.

[0156] That is, when a location (x, y, t) on the three-dimensional space indicated by the event time point t and the event location (x, y) included in event data is regarded as the space-time location of an event, in Fig. 9, the points representing the event data are plotted on the space-time locations (x, y, t) of the events.

[0157] The logic section 22 starts to generate frame data on the basis of event data by using, as a generation start time point at which frame data generation starts, a predetermined time point, for example, a time point at which frame data generation is externally instructed or a time point at which the sensor device 10 is powered on.

[0158] Here, cuboids each having the set frame width in the direction of the time axis t in the set frame intervals, which appear from the generation start time point, are referred to as a "frame volume." The size of the frame volume in the x-axis direction or the y-axis direction is equal to the number of the pixel blocks 41 or the pixels 51 in the x-axis direction or the y-axis direction, for example.16

[0159] The logic section 22 generates, in each set frame interval, frame data on a single frame on the basis of event data in the frame volume having the set frame width from the beginning of the set frame interval.

[0160] Frame data can be generated by, for example, setting white to a pixel (pixel value) in a frame at the event location (x, y) included in event data and setting a predetermined color such as gray to pixels at other locations in the frame.

[0161] Besides, in a case where event data includes the polarity of a change in light amount that is an event, frame data can be generated in consideration of the polarity included in the event data. For example, white can be set to pixels in the case a positive polarity, while black can be set to pixels in the case of a negative polarity.

[0162] In addition, in the case where pixel signals of the pixels 51 are also output when event data is output as described with reference to Fig. 3 and Fig. 4, frame data can be generated on the basis of the event data by using the pixel signals of the pixels 51. That is, frame data can be generated by setting, in a frame, a pixel at the event location (x, y) (in a block corresponding to the pixel block 41) included in event data to a pixel signal of the pixel 51 at the location (x, y) and setting a predetermined color such as gray to pixels at other locations.

[0163] Note that, in the frame volume, there are a plurality of pieces of event data that are different in the event time point t but the same in the event location (x, y) in some cases. In this case, for example, event data at the latest or oldest event time point t can be prioritized. Further, in the case where event data includes polarities, the polarities of a plurality of pieces of event data that are different in the event time point t but the same in the event location (x, y) can be added together, and a pixel value based on the added value obtained by the addition can be set to a pixel at the event location (x, y).

[0164] Here, in a case where the frame width and the frame interval are the same, the frame volumes are adjacent to each other without any gap. Further, in a case where the frame interval is larger than the frame width, the frame volumes are arranged with gaps. In a case where the frame width is larger than the frame interval, the frame volumes are arranged to be partly overlapped with each other.

[0165] As explained above, the pixel signal generating section 53 may also generate frame data in the conventional, synchronous manner.

[0166] Fig. 10 is a block diagram illustrating another configuration example of the quantization section 84 of Fig.

[0167] 5.

[0168] Note that, in Fig. 10, parts corresponding to those in the case of Fig. 7 are denoted by the same reference signs, and the description thereof is omitted as appropriate below.

[0169] In Fig. 10, the quantization section 84 includes comparators 111 and 112 and an output section 113.17

[0170] Thus, the quantization section 84 of Fig. 10 is similar to the case of Fig. 7 in including the comparator 111. However, the quantization section 84 of Fig. 10 is different from the case of Fig. 7 in newly including the comparator 112 and the output section 113.

[0171] The event detecting section 52 (Fig. 5) including the quantization section 84 of Fig. 10 detects, in addition to events, the polarities of changes in light amount that are events.

[0172] In the quantization section 84 of Fig. 10, the comparator 111 outputs, in the case where a difference signal is larger than the threshold Vth, the H level indicating 1, as event data indicating the occurrence of an event having the positive polarity. The comparator 111 outputs, in the case where a difference signal is not larger than the threshold Vth, the L level indicating 0, as event data indicating that no event having the positive polarity has occurred.

[0173] Further, in the quantization section 84 of Fig. 10, a threshold Vth' (<Vth) is supplied to the non-inverting input terminal (+) of the comparator 112, and difference signals are supplied to the inverting input terminal (-) of the comparator 112 from the subtraction section 83. Here, for the sake of simple description, it is assumed that the threshold Vth' is equal to -Vth, for example, which needs however not to be the case.

[0174] The comparator 112 compares a difference signal from the subtraction section 83 with the threshold Vth' applied to the inverting input terminal (-), thereby quantizing the difference signal. The comparator 112 outputs, as event data, the quantized value obtained by the quantization.

[0175] For example, in a case where a difference signal is smaller than the threshold Vth' (the absolute value of the difference signal having a negative value is larger than the threshold Vth), the comparator 112 outputs the H level indicating 1, as event data indicating the occurrence of an event having the negative polarity. Further, in a case where a difference signal is not smaller than the threshold Vth' (the absolute value of the difference signal having a negative value is not larger than the threshold Vth), the comparator 112 outputs the L level indicating 0, as event data indicating that no event having the negative polarity has occurred.

[0176] The output section 113 outputs, on the basis of event data output from the comparators 111 and 112, event data indicating the occurrence of an event having the positive polarity, event data indicating the occurrence of an event having the negative polarity, or event data indicating that no event has occurred to the transfer section 85.

[0177] For example, the output section 113 outputs, in a case where event data from the comparator 111 is the H level indicating 1, +V volts indicating +1, as event data indicating the occurrence of an event having the positive polarity, to the transfer section 85. Further, the output section 113 outputs, in a case where event data from the comparator 112 is the H level indicating 1, -V volts indicating -1, as event data indicating the occurrence of an event having the negative polarity, to the transfer section 85. In addition, the output section 113 outputs, in a case where each event data from the comparators 111 and 112 is the L73957

[0178] 18

[0179] level indicating 0, 0 volts (GND level) indicating 0, as event data indicating that no event has occurred, to the transfer section 85.

[0180] The transfer section 85 supplies a request to the arbiter 33 in the case where it is confirmed on the basis of event data from the output section 113 of the quantization section 84 that a change in light amount that is an event having the positive polarity or the negative polarity has occurred. After receiving a response indicating event data output permission, the transfer section 85 outputs event data indicating the occurrence of the event having the positive polarity or the negative polarity (+V volts indicating 1 or -V volts indicating -1) to the output section 35.

[0181] Preferably, the quantization section 84 has a configuration as illustrated in Fig. 10.

[0182] Fig. 11 is a diagram illustrating another configuration example of the event detecting section 52.

[0183] In Fig. 11, the event detecting section 52 includes a subtractor 430, a quantizer 440, a memory 451, and a controller 452. The subtractor 430 and the quantizer 440 correspond to the subtraction section 83 and the quantization section 84, respectively.

[0184] Note that, in Fig. 11, the event detecting section 52 further includes blocks corresponding to the currentvoltage converting section 81 and the buffer 82, but the illustrations of the blocks are omitted in Fig. 11.

[0185] The subtractor 430 includes a capacitor 431, an operational amplifier 432, a capacitor 433, and a switch 434. The capacitor 431, the operational amplifier 432, the capacitor 433, and the switch 434 correspond to the capacitor 101, the operational amplifier 102, the capacitor 103, and the switch 104, respectively.

[0186] The quantizer 440 includes a comparator 441. The comparator 441 corresponds to the comparator 111.

[0187] The comparator 441 compares a voltage signal (difference signal) from the subtractor 430 with the predetermined threshold voltage Vth applied to the inverting input terminal (-). The comparator 441 outputs a signal indicating the comparison result, as a detection signal (quantized value).

[0188] The voltage signal from the subtractor 430 may be input to the input terminal (-) of the comparator 441, and the predetermined threshold voltage Vth may be input to the input terminal (+) of the comparator 441.

[0189] The controller 452 supplies the predetermined threshold voltage Vth applied to the inverting input terminal (-) of the comparator 441. The threshold voltage Vth which is supplied may be changed in a time-division manner. For example, the controller 452 supplies a threshold voltage Vthl corresponding to ON events (for example, positive changes in photocurrent) and a threshold voltage Vth2 corresponding to OFF events (for example, negative changes in photocurrent) at different timings to allow the single comparator to detect a plurality of types of address events (events).

[0190] The memory 451 accumulates output from the comparator 441 on the basis of Sample signals supplied73957

[0191] 19

[0192] from the controller 452. The memory 451 may be a sampling circuit, such as a switch, plastic, or capacitor, or a digital memory circuit, such as a latch or flip-flop. For example, the memory 451 may hold, in a period in which the threshold voltage Vth2 corresponding to OFF events is supplied to the inverting input terminal (-) of the comparator 441, the result of comparison by the comparator 441 using the threshold voltage Vthl corresponding to ON events. Note that, the memory 451 may be omitted, may be provided inside the pixel (pixel block 41), or may be provided outside the pixel.

[0193] Fig. 12 is a block diagram illustrating another configuration example of the pixel array section 31 of Fig.

[0194] 2.

[0195] Note that, in Fig. 12, parts corresponding to those in the case of Fig. 3 are denoted by the same reference signs, and the description thereof is omitted as appropriate below.

[0196] In Fig. 12, the pixel array section 31 includes the plurality of pixel blocks 41. The pixel block 41 includes the lx J pixels 51 that are one or more pixels and the event detecting section 52.

[0197] Thus, the pixel array section 31 of Fig. 12 is similar to the case of Fig. 3 in that the pixel array section 31 includes the plurality of pixel blocks 41 and that the pixel block 41 includes one or more pixels 51 and the event detecting section 52. However, the pixel array section 31 of Fig. 12 is different from the case of Fig.

[0198] 3 in that the pixel block 41 does not include the pixel signal generating section 53.

[0199] As described above, in the pixel array section 31 of Fig. 12, the pixel block 41 does not include the pixel signal generating section 53, so that the sensor section 21 (Fig. 2) can be formed without the AD conversion section 34. As further described above, the pixel signal generating section 53 and pixels 51 feeding into it may be placed on a different part of the die 11 or even on another sensor chip.

[0200] Fig. 13 is a circuit diagram illustrating a configuration example of the pixel block 41 of Fig. 12.

[0201] As described with reference to Fig. 12, the pixel block 41 includes the pixels 51 and the event detecting section 52, but does not include the pixel signal generating section 53.

[0202] In this case, the pixel 51 can only include the photoelectric conversion element 61 without the transfer transistors 62 and 63.

[0203] Note that, in the case where the pixel 51 has the configuration illustrated in Fig. 13, the event detecting section 52 can output a voltage corresponding to a photocurrent from the pixel 51, as a pixel signal.

[0204] Above, the sensor device 10 was described to be an asynchronous imaging device configured to read out events by the asynchronous readout system. However, the event readout system is not limited to the asynchronous readout system and may be the synchronous readout system. An imaging device to which the synchronous readout system is applied is a scan type imaging device that is the same as a general imaging device configured to perform imaging at a predetermined frame rate. Further, the event data73957

[0205] 20

[0206] detection may be performed asynchronously, while the pixel signal generation may be performed synchronously.

[0207] Fig. 14 is a block diagram illustrating a configuration example of a scan type imaging device.

[0208] As illustrated in Fig. 14, an imaging device 510 includes a pixel array section 521, a driving section 522, a signal processing section 525, a read-out region selecting section 527, and a signal generating section 528.

[0209] The pixel array section 521 includes a plurality of pixels 530. The plurality of pixels 530 each output an output signal in response to a selection signal from the read-out region selecting section 527. The plurality of pixels 530 can each include an in-pixel quantizer as illustrated in Fig. 11, for example. The plurality of pixels 530 output output signals corresponding to the amounts of change in light intensity. The plurality of pixels 530 may be two-dimensionally disposed in a matrix as illustrated in Fig. 14.

[0210] The driving section 522 drives the plurality of pixels 530, so that the pixels 530 output pixel signals generated in the pixels 530 to the signal processing section 525 through an output line 514. Note that, the driving section 522 and the signal processing section 525 are circuit sections for acquiring grayscale information. Thus, in a case where only event information (event data) is acquired, the driving section 522 and the signal processing section 525 may be omitted.

[0211] The read-out region selecting section 527 selects some of the plurality of pixels 530 included in the pixel array section 521. For example, the read-out region selecting section 527 selects one or a plurality of rows included in the two-dimensional matrix structure corresponding to the pixel array section 521. The readout region selecting section 527 sequentially selects one or a plurality of rows on the basis of a cycle set in advance. Further, the read-out region selecting section 527 may determine a selection region on the basis of requests from the pixels 530 in the pixel array section 521.

[0212] The signal generating section 528 generates, on the basis of output signals of the pixels 530 selected by the read-out region selecting section 527, event signals corresponding to active pixels in which events have been detected of the selected pixels 530. The events mean an event that the intensity of light changes. The active pixels mean the pixel 530 in which the amount of change in light intensity corresponding to an output signal exceeds or falls below a threshold set in advance. For example, the signal generating section 528 compares output signals from the pixels 530 with a reference signal, and detects, as an active pixel, a pixel that outputs an output signal larger or smaller than the reference signal. The signal generating section 528 generates an event signal (event data) corresponding to the active pixel.

[0213] The signal generating section 528 can include, for example, a column selecting circuit configured to arbitrate signals input to the signal generating section 528. Further, the signal generating section 528 can output not only information regarding active pixels in which events have been detected, but also information regarding non-active pixels in which no event has been detected, i.e. it can operate as a conventional synchronous image sensorthat generates a series of consecutive image frames.73957

[0214] 21

[0215] The signal generating section 528 outputs, through an output line 515, address information and timestamp information (for example, (X, Y, T)) regarding the active pixels in which the events have been detected. However, the data that is output from the signal generating section 528 may not only be the address information and the timestamp information, but also information in a frame format (for example, (0, 0, 1, o, -)).

[0216] In the above description a sensor device 10 has been described in which event data generation and pixel signal generation may depend on each other or may be independent of each other. Moreover, pixels 51 may be shared between event detection circuitry and pixel signal generating circuitry either by respective circuitry or by time multiplexing. But pixels 51 may also be divided such that there are event detection pixels and pixel signal generating pixels. These pixels 51 may be interleaved in the same pixel array or may be part of different pixel arrays on the same die or even be arranged on different dies.

[0217] In all these configurations event detection circuitry operates on a first subset of pixels 51 and is configured to detect as event data intensity changes above a predetermined threshold of the light received by each of the first subset of the pixels 51, and pixel signal generating circuitry operates on a second subset of pixels 51 and is configured to generate a pixel signal indicating intensity values of the received light for each pixel of the second subset of the pixels 51. Here, at least a part of the pixels 51 may belong to the first subset of pixels 51 and the second subset of pixels 51, i.e. it is a shared pixel 51 or a pixel 51 whose output is distributed in a time multiplexed manner. But the pixels 51 in the first subset of pixels 51 may also be different from pixels 51 in the second subset of pixels 51, i.e. the sensor device 10 may comprise divided pixels 51 for event detection and intensity frame generation.

[0218] Fig. 15 shows schematically an according exemplary implementation of a sensor device 10. As stated above, the sensor device 10 comprises a plurality of pixels 51 that are each configured to receive light from a scene and perform photoelectric conversion to generate an electrical signal. Here, the pixels 51 might be arranged in pixel blocks 41 as described with respect to Fig. 3. For the ease of description it is assumed in the following without limitation that there is only one pixel 51 per pixel block 41, i.e. that pixels 51 and pixel blocks 41 refer to the same structural elements. However, the following description could be easily generalized by understanding any reference to a pixel 51 as a reference to a pixel block 41 comprising several pixels 51. In Fig. 15 a two-dimensional array of pixels 51 is shown. However, also a different arrangement of pixels 51 is possible, such as e.g. a one-dimensional line or a scattered, arbitrary arrangement. The following reference to a two-dimensional pixel array is only chosen to ease the description.

[0219] The sensor device 10 comprises event detection circuitry 20 that is configured to detect as event data intensity changes above a predetermined threshold of the light received by each of a first subset SI of the pixels 51. The event detection circuitry 20 is basically constituted by the event detecting sections 52 as described above that are capable to receive the photocurrent of the pixels 51 and to detect events due to intensity changes of the received light that are larger than predetermined (but potentially dynamically adjustable) event thresholds. Although the event detection circuitry 20 has been depicted in Fig. 15 as a73957

[0220] 22

[0221] block adjacent to the pixels 51 this has to be understood as merely symbolic. As discussed above, the event detecting sections 52 of the event detection circuitry 20 may be arranged close or next to the pixels 51. However, the event detecting sections 52 could also be grouped together in a dedicated area on the sensor chip. In particular, the event detection circuitry 20 could be arranged on a different wafer / die than the pixels 51 and be vertically connected with the pixels 51.

[0222] The event detection circuitry 20 is configured to detect the intensity changes by comparing a current intensity of the received light to an initial intensity of the received light that was received at a reset of the event detection circuitry 20. For example, as described above with respect to Fig. 7 or 11, the initial intensity is represented by the charge stored in the capacitor 101, 431 of the subtraction section 83 of the event detecting section, when the reset switch 104, 434 is closed. The current intensity is represented by the charge currently accumulating in said capacitor 101, 431, when the reset switch 104, 434 is opened. Thus, a reset of the event detection circuitry 20 is for example possible for each pixel 51 by closing the switch 104, 434 of the subtraction section 83 of the respective event detection section 52.

[0223] The pixels 51 comprise the first subset SI. In Fig. 15 the pixels 51 of the first subset SI are indicated by vertical hatching. These pixels 51 are connected to the event detection circuitry 20, e.g. as described above, and allow event detection on the light received by them.

[0224] The pixels 51 further comprise a second subset S2 that is formed in Fig. 15 by the unhatched pixels 51. The pixels 51 of the second subset S2 are used to generate full intensity information, i.e. to generate data representing the (absolute) intensity of the light received by the pixels 51. Here, it is understood that appropriate color filters may be provided to each of the pixels 51 of the second subset S2 to allow generation of a color image. Color filters may also be provided to the pixels 51 of the first subset S 1.

[0225] The pixels 51 of the first subset S 1 and the pixels 51 of the second subset S2 are arranged such that they observe the same scene. This means that pixels 51 of the first subset S 1 and pixels 51 of the second subset S2 receive light from the same location of the scene. A pixel 51 of the first subset S 1 and a pixel 51 of the second subset S2 are associated with each other when they receive light from the same location.

[0226] The sensor device 10 comprises pixel signal generating circuitry 30 that is configured to generate a pixel signal indicating intensity values of the received light for each pixel 51 of the second subset S2 of the pixels 51. For example, the pixel signal generating circuitry 30 may be constituted by the pixel signal generating sections 53 as described above that can produce pixel signals indicating the received light intensity in an asynchronous (event triggered) or conventional, synchronous manner. In particular, the pixels 51 of the second subset S2 may operate according to the known principles of an active pixel sensor, APS. As is commonly known, the pixel signal generating circuitry 30 is configured to generate the pixel signals based on the amount of light received at the respective pixels 51 of the second subset S2 during an exposure period of said pixel 51.

[0227] Just as the event detection circuitry 20 also the pixel signal generating circuitry 30 may be formed either in a distributed manner, where e.g. one pixel signal generating section 53 is arranged next to or close to73957

[0228] 23

[0229] one pixel 51 of the second subset S2, or in a dedicated region of the sensor chip. The pixel signal generating circuitry 30 may also be considered to include all control, reset, and / or selection lines and the like that are necessary to operate the sensor device 10 as an APS that is capable to generate a consecutive stream of pixel signals indicating the intensity of the received light over a given time period.

[0230] The sensor device comprises further a control unit 40 that is configured to control the event detection circuitry 20 and the pixel signal generating circuitry 30. The control unit 40 may be any arrangement of circuitry that is capable to carry out the functions necessary to control the event detection circuitry 20 and the pixel signal generating circuitry 30 that will be described below. For example, the control unit 40 may be constituted by a processor. The control unit 40 may be part of the pixel section of the sensor device 10 and may be placed on the same die(s) as the other components of the sensor device 10. But the control unit 10 may also be arranged separately, e.g. on a separate die. The functions of the control unit 10 may be fully implemented in hardware, in software or may be implemented as a mixture of hardware and software functions.

[0231] The control unit 40 is configured to change the pixel signal generation process for at least one pixel 51 of the second subset S2 when event data have been detected for the pixel 51 of the first subset S 1 that is associated with said at least one pixel 51 of the second subset S2. This means that if an event is detected by a pixel 51 of the first subset SI, a pixel 51 of the second subset S2 will be controlled such that the parameters used in the pixel signal generation process are adapted. Thus, the pixel signal generation is synchronized with event detection in the sense that information about the detected event (i.e. information about the location and time of a change of intensity) influences the manner in which the pixel signal is generated. This allows to select control parameters for optimal pixel signal generation (or image capturing) pixel-wise (or row-wise) depending on varying illumination conditions. In particular, pixel signal generation is adapted such as to maximize a signal-to-noise ratio and to avoid pixel saturation. Thereby the quality of the image resulting from the pixel signals can be improved.

[0232] This is exemplified in the lower part of Fig. 15. Here, image frames are shown that are captured with exposure times that start at tl, t2, and t3 and that end shortly before tl, t2, and t3. The shown second frame is therefore captured during an exposure time that starts at tl and that ends shortly before t2. The first frame is captured before tl and the third frame is captured during an exposure time that starts at t2 and that ends shortly before t3.

[0233] The intensity measured at an event detection pixel (i.e. a pixel 51 of the first subset SI), which is associated to a pixel 51 observing a central part of the captured scene is shown at the bottom of Fig. 15. As is schematically illustrated in the shown frames, the intensity at said pixel 51 and the event detection pixel increases and decreases around t2.

[0234] At tl, i.e. at the beginning of the exposure period, a first light intensity is measured. After an increase of the intensity by the event threshold Al a first positive polarity event is detected (white star). After a dead time caused by readout of the event and reset of the event detection pixel, a further increase of the intensity leads to a second positive polarity event (white star). Again, after a dead time of the event73957

[0235] 24

[0236] detection pixel the pixel will be reset and the intensity present at this time will be used as initial intensity for the next event detection. Further, in the example of Fig. 15, another exposure period starts at time t2. After the intensity drops by Al a further, negative polarity event (black star) is detected. Readout of the event and reset allows detection of another negative polarity event (black star).

[0237] After each detection of a positive polarity event pixel signal generation at the respective pixel 51 (or at all pixels 51 in the row of the respective pixel 51) is adapted to the increase in light intensity signaled by the event. This is schematically indicated in Fig. 15 by a shading of the second frame due to the increased light intensity. On the other hand, the negative polarity events lead to an adaption of pixel signal generation to the lowered intensity at the respective pixel. In Fig. 15, since two positive polarity events and two negative polarity events have been detected, pixel signal generation processing for the generation of the first frame and the third frame is identical, which is exemplified by the same shading of these frames.

[0238] The adaption to the varying illumination of the single pixels can for example be performed by adapting a conversion gain of each of the pixels 51 of the second subset S2 if for respectively associated pixels 51 of the first subset SI events are detected. In this manner a higher dynamic range can be achieved for the pixels 51, by adapting their conversion gain to the varying illumination conditions. However, instead of the conversion gain also other pixel signal generation parameters could be changed, such as e.g. binning of adjacent pixels 51 of the second subset S2.

[0239] According to another example for an adaption of the pixel signal generation process, the control unit 40 is configured to terminate the exposure period of the pixels 51 of the second subset S2 when event data have been detected for the respectively associated pixel 51 of the first subset SI. This means that whenever a change of light intensity beyond the event threshold Al is measured by pixels 51 of the first subset SI, the exposure period of a pixel 51 of the second subset S2 that observes the same position in a scene (or of all pixels 51 of its row) is terminated. Accordingly, a strongly varying intensity within one exposure period is avoided, since a new exposure period starts whenever the event threshold Al is exceeded. Differently stated, for each pixel 51 it is guaranteed that during one exposure period the maximum difference between the minimum intensity and the maximum intensity is smaller than 2 Al. This maximum is only met if e.g. the intensity drops from an initial intensity by a little less than the event threshold Al and rises then to Al above the initial intensity. For a monotonic increase or decrease of the intensity the maximum difference during one exposure period will be Al. Accordingly, pixels 51 do not have to cope with overly strong light intensity changes during one exposure period. This allows increasing their dynamic range.

[0240] In addition, if the intensity change is due to a movement of an object in the scene motion blur due to the movement can be reduced, since the exposure period is shortened.

[0241] This is exemplified in Fig. 16A which shows the same intensity development as discussed for Fig. 15. However, instead of having the first exposure period from tl to t2, the first exposure period stops at tl’, the time at which the first positive polarity event is detected. The consecutive second exposure period (that starts right after tl’) stops at tl”, when the second positive polarity event is detected. Thereafter, a73957

[0242] 25

[0243] third exposure period commences that extends across t2 and only terminates at t2’. This is followed by a fourth exposure period that terminates at t2”. A consecutive fifth exposure period continues across t3 until t3’, e.g. for the original length of exposure periods. Alternatively, the fifth exposure period could also terminate such that a sixth exposure period can start at t3, in order that pixels that do not observe changes in light intensity operate during the same exposure periods.

[0244] Alternatively, as illustrated in Fig. 16B, it is also conceivable that the exposure periods start at predetermined points in time, such as tl, t2, and t3. Then, the sensor device operates at a predetermined frame rate and the frames are separated by frame periods. In this case, the exposure periods may terminate at the first event detected between these starting times, i.e. within one frame period (solid lines in Fig.

[0245] 16B), and the next exposure period only starts at the predetermined points in time tl, t2, t3. This ensures that only one pixel signal per pixel 51 is generated for each frame period, which in turn means that each pixel value in each frame is based on only one pixel signal. This eases signal processing and image generation.

[0246] However, in both cases of Fig. 16A and 16B the exposure periods are ended such as to ensure that the variations in the received light intensity of all (or some) pixels 51 of the second subset S2 stay within a predetermined range that is correlated to the intensity threshold Al.

[0247] Although Figs. 15, 16A, and 16B refer only to a single pair of associated pixels 51 it is apparent that any number or even all of the pixels 51 of the second subset S2 may be used to adapt pixel signal generation processing in the manner described above.

[0248] Further, since the control unit 40 is able to terminate exposure periods of the pixels 51 of the second subset S2 separately, when events are detected for the associated pixels 51 of the first subset SI, it is irrelevant how the start and end times of the exposure periods are distributed in the pixels 51 of the second subset S2. In particular, in the context of a fixed frame rate and the generation of one pixel value per frame period as discussed above with respect to Fig. 16B, the exposure periods of all pixels 51 of the second subset S2 may start at the same time (global shutter), or they may start at different times. For example, a rolling shutter may be used, in which exposure periods start at the same time for pixels 51 of a single row (or column) but differently for pixels 51 of different rows (or columns). In this case, it is conceivable that if termination of the exposure period is triggered for one pixel in a row, then the exposure periods for all pixels 51 in that row are terminated. However, the exposure periods of all pixels 51 may start and end separately. Image frames are then generated by in principle known interpolation and normalization techniques.

[0249] Additionally or alternatively, the control unit 40 may be configured to reset the event detection circuitry 20 for at least one pixel 51 of the first subset of pixels 51 based on the exposure period of the associated pixel 51 of the second subset of pixels 51. This means that depending on the exposure period used to generate the pixel signals, the value of the light intensity is fixed against which event detection is performed. In this manner it is on the one hand possible to deduce the light intensity present during event detection from the light intensity measured for generating a respective pixel signal. On the other hand, it73957

[0250] 26

[0251] is possible to determine precisely the moment in time during the exposure period at which an event did occur. This helps to correct inaccuracies in the pixel signals due to sudden movements, i.e. to correct motion blur.

[0252] In the example of Fig. 15, after detection of the second positive polarity event the initial intensity for detection of the next event was set to the intensity present after the dead time of the respective event detection pixel. Instead, this initial intensity may be set to the intensity present at t2, i.e. to the intensity present at the start of the second exposure period. Just the same the initial intensity may be reset at each start of an exposure period, i.e. at times tl and t3 in Fig. 15.

[0253] Thus, for all events it is then clear during which exposure period they were generated. The two positive polarity events were generated during the exposure period starting at tl. The two negative polarity events were generated during the exposure period that started at t2. In addition, the absolute intensity measured at tl and t2 (i.e. during the previous exposure period) by the pixel 51 of the second subset S2 is known. This allows a reconstruction of the absolute intensity at the respective events. Finally, it is possible to limit the time at which the change in intensity started (or ended) to he between the start (or end) of the exposure period and the occurrence of the first (or last) event. This allows determining more precisely when a change of intensity did start to occur. In particular, in a situation in which no reset based on the exposure period is made, it is impossible to tell whether the intensity did change quickly and shortly before event detection or slowly over a long time period. This uncertainty makes the correction of resulting image deteriorations like motion blur very difficult. Limiting the time period during which an intensity change could occur (or determining time periods during which certainly no large intensity change did occur) allows therefore to address such image deteriorations more effectively.

[0254] Instead of resetting the event detection circuitry 20 for the pixels 51 of the first subset S 1 at the start of the exposure period of the associated pixel 51 of the second subset S2, i.e. at times tl, t2 and t3 in Fig.15, also other relations between the exposure time and the reset times are conceivable, as long as it is possible to understand at which time the reset did occur. For example, the reset of the event detection pixel may be made a predetermined time after the start of the exposure period of the associated pixel 51.

[0255] In the following, it will be described with respect to Figs. 17A to 18B how the sensor device 10 can be used to detect motion blur in an image constituted by the pixel signals generated by the pixels 51 of the second subset S2.

[0256] Figs. 17A and 17B illustrate schematically how an undefined reset time of an event detection pixel leads to an uncertainty about the intensity that is present at event detection and to an uncertainty regarding the start of an intensity change. Figs. 17A and 17B show the same development of intensity overtime. After a slow increase and a stable phase, the intensity starts to increase more rapidly before time t4. Times tl to t5 indicate start times of different exposure periods.

[0257] In Fig. 17A the event detection pixel is reset between times tl and t2. During the exposure period between t4 and t5 the event threshold Al is crossed, and an event is detected. However, the event only73957

[0258] 27

[0259] indicates that a change about Al has occurred, but not exactly when the change has started or became more rapid. In fact, a linear increase as indicated by the short-dashed line would lead to the same event, just as any other development of the intensity in time that does not exceed the event threshold Al (see e.g. dash-dotted line). Without ensuring that if an event is detected it will be detected temporally close to the last reset of the respective event-detection pixel, there is therefore a high uncertainty as to how the intensity did develop. The information obtainable from event detection is in this case not sufficient to correct errors due to motion blur.

[0260] In addition, without a fixed reset timing, reset can occur in principle at any time. Usually, reset is carried once an event has been detected and has been read out, i.e. after event readout there is a comparably short dead time after which the event detection pixel is ready to detect a new event starting from the then received light intensity. This means that reset can in principle occur at any time. This makes it in turn impossible to deduce the absolute intensity present at event detection from absolute intensities measured by the associated pixel 51 of the second subset S2.

[0261] This is illustrated in Fig. 17B, where reset occurs before tl. Since the intensity is lower before tl than after tl, i.e. than at the reset in the example of Fig. 17A, also the intensity at which the event threshold Al is crossed will be lower. It is therefore not possible to deduce an intensity at event detection merely from the time of event detection. This reduces the usefulness of the event data for motion blur correction further.

[0262] As indicated above, it must be ensured that reset and event detection are temporally sufficiently close to each other to obtain meaningful information about the start of a significant change of intensity. In fact, for the detection (and correction) of motion blur it is important to identify how much a sudden change in the received intensity affected the obtained measurement result. The light intensity obtained during the exposure period results in an accumulation of respective charges in the pixels 51 of the second subset S2. Accordingly, the pixels 51 of the second subset S2 temporally integrate over the received intensity. Its output corresponds therefore to the temporal average of the received intensity, i.e. the ratio of integrated intensity to the exposure period (which is the integration time). This will give correct results for situations in which the received intensity changes linearly. Further, it will be a good approximation if the intensity changes only a little. However, if a period of slow and / or linear change and a period of rapid and / or linear change with a different gradient are covered by a single exposure period, the average obtained by the respective pixel 51 will not give an accurate representation of the true intensity distribution. However, when it is possible to find the point in time at which the change of intensity began to change, this error can be corrected. This can be done via event data if it is possible to keep event detection close to the reset time.

[0263] One seeming possibility to achieve this is to reduce the event threshold Al. This will lead quicker to event detection. However, also an increase of irrelevant events comes typically with a reduced event threshold Al. This might endanger the sparseness and hence the low latency of event detection.

[0264] For example, as illustrated in Fig. 18A, for a small event threshold Al an event will already be triggered73957

[0265] 28

[0266] in the period of slow increase of intensity between tl and t4. In addition, the problem remains that since reset may occur in principle at any time, it is not possible to deduce absolute intensity values from the detected events.

[0267] Thus, using small event thresholds Al might allow a detection of motion blur. But it comes with a potential increase of the latency of event detection due to an increase of detected events that need to be read out. In addition, it provides no synchronization between pixel signal generation by the pixels 51 of the second subset S2 and event detection by the pixels 51 of the first subset SI. Therefore, no intensity information can be obtained therefrom.

[0268] These problems are solved if one resets the event detection pixels based on the exposure periods of the image signal generation pixels. Fig. 18B shows the same temporal development of the intensity as Fig.

[0269] 18A and uses the same event threshold Al. However, resets are carried out depending on the exposure periods (in Fig. 18B at the start of the exposure periods at tl to t5). In none of the exposure periods between tl and t4 an event is detected, despite the comparably small event threshold Al. This means that during these regions the pixel signals are sufficiently accurate. Shortly before t4 the intensity begins to increase more rapidly. Accordingly, two event are detected between t4 and t5. Based on the size of the event threshold Al, the intensity value measured during the exposure period from t3 to t4, and the number of events it is possible to deduce that the change in the development of the intensity occurred around t4. Hence, no correction is necessary in the example of Fig. 18B.

[0270] However, going back to the example of Fig. 15 one sees that from the number of events and the event threshold Al it is reasonable to estimate that the intensity rise (or decrease) is approximately linear and to deduce from this assumption when it started (or when it finished). This allows a correction of the intensity measured during the exposure period by post-processing e.g. by allocating the measured intensities to a time of constant intensity and a time of rising intensities. In this manner it is not only possible to detect motion blur, but also to correct it.

[0271] This correction is even simplified, if the exposure period is terminated at the first event detection, as e.g. illustrated in Figs. 16A and 16B. An exposure period starts then with a reset of the associated event detection pixel. Once an event has been detected, exposure stops. As discussed above, this means that the maximum variation of the received intensity is bound by a predetermined value, which depends on the event threshold Al. The possible deviation due to motion blur is therefore bounded, which eases the correction.

[0272] In the schematic illustration of the sensor device 10 given in Fig. 15, which is also shown in Fig. 19 a) the pixels 51 in the first subset SI differ from the pixels 51 in the second subset S2. Thus, while the event detection circuitry 20 receives signals only from the first subset SI in this example, the pixel signal generating unit 30 receives signals only from the second subset S2.

[0273] As shown in Figs. 15 and 19 a) one implementation for such a division of pixels 51 is to arrange all pixels 51 in a single pixel array. Then, a specific number of pixels 51 can be designed for event detection, such73957

[0274] 29

[0275] as by providing circuitry as described above with respect to Fig. 13, for example for one pixel of a N x M pixel group, e.g. of a 2 x 2 pixel group as shown in Figs. 15 and 19 a). These pixels 51 form then the first subset SI of pixels 51. The remaining pixels 51 form the second subset S2 of pixels 51, and can be provided with conventional circuitry for a synchronous readout.

[0276] However, at least a part of the pixels 51 may belong to the first subset of pixels 51 as well as to the second subset of pixels 51. For example, all pixels 51 of a pixel array may function as pixels 51 of the first subset SI and the second subset S2 as schematically illustrated in Fig. 19 b). The pixels 51 may then be connected to the event detection circuitry 20 and the pixel signal circuitry 30 e.g. as described above with respect to Fig. 4, i.e. each pixel 51 can be selectively chosen to transfer its photocurrent to the event detection circuitry 20 or the pixel signal generation circuitry 30, e.g. in a time multiplexed manner. This shared pixel architecture allows to have the same spatial resolution for event detection and intensity information generation, which leads to improved quality of the reconstructed intensity information.

[0277] In the above description it was assumed that pixels 51 of both pixel subsets SI, S2 were part of the same pixel array. However, as schematically illustrated in Fig. 19 c) it is also conceivable that two different pixel arrays are used that each contain only pixels 51 of one pixel subset, either on different parts of a single die or even on different sensor chips. As long as both pixel arrays observe the same scene it will nevertheless be possible to reconstruct intensity information for the pixels 51 of the second subset S2 based on the events detected by the pixels 51 of the first subset S 1.

[0278] In general, any geometrical arrangement of pixels 51 with shared or divided functionality will be sufficient as long as event data and intensity information of the same scene can be obtained. If there is a mismatch in spatial resolution or if it is not possible to map the solid angles observed by pixels 51 in different subsets in a one to one manner, this will be solvable in principle by using interpolation techniques to adapt / align the two different data sets.

[0279] Above various implementations of the basic idea to synchronize event detection and pixel signal generation have been described. The basic method underlying all these implementations is summarized below with respect to Fig. 20.

[0280] At S 101 light is received from a scene and photoelectric conversion is performed with a plurality of pixels 51 of the sensor device 10 to generate an electrical signal. At SI 02 event data are detected as intensity changes above a predetermined threshold of the light received by each of a first subset of the pixels 51 by event detection circuitry 20 of the sensor device 10. At SI 03 a pixel signal indicating intensity values of the received light for each pixel 51 of a second subset of the pixels 51 is generated by pixel signal generating circuitry 30 of the sensor device 10. Here, a pixel 51 of the first subset SI of pixels 51 and a pixel 51 of the second subset S2 of pixels 51 that receive light from the same location of the scene are associated with each other. At SI 04 the event detection circuitry 20 and the pixel signal generating circuitry 30 are controlled to change the pixel signal generation process for at least one pixel 51 of the second subset S2 of pixels 51 when event data have been detected for the pixel 51 of the first subset S 1 of pixels 51 that is associated with said at least one pixel 51 of the second subset of pixels 51.73957

[0281] 30

[0282] In this manner it is possible to synchronize pixel signal generation and event detection such as to use the obtained event data for improving the image quality of an image obtained from the pixel signals.

[0283] Fig. 21 is a block diagram illustrating an example of a schematic configuration of a smartphone 900 that may constitute or comprise a sensor device 10 as described above. The smartphone 900 is equipped with a processor 901, memory 902, storage 903, an external connection interface 904, a camera 906, a sensor 907, a microphone 908, an input device 909, a display device 910, a speaker 911, a radio communication interface 912, one or more antenna switches 915, one or more antennas 916, a bus 917, a battery 918, and an auxiliary controller 919.

[0284] The processor 901 may be a CPU or system -on-a-chip (SoC), for example, and controls functions in the application layer and other layers of the smartphone 900. The memory 902 includes RAM and ROM, and stores programs executed by the processor 901 as well as data. The storage 903 may include a storage medium such as semiconductor memory or a hard disk. The external connection interface 904 is an interface for connecting an externally attached device, such as a memory card or Universal Serial Bus (USB) device, to the smartphone 900. The processor may function as control unit 40.

[0285] The camera 906 includes an image sensor as described above. The sensor 907 may include a sensor group such as a positioning sensor, a gyro sensor, a geomagnetic sensor, and an acceleration sensor, for example. The microphone 908 converts audio input into the smartphone 900 into an audio signal. The input device 909 includes devices such as a touch sensorthat detects touches on a screen of the display device 910, a keypad, a keyboard, buttons, or switches, and receives operations or information input from a user. The display device 910 includes a screen such as a liquid crystal display (UCD) or an organic light-emitting diode (OUED) display, and displays an output image of the smartphone 900. The speaker 911 converts an audio signal output from the smartphone 900 into audio.

[0286] The radio communication interface 912 supports a cellular communication scheme such as UTE or LTE-Advanced, and executes radio communication. Typically, the radio communication interface 912 may include a BB processor 913, an RF circuit 914, and the like. The BB processor 913 may conduct processes such as encoding / decoding, modulation / demodulation, and multiplexing / demultiplexing, for example, and executes various signal processing for radio communication. Meanwhile, the RF circuit 914 may include components such as a mixer, a fdter, and an amp, and transmits or receives a radio signal via an antenna 916. The radio communication interface 912 may also be a one-chip module integrating the BB processor 913 and the RF circuit 914. The radio communication interface 912 may also include multiple BB processors 913 and multiple RF circuits 91. Note that although Fig. 21 illustrates an example of the radio communication interface 912 including multiple BB processors 913 and multiple RF circuits 914, the radio communication interface 912 may also include a single BB processor 913 or a single RF circuit 914.

[0287] Furthermore, in addition to a cellular communication scheme, the radio communication interface 912 may also support other types of radio communication schemes such as a short-range wireless communication73957

[0288] 31

[0289] scheme, a near field wireless communication scheme, or a wireless local area network (LAN) scheme. In this case, a BB processor 913 and an RF circuit 914 may be included for each radio communication scheme.

[0290] Each antenna switch 915 switches the destination of an antenna 916 among multiple circuits included in the radio communication interface 912 (for example, circuits for different radio communication schemes).

[0291] Each antenna 916 includes a single or multiple antenna elements (for example, multiple antenna elements constituting a MIMO antenna), and is used by the radio communication interface 912 to transmit and receive radio signals. The smartphone 900 may also include multiple antennas 916 as illustrated in Fig. 21. Note that although Fig. 21 illustrates an example of the smartphone 900 including multiple antennas 916, the smartphone 900 may also include a single antenna 916.

[0292] Furthermore, the smartphone 900 may also be equipped with an antenna 916 for each radio communication scheme. In this case, the antenna switch 915 may be omitted from the configuration of the smartphone 900.

[0293] The bus 917 interconnects the processor 901, the memory 902, the storage 903, the external connection interface 904, the camera 906, the sensor 907, the microphone 908, the input device 909, the display device 910, the speaker 911, the radio communication interface 912, and the auxiliary controller 919. The battery 918 supplies electric power to the respective blocks of the smartphone 900 illustrated in Fig. 21 via power supply lines partially illustrated with dashed lines in the drawing. The auxiliary controller 919 causes minimal functions of the smartphone 900 to operate while in a sleep mode, for example.

[0294] Fig. 22 is a block diagram depicting an example of schematic configuration of a vehicle control system as an example of a mobile body control system to which the technology according to an embodiment of the present disclosure can be applied.

[0295] The vehicle control system 12000 includes a plurality of electronic control units connected to each other via a communication network 12001. In the example depicted in Fig. 22, the vehicle control system 12000 includes a driving system control unit 12010, a body system control unit 12020, an outside-vehicle information detecting unit 12030, an in-vehicle information detecting unit 12040, and an integrated control unit 12050. In addition, a microcomputer 12051, a sound / image output section 12052, and a vehicle-mounted network interface (I / F) 12053 are illustrated as a functional configuration of the integrated control unit 12050.

[0296] The driving system control unit 12010 controls the operation of devices related to the driving system of the vehicle in accordance with various kinds of programs. For example, the driving system control unit 12010 functions as a control device for a driving force generating device for generating the driving force of the vehicle, such as an internal combustion engine, a driving motor, or the like, a driving force transmitting mechanism for transmitting the driving force to wheels, a steering mechanism for adjusting the steering angle of the vehicle, a braking device for generating the braking force of the vehicle, and the73957

[0297] 32

[0298] like.

[0299] The body system control unit 12020 controls the operation of various kinds of devices provided to a vehicle body in accordance with various kinds of programs. For example, the body system control unit 12020 functions as a control device for a keyless entry system, a smart key system, a power window device, or various kinds of lamps such as a headlamp, a backup lamp, a brake lamp, a turn signal, a fog lamp, or the like. In this case, radio waves transmitted from a mobile device as an alternative to a key or signals of various kinds of switches can be input to the body system control unit 12020. The body system control unit 12020 receives these input radio waves or signals, and controls a door lock device, the power window device, the lamps, or the like of the vehicle.

[0300] The outside-vehicle information detecting unit 12030 detects information about the outside of the vehicle including the vehicle control system 12000. For example, the outside-vehicle information detecting unit 12030 is connected with an imaging section 12031. The outside-vehicle information detecting unit 12030 makes the imaging section 12031 image an image of the outside of the vehicle, and receives the imaged image. On the basis of the received image, the outside-vehicle information detecting unit 12030 may perform processing of detecting an object such as a human, a vehicle, an obstacle, a sign, a character on a road surface, or the like, or processing of detecting a distance thereto.

[0301] The imaging section 12031 is an optical sensor that receives light, and which outputs an electric signal corresponding to a received light amount of the light. The imaging section 12031 can output the electric signal as an image, or can output the electric signal as information about a measured distance. In addition, the light received by the imaging section 12031 may be visible light, or may be invisible light such as infrared rays or the like.

[0302] The in-vehicle information detecting unit 12040 detects information about the inside of the vehicle. The in-vehicle information detecting unit 12040 is, for example, connected with a driver state detecting section 12041 that detects the state of a driver. The driver state detecting section 12041, for example, includes a camera that images the driver. On the basis of detection information input from the driver state detecting section 12041, the in-vehicle information detecting unit 12040 may calculate a degree of fatigue of the driver or a degree of concentration of the driver, or may determine whether the driver is dozing.

[0303] The microcomputer 12051 can calculate a control target value for the driving force generating device, the steering mechanism, or the braking device on the basis of the information about the inside or outside of the vehicle which information is obtained by the outside-vehicle information detecting unit 12030 or the in-vehicle information detecting unit 12040, and output a control command to the driving system control unit 12010. For example, the microcomputer 12051 can perform cooperative control intended to implement functions of an advanced driver assistance system (ADAS) which functions include collision avoidance or shock mitigation for the vehicle, following driving based on a following distance, vehicle speed maintaining driving, a warning of collision of the vehicle, a warning of deviation of the vehicle from a lane, or the like.In addition, the microcomputer 12051 can perform cooperative control intended for automatic driving, which makes the vehicle to travel autonomously without depending on the operation of the driver, or the like, by controlling the driving force generating device, the steering mechanism, the braking device, or the like on the basis of the information about the outside or inside of the vehicle which information is obtained by the outside-vehicle information detecting unit 12030 or the in-vehicle information detecting unit 12040.

[0304] In addition, the microcomputer 12051 can output a control command to the body system control unit 12020 on the basis of the information about the outside of the vehicle which information is obtained by the outside-vehicle information detecting unit 12030. For example, the microcomputer 12051 can perform cooperative control intended to prevent a glare by controlling the headlamp so as to change from a high beam to a low beam, for example, in accordance with the position of a preceding vehicle or an oncoming vehicle detected by the outside-vehicle information detecting unit 12030.

[0305] The sound / image output section 12052 transmits an output signal of at least one of a sound and an image to an output device capable of visually or auditorily notifying information to an occupant of the vehicle or the outside of the vehicle. In the example ofFig. 22, an audio speaker 12061, a display section 12062, and an instrument panel 12063 are illustrated as the output device. The display section 12062 may, for example, include at least one of an on-board display and a head-up display.

[0306] Fig. 23 is a diagram depicting an example of the installation position of the imaging section 12031.

[0307] In Fig. 23, the imaging section 12031 includes imaging sections 12101, 12102, 12103, 12104, and 12105.

[0308] The imaging sections 12101, 12102, 12103, 12104, and 12105 are, for example, disposed at positions on a front nose, sideview mirrors, a rear bumper, and a back door of the vehicle 12100 as well as a position on an upper portion of a windshield within the interior of the vehicle. The imaging section 12101 provided to the front nose and the imaging section 12105 provided to the upper portion of the windshield within the interior of the vehicle obtain mainly an image of the front of the vehicle 12100. The imaging sections 12102 and 12103 provided to the sideview mirrors obtain mainly an image of the sides of the vehicle 12100. The imaging section 12104 provided to the rear bumper or the back door obtains mainly an image of the rear of the vehicle 12100. The imaging section 12105 provided to the upper portion of the windshield within the interior of the vehicle is used mainly to detect a preceding vehicle, a pedestrian, an obstacle, a signal, a traffic sign, a lane, or the like.

[0309] Incidentally, Fig. 23 depicts an example of photographing ranges of the imaging sections 12101 to 12104. An imaging range 12111 represents the imaging range of the imaging section 12101 provided to the front nose. Imaging ranges 12112 and 12113 respectively represent the imaging ranges of the imaging sections 12102 and 12103 provided to the sideview mirrors. An imaging range 12114 represents the imaging range of the imaging section 12104 provided to the rear bumper or the back door. A bird’s-eye image of the vehicle 12100 as viewed from above is obtained by superimposing image data imaged by the imaging sections 12101 to 12104, for example.34

[0310] At least one of the imaging sections 12101 to 12104 may have a function of obtaining distance information. For example, at least one of the imaging sections 12101 to 12104 may be a stereo camera constituted of a plurality of imaging elements, or may be an imaging element having pixels for phase difference detection.

[0311] For example, the microcomputer 12051 can determine a distance to each three-dimensional object within the imaging ranges 12111 to 12114 and a temporal change in the distance (relative speed with respect to the vehicle 12100) on the basis of the distance information obtained from the imaging sections 12101 to 12104, and thereby extract, as a preceding vehicle, a nearest three-dimensional object in particular that is present on a traveling path of the vehicle 12100 and which travels in substantially the same direction as the vehicle 12100 at a predetermined speed (for example, equal to or more than 0 km / hour). Further, the microcomputer 12051 can set a following distance to be maintained in front of a preceding vehicle in advance, and perform automatic brake control (including following stop control), automatic acceleration control (including following start control), or the like. It is thus possible to perform cooperative control intended for automatic driving that makes the vehicle travel autonomously without depending on the operation of the driver or the like.

[0312] For example, the microcomputer 12051 can classify three-dimensional object data on three-dimensional objects into three-dimensional object data of a two-wheeled vehicle, a standard-sized vehicle, a largesized vehicle, a pedestrian, a utility pole, and other three-dimensional objects on the basis of the distance information obtained from the imaging sections 12101 to 12104, extract the classified three-dimensional object data, and use the extracted three-dimensional object data for automatic avoidance of an obstacle. For example, the microcomputer 12051 identifies obstacles around the vehicle 12100 as obstacles that the driver of the vehicle 12100 can recognize visually and obstacles that are difficult for the driver of the vehicle 12100 to recognize visually. Then, the microcomputer 12051 determines a collision risk indicating a risk of collision with each obstacle. In a situation in which the collision risk is equal to or higher than a set value and there is thus a possibility of collision, the microcomputer 12051 outputs a warning to the driver via the audio speaker 12061 or the display section 12062, and performs forced deceleration or avoidance steering via the driving system control unit 12010. The microcomputer 12051 can thereby assist in driving to avoid collision.

[0313] At least one of the imaging sections 12101 to 12104 may be an infrared camera that detects infrared rays. The microcomputer 12051 can, for example, recognize a pedestrian by determining whether or not there is a pedestrian in imaged images of the imaging sections 12101 to 12104. Such recognition of a pedestrian is, for example, performed by a procedure of extracting characteristic points in the imaged images of the imaging sections 12101 to 12104 as infrared cameras and a procedure of determining whether or not it is the pedestrian by performing pattern matching processing on a series of characteristic points representing the contour of the object. When the microcomputer 12051 determines that there is a pedestrian in the imaged images of the imaging sections 12101 to 12104, and thus recognizes the pedestrian, the sound / image output section 12052 controls the display section 12062 so that a square contour line for emphasis is displayed so as to be superimposed on the recognized pedestrian. The35

[0314] sound / image output section 12052 may also control the display section 12062 so that an icon or the like representing the pedestrian is displayed at a desired position.

[0315] An example of the vehicle control system to which the technology according to the present disclosure is applicable has been described above. The technology according to the present disclosure is applicable to the imaging section 12031 among the above-mentioned configurations. Specifically, the sensor device 10 is applicable to the imaging section 12031. The imaging section 12031 to which the technology according to the present disclosure has been applied flexibly acquires event data and performs data processing on the event data, thereby being capable of providing appropriate driving assistance.

[0316] Further possible implementations of the sensor device 10 are mobile devices 3000 such as cell phones, tablets, smart watches and the like as shown in Fig. 24A or head-mounted displays 4000 as shown in Fig.

[0317] 24B. Further, the sensor device 10 is useable in augmented and / or virtual reality applications / cameras or in surveillance systems like 360° cameras.

[0318] Note that, the embodiments of the present technology are not limited to the above-mentioned embodiment, and various modifications can be made without departing from the gist of the present technology.

[0319] Further, the effects described herein are only exemplary and not limited, and other effects may be provided.

[0320] Note that, the present technology can also take the following configurations.

[0321]

[0001] A sensor device (10) comprising :

[0322] a plurality of pixels (51) each configured to receive light from a scene and to perform photoelectric conversion to generate an electrical signal;

[0323] event detection circuitry (20) that is configured to detect as event data intensity changes above a predetermined threshold of the light received by each of a first subset of the pixels (51);

[0324] pixel signal generating circuitry (30) that is configured to generate a pixel signal indicating intensity values of the received light for each pixel (51) of a second subset of the pixels (51); and

[0325] a control unit (40) that is configured to control the event detection circuitry (20) and the pixel signal generating circuitry (30); wherein

[0326] a pixel (51) of the first subset of pixels (51) and a pixel (51) of the second subset of pixels (51) that receive light from the same location of the scene are associated with each other; and

[0327] the control unit (40) is configured to change the pixel signal generation process for at least one pixel (51) of the second subset of pixels (51) when event data have been detected for the pixel (51) of the first subset of pixels (51) that is associated with said at least one pixel (51) of the second subset of pixels (51).

[0328] [2] The sensor device (10) according to [1], wherein

[0329] the pixel signal generating circuitry (30) is configured to generate the pixel signals based on the amount of light received at the respective pixels (51) of the second subset of pixels (51) during an73957

[0330] 36

[0331] exposure period of said pixel (51); and

[0332] the control unit (40) is configured to terminate the exposure period of said at least one pixel (51) of the second subset of pixels (51) when event data have been detected for the pixel (51) of the first subset of pixels (51) that is associated with said at least one pixel (51) of the second subset of pixels (51).

[0333] [3] The sensor device (10) according to [1] or [2], wherein

[0334] the control unit (40) is configured to adapt a conversion gain of said at least one pixel (51) of the second subset of pixels (51) when event data have been detected for the pixel / 51) of the first subset of pixels (51) that is associated with said at least one pixel (51) of the second subset of pixels (51).

[0335] [4] The sensor device (10) according to any one of [1] to [3], wherein

[0336] the exposure periods of all pixels (51) of the second subset of pixels (51) start at the same time.

[0337] [5] The sensor device (10) according to any one of [1] to [3], wherein

[0338] the exposure periods of the pixels (51) of the second subset of pixels (51) start at different times.

[0339] [6] The sensor device (10) according to any one of [1] to [5], wherein

[0340] the pixels (51) are arranged in a two-dimensional pixel array; and

[0341] at least a part of the pixels (51) belongs to both, the first subset of pixels (51) and the second subset of pixels (51).

[0342] [7] The sensor device (10) according to any one of [1] to [5], wherein

[0343] the pixels (51) are arranged in a two-dimensional pixel array; and

[0344] pixels (51) in the first subset of pixels (51) are different from pixels in the second subset of pixels (51).

[0345] [8] The sensor device (10) according to any one of [1] to [5], wherein

[0346] the pixels (51) in the first subset of pixels (51) are arranged in a first two-dimensional pixel array; and

[0347] the pixels (51) in the second subset of pixels (51) are arranged in a different, second two-dimensional pixel array.

[0348] [9] The sensor device (10) according to any one of [1] to [8], wherein

[0349] the event detection circuitry (20) is configured to detect the intensity changes by comparing a current intensity of the received light to an initial intensity of the received light that was received at a reset of the event detection circuitry (20); and

[0350] the control unit (40) is configured to reset the event detection circuitry (20) for at least one pixel (51) of the first subset of pixels (51) based on the exposure period of the associated pixel (51) of the second subset of pixels (51).73957

[0351] 37

[0352]

[0010] The sensor device (10) according to [9], wherein

[0353] the control unit (40) is configured to reset the event detection circuitry (20) for the at least one pixel (51) of the first subset of pixels (51) at the start of the exposure period of the associated pixel (51) of the second subset of pixels (51).

[0354]

[0011] The sensor device according to [9], wherein

[0355] the control unit (40) is configured to reset the event detection circuitry (20) for the at least one pixel (51) of the first subset of pixels (51) a predetermined time after the start of the exposure period of the associated pixel (51) of the second subset of pixels (51).

[0356]

[0012] A method for operating a sensor device (10), the method comprising:

[0357] receiving light from a scene and performing photoelectric conversion with a plurality of pixels (51) of the sensor device (10) to generate an electrical signal;

[0358] detecting, by event detection circuitry (20) of the sensor device (10), as event data intensity changes above a predetermined threshold of the light received by each of a first subset of the pixels (51);

[0359] generating, by pixel signal generating circuitry (30) of the sensor device (10), a pixel signal indicating intensity values of the received light for each pixel (51) of a second subset of the pixels (51), wherein a pixel (51) of the first subset of pixels (51) and a pixel (51) of the second subset of pixels (51) that receive light from the same location of the scene are associated with each other

[0360] controlling the event detection circuitry (20) and the pixel signal generating circuitry (30) to change the pixel signal generation process for at least one pixel (51) of the second subset of pixels (51) when event data have been detected for the pixel (51) of the first subset of pixels (51) that is associated with said at least one pixel (51) of the second subset of pixels (51).

Claims

1. 7395738CLAIMS1. A sensor device comprising:a plurality of pixels each configured to receive light from a scene and to perform photoelectric conversion to generate an electrical signal;event detection circuitry that is configured to detect as event data intensity changes above a predetermined threshold of the light received by each of a first subset of the pixels;pixel signal generating circuitry that is configured to generate a pixel signal indicating intensity values of the received light for each pixel of a second subset of the pixels; anda control unit that is configured to control the event detection circuitry and the pixel signal generating circuitry; whereina pixel of the first subset of pixels and a pixel of the second subset of pixels that receive light from the same location of the scene are associated with each other; andthe control unit is configured to change the pixel signal generation process for at least one pixel of the second subset of pixels when event data have been detected for the pixel of the first subset of pixels that is associated with said at least one pixel of the second subset of pixels.

2. The sensor device according to claim 1, whereinthe pixel signal generating circuitry is configured to generate the pixel signals based on the amount of light received at the respective pixels of the second subset of pixels during an exposure period of said pixel; andthe control unit is configured to terminate the exposure period of said at least one pixel of the second subset of pixels when event data have been detected for the pixel of the first subset of pixels that is associated with said at least one pixel of the second subset of pixels.

3. The sensor device according to claim 1, whereinthe control unit is configured to adapt a conversion gain of said at least one pixel of the second subset of pixels when event data have been detected for the pixel of the first subset of pixels that is associated with said at least one pixel of the second subset of pixels.

4. The sensor device according to claim 1, whereinthe exposure periods of all pixels of the second subset of pixels start at the same time.

5. The sensor device according to claim 1, whereinthe exposure periods of the pixels of the second subset of pixels start at different times.

6. The sensor device according to claim 1, whereinthe pixels are arranged in a two-dimensional pixel array; andat least a part of the pixels belongs to both, the first subset of pixels and the second subset of pixels.

7. The sensor device according to claim 1, wherein7395739the pixels are arranged in a two-dimensional pixel array; andpixels in the first subset of pixels are different from pixels in the second subset of pixels.

8. The sensor device according to claim 1, whereinthe pixels in the first subset of pixels are arranged in a first two-dimensional pixel array; and the pixels in the second subset of pixels are arranged in a different, second two-dimensional pixel array.

9. The sensor device according to claim 1, whereinthe event detection circuitry is configured to detect the intensity changes by comparing a current intensity of the received light to an initial intensity of the received light that was received at a reset of the event detection circuitry; andthe control unit is configured to reset the event detection circuitry for at least one pixel of the first subset of pixels based on the exposure period of the associated pixel of the second subset of pixels.

10. The sensor device according to claim 9, whereinthe control unit is configured to reset the event detection circuitry for the at least one pixel of the first subset of pixels at the start of the exposure period of the associated pixel of the second subset of pixels.

11. The sensor device according to claim 9, whereinthe control unit is configured to reset the event detection circuitry for the at least one pixel of the first subset of pixels a predetermined time after the start of the exposure period of the associated pixel of the second subset of pixels.

12. A method for operating a sensor device, the method comprising:receiving light from a scene and performing photoelectric conversion with a plurality of pixels of the sensor device to generate an electrical signal;detecting, by event detection circuitry of the sensor device, as event data intensity changes above a predetermined threshold of the light received by each of a first subset of the pixels;generating, by pixel signal generating circuitry of the sensor device, a pixel signal indicating intensity values of the received light for each pixel of a second subset of the pixels, wherein a pixel of the first subset of pixels and a pixel of the second subset of pixels that receive light from the same location of the scene are associated with each other; andcontrolling the event detection circuitry and the pixel signal generating circuitry to change the pixel signal generation process for at least one pixel of the second subset of pixels when event data have been detected for the pixel of the first subset of pixels that is associated with said at least one pixel of the second subset of pixels.

Images