Imaging system and imaging method

Abstract

[Problem] To suppress an event caused by flickering of a light source. [Solution] Provided is an imaging system comprising: a signal generation unit that generates a time-series event signal in which a first event signal and a second event signal are positively and negatively inverted at a baseline, the first event signal indicating, for each time-series time, the number of times that the absolute value of a positive-side amount of change in which a pixel signal generated by a pixel having a photoelectric conversion element increases exceeds a prescribed threshold value, the second event signal indicating, for each time-series time, the number of times that the absolute value of a negative-side amount of change in which the pixel signal decreases exceeds a prescribed threshold value; and a determination unit that determines whether or not the event signal is caused by flickering on the basis of a value relating to a phase in a multiplication value obtained from the event signal and a flicker detection signal which has the same frequency as a frequency to be detected.

Description

Imaging system and imaging method

[0001] This disclosure relates to an imaging system and an imaging method.

[0002] In recent years, development has been progressing on image sensors that detect changes in brightness for each pixel as events in real time (hereinafter referred to as EVS (Event-based Vision Sensor)).

[0003] Japanese Patent Publication No. 2023-093778

[0004] However, using it in an environment with a light source that has a periodic emission may cause events to occur due to the flicker of the light source.

[0005] Therefore, this disclosure provides an imaging system and an imaging method capable of suppressing events caused by light source flicker.

[0006] To solve the above problems, the present disclosure provides an imaging system comprising: a signal generation unit that generates a time-series event signal in which the absolute value of the positive change amount of the pixel signal generated by a pixel having a photoelectric conversion element exceeds a predetermined threshold in a time-series manner, and a second event signal in which the absolute value of the negative change amount of the pixel signal decreases exceeds a predetermined threshold in a time-series manner, and the two event signals are inverted in sign at the baseline; and a determination unit that determines whether the event signal is caused by flicker based on a value relating to the phase in the convolution calculation value of the event signal and a flicker detection signal having the same frequency as the flicker frequency to be detected.

[0007] The system further includes a flicker intensity calculation unit that performs a convolution operation on the aforementioned time-series event signal using a flicker detection signal composed of a cosine wave and a sine wave having the same frequency as the flicker frequency to be detected, generates the convolution result using the cosine wave as the real part of the amplitude component and the convolution value using the sine wave as the imaginary part of the amplitude component, and generates an absolute value by adding the real part and the imaginary part of the amplitude component, respectively, and the determination unit may determine that the event signal is caused by flicker if the absolute value obtained by adding the real part and the imaginary part of the amplitude component increases monotonically with increasing frame count.

[0008] The flicker frequency to be detected may also be the frequency of the light source that generates the flicker.

[0009] The determination unit may determine whether the event signal is caused by flicker if the absolute value over a predetermined number of frames exceeds a predetermined value.

[0010] The system further includes an evaluation processing unit that generates a second absolute value that suppresses random fluctuations in the aforementioned absolute value, and the determination unit may determine that the event signal is caused by flicker if the second absolute value increases monotonically in accordance with the increase in the second absolute value of the number of frames.

[0011] The evaluation processing unit may generate the moving average of the absolute values ​​as the second absolute value with the random fluctuations suppressed.

[0012] The evaluation processing unit may increase the evaluation value indicating stability when the moving average values ​​of the absolute values ​​are larger on the side where the number of frames was increased, and decrease the evaluation value when the moving average values ​​of the adjacent frames are smaller on the side where the number of frames was increased. The determination unit may determine that the event signal is caused by flicker when the evaluation value exceeds a predetermined value.

[0013] The absolute value may also be the square root of the sum of the squares of the real-part sums obtained by adding the real parts of each of the amplitude components and the squares of the imaginary-part sums obtained by adding the imaginary parts of each of the amplitude components.

[0014] The system may further include: a plurality of photoelectric conversion elements, each of which converts incident light into electrical signals; a plurality of detection units that generate event information corresponding to each of the plurality of photoelectric conversion elements, wherein the absolute value of the positive change in the voltage corresponding to the electrical signals generated by each of the plurality of photoelectric conversion elements exceeds a predetermined threshold, making it positive polarity, and the absolute value of the negative change in the voltage corresponding to the pixel signal exceeding a predetermined threshold, making it negative polarity; coordinate information based on the photoelectric conversion element that exceeds the threshold; and time information at the time the threshold is exceeded; and an event signal generation unit that generates the first event signal and the second event signal using at least one of the event information generated by the plurality of detection units.

[0015] The flicker intensity calculation unit may perform convolution integration on the event signal over a predetermined period using sine waves and cosine waves with sequentially increasing rotational phases in predetermined time units, thereby generating the real and imaginary parts of the amplitude components at the different rotational phases.

[0016] The event signal generation unit may generate the first event signal and the second event signal using event information generated by one or more detection units.

[0017] The system may further include an analog-to-digital converter that converts pixel signals corresponding to the electrical signals generated by each of the plurality of photoelectric conversion elements into digital image signals.

[0018] The system may further include a region selection unit that, upon receiving the event information, causes the analog-to-digital converter to selectively output an electrical signal generated by the photoelectric conversion element, which is identified by the coordinate information contained in the event information.

[0019] The determination unit may further include a control unit that controls the supply of event information from the detection unit used to generate the event signal to the region selection unit when the determination unit determines that the event signal is caused by flicker.

[0020] The imaging data based on the digital image signal further includes a recognition processing unit that performs recognition processing on a region identified by coordinate information included in the event information, and if the determination unit determines that the event signal is caused by flicker, it may perform the recognition processing without using the event information supplied from the detection unit used to generate the event signal.

[0021] The determination unit may, with respect to the time-series event signal, determine whether the event signal is caused by flicker based on the phase of the amplitude component, where the convolution result of the flicker detection signal, which is composed of a cosine wave and a sine wave, is the real part of the amplitude component, and the convolution result of the event signal and the sine wave is the imaginary part of the amplitude component.

[0022] The determination unit may determine that the event signal is caused by flicker if the phase of each of the amplitude components is within a predetermined range.

[0023] The determination unit may determine that the phase of each amplitude component is within a predetermined range if the magnitude of the complex vector composed of the real and imaginary parts of the amplitude component increases monotonically in accordance with the increase in the number of frames.

[0024] To solve the above problems, the present disclosure provides an imaging method comprising: a signal generation step that generates a time-series event signal in which a first event signal indicates in time series the number of times the absolute value of the positive change amount of the pixel signal generated by a pixel having a photoelectric conversion element exceeds a predetermined threshold, and a second event signal indicates in time series the number of times the absolute value of the negative change amount of the pixel signal decreases exceeds a predetermined threshold, with the positive and negative signs reversed at the baseline; and a determination step that determines whether the event signal is caused by flicker based on a value relating to the phase in the multiplication value of the event signal and a flicker detection signal having the same frequency as the frequency to be detected.

[0025] A block diagram showing an example of the system configuration of an imaging system to which the technology relating to this disclosure is applied. A diagram schematically showing a streetlamp and a person, which are the objects of imaging in the captured image. A diagram explaining multiple imaging modes in the imaging device according to this embodiment. A block diagram showing an example of the configuration of the imaging device. A block diagram showing an example of the configuration of the pixel array section. A circuit diagram showing an example of the circuit configuration of a pixel. A block diagram showing a first example of the configuration of the address event detection section. A diagram illustrating an example of the configuration related to flicker determination. A diagram showing frame images in chronological order. A diagram explaining an example of event signal generation. A diagram showing the characteristics of the midpoint value of flicker intensity. A diagram showing the characteristics of the midpoint value of flicker intensity with respect to the number of events. A diagram showing the characteristics of the midpoint value of flicker intensity. A diagram showing an example of the characteristics of low-intensity flicker and noisy pseudo-flicker in a comparative example. A diagram showing an example of smoothing processing for an increase in the frame number of the midpoint value of flicker intensity. A diagram showing the change in evaluation value. A diagram illustrating another example of the configuration related to flicker determination. A diagram illustrating yet another example of the configuration related to flicker determination. A flowchart showing an example of flicker determination processing. A flowchart showing another example of flicker determination processing. A flowchart showing an example of imaging control of the control unit 13.

[0026] The following description will focus on the main components of the imaging system and imaging method, with reference to the drawings. While the following description will primarily focus on the main components of the imaging system and imaging method, the imaging device and imaging method may contain components and functions not shown or described. The following description does not exclude any components or functions not shown or described.

[0027] Figure 1 is a block diagram showing an example of the system configuration of an imaging system to which the technology described herein is applied. As shown in Figure 1, the imaging system 1 to which the technology described herein is applied comprises an imaging lens 11, an imaging device 10, and a processing device 20. This imaging system 1 is an example of the electronic equipment described herein, and examples of such electronic equipment include camera systems mounted on industrial robots and in-vehicle camera systems.

[0028] Furthermore, the imaging device 10 has a first signal processing unit 25. The processing unit 20 is, for example, an application processor (AP), and performs processing on the electronic image received from the imaging device 10 according to its intended use (application). The processing unit 20 includes a recording unit 12, a control unit 13, a second signal processing unit 26, and a CPU 55.

[0029] In the imaging system 1 with the above configuration, the imaging lens 11 captures incident light from the subject and forms an image on the imaging surface of the imaging device 10. The imaging device 10 converts the incident light captured by the imaging lens 11 into photoelectric data using a photoelectric conversion element at the pixel level to acquire imaging data.

[0030] The first signal processing unit 25 performs predetermined signal processing, such as image recognition processing, on the captured image data and outputs the processing results to the recording unit 12. The first signal processing unit 25 also performs flicker detection processing together with the second signal processing unit 26. Details of the detection processing will be described later.

[0031] The recording unit 12 stores data supplied from the imaging device 10 via the signal line 14. The control unit 13 is configured, for example, by a microcomputer and controls the imaging operation of the imaging device 10. The CPU 55 performs various information processing based on event signals output from the imaging device 10. The CPU 55 may be installed inside the imaging device 10.

[0032] [Imaging Device] Here, we will explain examples of imaging modes of the imaging device 10 using Figures 2A and 2B. Figure 2A is a schematic diagram showing the streetlamp O2 and the person O4, which are the targets of imaging in the captured image g0. In the following explanation, for the sake of simplicity, the diagram may be divided into eight regions sa1 to sa8, but it is not limited to this. Region sa4, which has a circular region a4, is imaged, showing the lamp portion of the streetlamp O2 that generates an event caused by flicker. On the other hand, region sa6, which has a circular region a6, is imaged, showing the person O4. The person O generates an event due to movement.

[0033] Figure 2B is a diagram illustrating multiple imaging modes in the imaging device 10 according to this embodiment. The imaging device 10 according to this embodiment has multiple imaging modes. Images E0 and E2 are diagrams in which event information is represented as a two-dimensional image.

[0034] In the first mode, after acquiring the initial image g0, it is possible to read out images of row ranges L30 and L32 that include the event regions A20 and A40 in image E0. The first mode is a mode in which the flicker occurrence region A20 can be determined based on the read-out event information.

[0035] The second mode, as shown in image E2 as an example of the embodiment, can suppress event information in the flicker-causing region A20. In other words, the second mode is a mode that can read out image information from the range excluding the flicker-causing region A20. For example, in the second mode, the image of the row range L32, which includes the event region A40 that occurred in the range excluding the flicker-causing region A20, is read out. As a result, for example, the image of the row range L32 is updated to the captured image. Therefore, the update time of the captured image is shortened and power consumption can be suppressed.

[0036] [Imaging Device] Figure 3A is a block diagram showing an example of the configuration of an imaging device. As shown in Figure 3A, the imaging device 10 according to the first configuration example of the imaging device of this disclosure is an asynchronous imaging device called an EVS, and comprises a pixel array unit 21, a drive unit 22, a readout area selection unit 23, a column processing unit 24, and a first signal processing unit 25.

[0037] In the imaging device 10 with the above configuration, the pixel array section 21 has a plurality of pixels 30 arranged in a matrix in a two-dimensional arrangement. A vertical signal line VSL, which will be described later, is wired to each pixel row in this matrix-like pixel arrangement.

[0038] Each of the multiple pixels 30 generates an analog signal of voltage corresponding to an electrical signal as a pixel signal. Furthermore, each of the multiple pixels 30 detects the presence or absence of an address event based on whether the absolute value of the positive change in the pixel signal exceeds a first threshold, or whether the absolute value of the negative change in the change exceeds a second threshold. Note that address events are sometimes simply referred to as events.

[0039] In the first mode, when an event occurs in pixel 30, event information (x, y, t, p) is generated, which includes the coordinates (x, y) of pixel 30, a timestamp t indicating the time the event occurred, and the polarity p of the event. The polarity p is assigned, for example, 1 if it exceeds a first threshold, and -1 if it exceeds a second threshold. In the first mode, pixel 30 outputs the event information (x, y, t, p) to the first signal processing unit 25 and the second signal processing unit 26.

[0040] In the first mode, when an event occurs, each pixel 30 outputs event information (x, y, t, p) to the first signal processing unit 25, the second signal processing unit 26, and the readout area selection unit 23. The readout area selection unit 23 drives the pixel 30 corresponding to the coordinates (x, y) of the event information to the drive unit 22. As a result, the drive unit 22 drives the pixel 30 at coordinates (x, y) and outputs the pixel signal to the column processing unit 24. The drive unit 22 and the column processing unit 24 are circuit units for acquiring grayscale information. Therefore, in the first mode, which acquires only event information, it is possible to have a configuration without the drive unit 22 and the column processing unit 24. In the first mode, the pixel area caused by flicker is determined and stored as the coordinates (x, y) of the pixel 30.

[0041] In the second mode, flicker countermeasures are performed on the pixel 30 or the pixel signal of the pixel 30 that was detected as flicker in the first mode. This makes it possible to suppress flicker caused by the light source. As described above, for example, it is possible to stop the transmission or reading of event information (x, y, t, p) from the coordinates (x, y) of the pixel region caused by flicker. Details of the flicker countermeasures will be described later.

[0042] The column processing unit 24 consists of, for example, an analog-to-digital converter. The column processing unit 24 performs a process to convert the analog pixel signal output from the pixels 30 of each pixel row in the pixel array unit 21 into a digital signal. The column processing unit 24 then supplies the digital signal after analog-to-digital conversion to the first signal processing unit 25.

[0043] In the second mode, the first signal processing unit 25 performs predetermined signal processing, such as CDS (Correlated Double Sampling) processing and image recognition processing, on the digital signal supplied from the column processing unit 24. The first signal processing unit 25 can also supply data indicating the processing result and event information (x, y, t, p) to the recording unit 12 (see Figure 1) via the signal line 14.

[0044] [Example of Pixel Array Configuration] Figure 3B is a block diagram showing an example of the configuration of the pixel array unit 21. As shown in Figure 3B, in the pixel array unit 21, which consists of a plurality of pixels 30 arranged in a matrix in a two-dimensional arrangement, each of the plurality of pixels 30 has a configuration that includes a light receiving unit 31, an image signal generation unit 32, and an address event detection unit 33.

[0045] In the pixel 30 with the above configuration, the light receiving unit 31 converts incident light into electrical signals. The light receiving unit 31 then supplies the generated electrical signals to either the image signal generation unit 32 or the address event detection unit 33, in accordance with the control of the drive unit 22 (see Figure 3A).

[0046] The image signal generation unit 32 generates an image signal SIG as a voltage signal corresponding to the electrical signal supplied from the light receiving unit 31, and supplies this generated image signal SIG to the column processing unit 24 (see Figure 3A) via the vertical signal line VSL.

[0047] The address event detection unit 33 detects the presence or absence of an address event based on whether the amount of change in the pixel signal from each of the light receiving units 31 exceeds a predetermined threshold. As described above, when an event occurs, event information (x, y, t, p) is generated and stored.

[0048] When an address event occurs, the address event detection unit 33 supplies a request to the read area selection unit 23 (see Figure 3A) to send an address event detection signal. Upon receiving a response to the request from the read area selection unit 23, the address event detection unit 33 supplies it to the drive unit 22, the first signal processing unit 25, and the second signal processing unit 26.

[0049] [Example of Pixel Circuit Configuration] Figure 4 is a circuit diagram showing an example of the circuit configuration of a pixel 30. As described above, each of the multiple pixels 30 has a configuration that includes a light receiving unit 31, an image signal generation unit 32, and an address event detection unit 33.

[0050] In the pixel 30 with the above configuration, the light-receiving unit 31 has a configuration comprising a light-receiving element (photoelectric conversion element) 311, a transfer transistor 312, and an OFG (Over Flow Gate) transistor 313. For example, N-type MOS (Metal Oxide Semiconductor) transistors are used as the transfer transistor 312 and the OFG transistor 313. The transfer transistor 312 and the OFG transistor 313 are connected in series with each other.

[0051] The light-receiving element 311 is connected between the common connection node N1 between the transfer transistor 312 and the OFG transistor 313 and ground, and converts incident light into electricity to generate an amount of charge corresponding to the amount of incident light.

[0052] A transfer signal TRG is supplied to the gate electrode of the transfer transistor 312 from the drive unit 22 shown in Figure 3A. In response to the transfer signal TRG, the transfer transistor 312 supplies the charge converted by photoelectricity in the photodetector 311 to the image signal generation unit 32.

[0053] The gate electrode of the OFG transistor 313 is supplied with a control signal OFG from the drive unit 22. In response to the control signal OFG, the OFG transistor 313 supplies an electrical signal generated by the photodetector 311 to the address event detection unit 33. The electrical signal supplied to the address event detection unit 33 is an electrical signal consisting of electric charge.

[0054] The image signal generation unit 32 has a configuration that includes a reset transistor 321, an amplification transistor 322, a selection transistor 323, and a floating diffusion layer 324. For example, N-type MOS transistors are used as the reset transistor 321, the amplification transistor 322, and the selection transistor 323.

[0055] The image signal generation unit 32 is supplied with charge converted photoelectrically by the photodetector 311 via the transfer transistor 312 from the photodetector 31. The charge supplied from the photodetector 31 is stored in the floating diffusion layer 324. The floating diffusion layer 324 generates a voltage signal with a voltage value corresponding to the amount of charge stored. In other words, the floating diffusion layer 324 converts charge into voltage.

[0056] The reset transistor 321 is connected between the power supply line with power supply voltage VDD and the floating diffusion layer 324. The gate electrode of the reset transistor 321 is supplied with a reset signal RST from the drive unit 22. In response to the reset signal RST, the reset transistor 321 initializes (resets) the charge amount of the floating diffusion layer 324.

[0057] The amplification transistor 322 is connected in series with the selection transistor 323 between the power supply line of the power supply voltage VDD and the vertical signal line VSL. The amplification transistor 322 amplifies the voltage signal that has been converted from charge to voltage in the floating diffusion layer 324.

[0058] A selection signal SEL is supplied to the gate electrode of the selection transistor 323 from the drive unit 22. In response to the selection signal SEL, the selection transistor 323 outputs the voltage signal amplified by the amplification transistor 322 as an image signal SIG to the column processing unit 24 (see Figure 3A) via the vertical signal line VSL.

[0059] (First Mode) In the first mode, when the control unit 13 shown in Figure 1 instructs the drive unit 22 to start detecting an address event, it drives the OFG transistor 313 of the light receiving unit 31 by supplying a control signal OFG to the OFG transistor 313, thereby supplying an electrical signal to the address event detection unit 33.

[0060] When an address event is detected in a pixel 30, the drive unit 22 turns off the OFG transistor 313 of that pixel 30, stopping the supply of electrical signals to the address event detection unit 33. Next, the drive unit 22 drives the transfer transistor 312 by supplying a transfer signal TRG to it, in accordance with the control of the read area selection unit 23, thereby transferring the charge photoelectrically converted by the light-receiving element 311 to the floating diffusion layer 324.

[0061] (Second Mode) In the second mode, for example, when the control unit 13 shown in Figure 1 instructs the drive unit 22 according to this embodiment to start detecting an address event, it turns off the OFG transistor 313 of the pixel 30 in the flicker region determined in the first mode and stops supplying an electrical signal to the address event detection unit 33.

[0062] On the other hand, when the drive unit 22 detects an address event in a pixel 30 outside the flicker region, the drive unit 22 turns off the OFG transistor 313 of that pixel 30 and stops supplying an electrical signal to the address event detection unit 33. Next, the drive unit 22 drives the transfer transistor 312 by supplying a transfer signal TRG to the transfer transistor 312 in accordance with the control of the read region selection unit 23, thereby transferring the charge photoelectrically converted by the photodetector 311 to the floating diffusion layer 324.

[0063] In this way, the imaging device 10 having a pixel array section 21 in which the pixels 30 with the above configuration are arranged in two dimensions outputs only the pixel signals in the pixel row where the pixel 30 for which an address event has been detected is located to the column processing section 24. This reduces the power consumption of the imaging device 10 and the amount of image processing required compared to the case where the pixel signals of all pixels are output regardless of whether an address event has occurred. In particular, in the second mode, the address events of pixels 30 in the flicker region are not output, so the power consumption of the imaging device 10 and the amount of image processing required can be further reduced.

[0064] The pixel configuration 30 illustrated here is merely an example and is not limited to this configuration. For example, a pixel configuration without an image signal generation unit 32 is also possible. In this pixel configuration, the OFG transistor 313 can be omitted in the light receiving unit 31, and the function of the OFG transistor 313 can be assigned to the transfer transistor 312.

[0065] [Example of Address Event Detection Unit Configuration] Figure 5 is a block diagram showing a first example configuration of the address event detection unit 33. As shown in Figure 5, the address event detection unit 33 according to this example configuration has a current-voltage conversion unit 331, a buffer 332, a subtractor 333, a quantizer 334, and a transfer unit 335.

[0066] The current-voltage conversion unit 331 converts the electrical signal from the light-receiving unit 31 of the pixel 30 into a logarithmic pixel signal. The current-voltage conversion unit 331 supplies the converted pixel signal to the buffer 332. The buffer 332 buffers the voltage signal supplied from the current-voltage conversion unit 331 and supplies it to the subtractor 333.

[0067] The subtractor 333 is supplied with a row drive signal from the drive unit 22. The subtractor 333 reduces the level of the pixel signal supplied from the buffer 332 according to the row drive signal. The subtractor 333 then supplies the reduced-level pixel signal to the quantizer 334. The quantizer 334 quantizes the pixel signal supplied from the subtractor 333 into a digital signal and outputs it to the transfer unit 335 as an address event detection signal.

[0068] The transfer unit 335 reads the detection signal of the address event supplied from the quantizer 334, generates event information (x, y, t, p), and transfers it to the region selection unit 23, etc. As described above, the transfer unit 335 sets (x, y) to the coordinates of the pixel 30 where the address event occurred, and t to the time when the address event occurred. The transfer unit 335 also sets p to 1 when the absolute value of the positive change in the pixel signal generated by the pixel 30 exceeds a predetermined first threshold. Similarly, the transfer unit 335 sets p to -1 when the absolute value of the negative change in the pixel signal generated by the pixel 30 exceeds a predetermined second threshold.

[0069] Furthermore, when the transfer unit 335 receives a response to a request from the read area selection unit 23, it supplies an address event detection signal to the first signal processing unit 25 in the first mode. On the other hand, in the second mode, it supplies it to the drive unit 22 and the first signal processing unit 25.

[0070] [Example of Flicker Detection Unit Configuration] Figure 6 is a diagram illustrating an example of the configuration of the first signal processing unit 25 and the second signal processing unit 26 related to flicker detection. The processing units related to flicker detection of the first signal processing unit 25 and the second signal processing unit 26 constitute the flicker detection unit 27. In Figure 6, for the sake of simplicity, only one of the multiple pixels 30 is shown.

[0071] The first signal processing unit 25 includes a storage unit 40, an event count unit 42, an event difference calculation unit 44, a flicker intensity calculation unit 46, and a flicker stability calculation unit 48. The second signal processing unit 26 includes a flicker determination unit 50 and a subsequent processing unit 60.

[0072] Referring to Figure 2A, an example of event generation due to light source flicker will be explained using Figures 7A and 7B. Figure 7A is a diagram showing frame images 0 to i in chronological order. Frame images 0 to i are images in which event information (x, y, t, p) is extracted at the same sampling interval Δt and arranged in coordinate order. Frame numbers n=0 to i indicate that time has passed as the frame number increases. Note that the sampling interval Δt in this implementation corresponds to a predetermined time unit.

[0073] In other words, the elapsed time t from the time origin of the image corresponds to the sampling interval Δt × n. To put it another way, the frame number n = Δt × k, where k is an integer k ≥ 0. That is, when k is 1, n is 1 and corresponds to Δt; when k is 2, n is 2 and corresponds to Δt × 2; when k is 3, n is 3 and corresponds to Δt × 3. Thus, the frame number n corresponds to discrete time with a sampling interval Δt × k and k ≥ 0.

[0074] Figure 7A shows a 4x8 pixel region ga4 corresponding to circular region a4 and a 4x8 pixel region ga6 corresponding to circular region a6. In this embodiment, a positive event is defined as a case where the absolute value of the change in the positive polarity p-side, where the pixel signal increases, exceeds a predetermined threshold. A negative event is defined as a case where the absolute value of the change in the negative polarity n-side, where the pixel signal decreases, exceeds a predetermined threshold. Furthermore, positive events may be referred to as on-events and negative events as off-events. Additionally, positive events may be referred to as positive polarity events and negative events as negative polarity events.

[0075] As shown in pixel region ga4, flicker events tend to repeat periodically in the same pixel 30, with positive and negative events alternating. On the other hand, as shown in pixel region ga6, object movement events, both positive and negative, do not exhibit periodicity and occur in accordance with the movement of the object.

[0076] Figure 7B illustrates an example of event signal x(n) generation. Figure 7B(a) shows the number of events in each region in chronological order. The vertical axis represents the number of events, and the horizontal axis represents the file number n. The first event signal xp(n) shows the value obtained by adding the number of events with positive polarity p in region sa6 for each file in chronological order. Similarly, the first event signal xn(n) shows the value obtained by adding the number of events with negative polarity n in region sa6 for each file in chronological order. For example, in an environment using a flicker light source that blinks at a frequency of 100 Hz, the light emission intensity detected by the address event detection unit 33 will similarly increase and decrease at a frequency of 100 Hz.

[0077] Figure 7B(b) shows the p / n event count difference, which is obtained by subtracting the number of negative events on the negative polarity n side from the number of positive events on the positive polarity p side. The vertical axis represents the number of events, and the horizontal axis represents the file number n. Since the period of the first event signal xp(n) and the period of the second event signal xn(n) are the same, the period of the event signal x(n), which is the p / n event count difference, is the same as that of the first event signal xp(n) and the second event signal xn(n). In this embodiment, the time origin of the first event signal xp(n) and the second event signal xn(n) is assumed to be the same point in time.

[0078] The memory unit 40 stores event information (x, y, t, p) having a timestamp t corresponding to the period of the first mode. The period of the first mode can be set to a desired period.

[0079] The event count unit 42 adds the number of positive events and the number of negative events for each set region sa1 to sa8, based on the timestamp t. In this case, the event count unit 42 adds the count for each sampling interval Δt. As described above, the number of repetitions of Δt corresponds to the number of frames n. The set regions sa1 to sa8 for each pixel can be set to any range.

[0080] The event number difference calculation unit 44 calculates the event signal x(n) by adding the positive and negative events, with one of them being negative. As shown in Figure 7B(b), the event signal x(n) is an AC waveform in which positive and negative events alternately take positive and negative values.

[0081] Thus, the event number difference calculation unit 44 generates a time-series event signal x(n) in which the first event signal xp(n) indicates the number of times in time that the absolute value of the positive change in the pixel signal generated by the pixel 30 having the photoelectric conversion element 311 exceeds a predetermined threshold, and the second event signal xn(n) indicates the number of times in time that the absolute value of the negative change in the pixel signal exceeds a predetermined threshold, and the two event signals x(n) are reversed in sign at the baseline. In Figure 7(b), for the sake of simplicity, the baseline is set to event number = 0, but it is not limited to this. For example, the baseline may be event number = constant value Offset.

[0082] The flicker intensity calculation unit 46 generates the real and imaginary parts of the amplitude component at angular velocity ω for a time-series event signal x(n), and adds the real and imaginary parts of the amplitude component together to obtain the real part Re. n , and the imaginary part Im n It generates an absolute value. More specifically, the flicker intensity calculation unit 46 calculates the intermediate value Amp of the flicker intensity of the time-series event signal x(n). n This is calculated according to equations (1) and (3). In this embodiment, F(n) is the vector V1 n = Ren +iIm n is sometimes referred to as a value related to the phase. Also, the vector V2 n is denoted as V2 n = V1 n −V1 n－１ = Rn + iIn is sometimes referred to as the amplitude component. The vector V1 n and the vector V2 n have a phase (see FIGS. 9 and 10 described later).

[0083] For example, the flicker intensity calculation unit 46 performs a convolution operation of a sine wave and a cosine wave of an angular velocity ω on an event signal x(n) that changes with respect to the frame number n. In this embodiment, the sine wave and the cosine wave of the angular velocity ω may be referred to as flicker detection signals.

[0084] As described above, the elapsed time t from the time origin of imaging corresponds to the sampling interval Δt × n. In other words, the frame number n = Δt × k, where k is an integer such that k ≥ 0. That is, when k is 1, n becomes 1, and n is a number equivalent to the number of repetitions of the sampling interval Δt.

[0085] Here, the frame number n is an integer such that 0 ≤ n ≤ N - 1. In this case, n is increased monotonically. As described above, since the frame number n = Δt × k and k is an integer such that k ≥ 0, Equation (1) becomes a discrete convolution integral for each sampling interval Δt. N is a constant and is the number of frames when the convolution operation is completed (when the flicker stability is extracted). Note that the sampling interval Δt according to this embodiment corresponds to a predetermined time unit.

[0086]

[0087] Let the flicker detection target frequency be f [Hz] and the sampling frequency of the EVS be fevs [Hz]. At this time, the sampling frequency fevs = 1 / Δt [Hz]. In other words, in this embodiment, the sampling interval Δt is set as Δt = 1 / fevs [t]. As a result, the unit of the flicker detection angular velocity ω in Equation (2) becomes [rad / sec].

[0088] When the flicker detection target frequency f is 100 [Hz] and the EVS sampling frequency fevs is 1000 [Hz] (frame interval Δt: 0.001 [sec]), the flicker detection angular velocity ω is ω = (2π × 100 [Hz]) / 1000 [Hz] = 2π / 10 [rad / Δt], and 10 frames constitute one period (= 2π).

[0089] The flicker intensity calculation unit 46 calculates the absolute values ​​of the real part Ren and the imaginary part Imn according to equation (3) after performing the calculation in equation (1). The flicker intensity calculation unit 46 continues to increase the Amp until the frame number n reaches N-1. n The value is calculated each time.

[0090] In the calculation of the intermediate flicker intensity Amp0, when n=0, it is the square root of the sum of the squares of R0, which is the value obtained by dividing the event signal x(0) multiplied by cos(ω×0) by N, and I0, which is the value obtained by dividing the event signal x(0) multiplied by sin(ω×0) by N. In the calculation of the intermediate flicker intensity Amp1, when n=1, the flicker intensity calculation unit 46 calculates R1 (R0 + x(1)*cos(ω×1)÷N) by adding the value obtained by dividing the product of x(1) and cos(ω×1) by N to R0, and I1 (I0 + x(1)*sin(ω×1)÷N) by adding the value obtained by dividing the product of x(1) and sin(ω×1) by N to I0. Then, the square root of the sum of the squares of R1 and I1 is calculated as the intermediate flicker intensity Amp1.

[0091] Similarly, in the calculation of the intermediate flicker intensity Amp2, when n=2, the flicker intensity calculation unit 46 calculates R2 (R1 + x(2)*cos(ω×2)÷N) by adding the value obtained by dividing the product of x(2) and cos(ω×2) by N to R1, and I2 (I1 + x(2)*sin(ω×2)÷N) by adding the value obtained by dividing the product of x(2) and sin(ω×2) by N to I1. Then, the square root of the sum of the squares of R2 and I2 is calculated as the intermediate flicker intensity Amp2.

[0092] Thus, the flicker intensity calculation unit 46 is R n-1 and I n-1 Vector V1 consisting of n-1The vector V2 is obtained by multiplying the time-series event signal x(n) and the flicker detection signal. n By taking the sum of, the real part Re n , and the imaginary part Im n Vector V1 consisting of n This generates (see equation (1)). Then, the real part Re n , and the imaginary part Im n By taking the square root of the sum of the squares of each, we obtain vector V1 n This generates the absolute value of the magnitude (see equation (3)).

[0093] Figure 8 shows the median flicker intensity Amp n This diagram shows the characteristics of [the frame number]. The horizontal axis represents the frame number n, and the vertical axis represents the median flicker intensity value Amp. n This shows an example in an environment using a flicker light source that blinks at 100 Hz, where the flicker detection target frequency f is 100 [Hz] and the EVS sampling frequency fevs is 1000 [Hz] (frame interval Δt: 0.001 [sec]).

[0094] Line L10 is the median flicker intensity Amp n This shows that line L10 increases monotonically with increasing frame number n. That is, as the number of convolution operations n shown in equations (1) and (3) increases, the intermediate flicker intensity Amp n It increases monotonically.

[0095] On the other hand, line L12 is the midpoint value of the flicker intensity Amp for the noisy pseudo-flicker in Figure 8(b). n This shows the median flicker intensity Amp in noisy pseudo-flicker. n This value increases or decreases randomly as the frame number n increases.

[0096] Line L14 is the midpoint value of the flicker intensity Amp when the flicker detection target frequency f is not set to the same 100 [Hz] as the flicker light source. n This shows that, as shown in line L14, if the flicker detection target frequency f is not the same as the flicker light source (100 Hz), the intermediate flicker intensity value Amp increases with increasing frame number n. n For example, it repeats a periodic increase or decrease.

[0097] Thus, for low-intensity flicker under a light source with a certain flicker frequency, the flicker detection target frequency f is set to the same value as the flicker frequency of the light source, and the midpoint value of the flicker intensity Amp n When calculated, it increases monotonically with increasing frame number n. In contrast, the median flicker intensity Amp in noisy pseudo-flicker ｊ This increases or decreases randomly with increasing frame number n. As can be seen from this, for a low-intensity flicker at a certain flicker frequency, the flicker detection target frequency f is set to the same value as the flicker frequency, and the flicker intensity Amp of equation (3) is expressed. n By performing this calculation, it becomes possible to distinguish it from noisy pseudo-flicker.

[0098] Figure 9 shows the midpoint flicker intensity Amp for the event signal x(n). n This figure shows the characteristics of [the compound]. The horizontal axis represents the real part Re of equation (1). n The vertical axis represents the imaginary part Im of equation (1). n This shows that, as mentioned above, vector V1 n Re n +iIm n This shows that vector V1 n This is represented by a vector filled with black. On the other hand, as mentioned above, vector V2 n V1 n -V1 n－１ This shows that, in a time-series event signal x(n) under a light source with flicker frequency f, Re n +iIm n Phase and V2 n The phase of is the same regardless of n. Therefore, vector V1 n The size increases monotonically.

[0099] Figure 10 shows the midpoint flicker intensity Amp when the flicker detection target frequency f is not the same frequency as the flicker light source. n Alternatively, the median flicker intensity Amp for noisy pseudo-flicker n This figure shows the characteristics of [the compound]. The horizontal axis represents the real part Re of equation (1). n The vertical axis represents the imaginary part Im of equation (1). nThis shows that, as mentioned above, vector V1 n Re n +iIm n This shows that vector V1 n This is represented by a vector filled with black. On the other hand, as mentioned above, vector V2 n V1 n -V1 n－１ This indicates.

[0100] Because the flicker detection target frequency f is not the same frequency as the flicker light source, vector V2 n The phase is vector V1 n-1 It is thought that it will shift with a predetermined angular velocity difference. As a result, vector V1 n The size does not increase monotonically.

[0101] Figure 11 shows examples of the characteristics of low-intensity flicker and noisy pseudo-flicker in a comparative example. The horizontal axis represents frequency, and the vertical axis represents the power spectrum. Figure 11(a) is an example of low-intensity flicker that blinks at 100 Hz. Here, low intensity refers to cases where the light intensity of the light source is small or where the illumination area of ​​the light source is small relative to the measurement area.

[0102] In the example shown in Figure 11(a), a peak appears in the power spectrum at a frequency of 100 Hz, exceeding the threshold th. Therefore, flicker is detected. To detect low-intensity flicker, the threshold th needs to be set to a smaller value.

[0103] On the other hand, Figure 11(b) shows an example of noisy pseudo-flicker. When the threshold th is small, noisy pseudo-flicker may exceed the threshold th at a frequency of 100 Hz even though there is no peak. As a result, it is incorrectly identified as flicker. Thus, as in the comparative example, when setting the threshold th to a smaller value to detect low-intensity flicker, the possibility of identifying noisy pseudo-flicker as flicker increases.

[0104] In contrast, the intermediate flicker intensity Amp according to this embodiment n For low-intensity flicker under a light source with power frequency f, the frequency increases monotonically with increasing n, regardless of the light source intensity. Therefore, the intermediate value Amp nBy checking whether or not the flicker is increasing monotonically, it becomes possible to distinguish low-intensity flicker from noisy pseudo-flicker with higher accuracy.

[0105] The flicker stability calculation unit 48 calculates the intermediate flicker intensity value Amp n An evaluation value is calculated that indicates the degree of monotonically increasing with respect to the increase in frame number n. The flicker determination unit 50 calculates an evaluation value that indicates the degree of monotonically increasing flicker intensity Amp with respect to the increase in frame number n. n If the value increases monotonically, it is determined that the flicker is caused by a flicker light source. For example, the flicker determination unit 50 determines whether or not the flicker is caused by flicker based on the evaluation value calculated by the flicker stability calculation unit 48.

[0106] Here, we will explain an example of the processing of the flicker stability calculation unit 48 using Figures 12 and 13. Figure 12 shows the flicker intensity Amp n This figure shows an example of smoothing processing for increasing frame number n. The horizontal axis represents the frame number n, and the vertical axis represents the midpoint flicker intensity Amp. n This shows the median flicker intensity (Amp). n The solid line represents the median flicker intensity (Amp). n This shows the result after smoothing.

[0107] As shown in Figure 12, the flicker stability calculation unit 48 calculates, for example, the intermediate flicker intensity value Amp n The moving average of the window size m is calculated. Here, m = 3. This gives the median flicker intensity Amp after smoothing. n Variation is suppressed.

[0108] The flicker stability calculation unit 48 calculates the intermediate flicker intensity value Amp n After calculating the moving average value, the evaluation value count is calculated using the conditional expression according to equation (4). In other words, this flicker stability calculation unit 48 calculates the midpoint value of the flicker intensity Amp at frame number n-1. n－１ The moving average value and the median flicker intensity Amp at frame number n. n The relationship between the moving average and the midpoint of the flicker intensity is determined, and the median flicker intensity Amp n－１ The moving average value of the median flicker intensity Amp nIf the moving average is large, add 1 to the evaluation value count. On the other hand, the median flicker intensity Amp n－１ The moving average value of the median flicker intensity Amp n If the moving average is small, -1 is added to the evaluation value count.

[0109] The flicker stability calculation unit 48 calculates the intermediate flicker intensity value Amp without smoothing. ｊ It is also possible to generate an evaluation value for this. Furthermore, in this embodiment, the evaluation value count may be referred to as the flicker stability.

[0110] The closer the evaluation value `count` is to the final value of frame number `n`, the higher the stability. In other words, the midpoint of the flicker intensity `Amp` n As the moving average increases monotonically over time, the stability of the variable increases.

[0111]

[0112] Figure 13 shows the change in the evaluation value count. The horizontal axis represents the frame number n, and the vertical axis represents the evaluation value count. Since the window size m used when calculating the moving average is 3, the evaluation value count is set to 0 during period P10. The solid line represents the case where flicker occurs, and the dashed line represents noisy pseudo-flicker. As shown in Figure 13, when flicker occurs, the evaluation value count increases in proportion to the increase in the frame number n. On the other hand, in the case of noisy pseudo-flicker, the evaluation value count increases or decreases randomly in proportion to the increase in the frame number n.

[0113] The flicker determination unit 50 of the second signal processing unit 26 determines whether or not there is flicker based on the evaluation value count calculated by the flicker stability calculation unit 48. That is, the flicker determination unit 50 determines that there is flicker if the evaluation value count exceeds a predetermined value. In this way, the flicker determination unit 50 determines the intermediate flicker intensity value Amp n A flicker is identified when the moving average of the parameters increases monotonically by a predetermined percentage or more.

[0114] The flicker detection unit 50 records information of the pixel coordinates (x, y) that it has determined to be flicker in the recording unit 12. In the second mode, the control unit 13 stops the transfer unit 335 (see Figure 5) from transmitting event information of the pixel coordinates (x, y) that it has determined to be flicker to the readout area selection unit 23. As a result, even if an address event is detected, the reading of the image signal from the pixel coordinates (x, y) that it has determined to be flicker is suppressed, thus suppressing flicker. In this case as well, since noisy pseudo-flicker is not distinguished as flicker, noisy information can also be transmitted in the same way as natural images.

[0115] The subsequent processing unit 60 performs recognition processing on a region based on the coordinates (x, y) included in the event information, for example. This allows it to perform processing to recognize the category of moving objects in the imaging region as people, animals, etc. In the second mode, the control unit 13 controls the subsequent processing unit 60 so as not to use the event information of the pixel coordinates (x, y) that have been determined to be flicker. In such cases, the subsequent processing unit 60 can perform recognition processing without using the region information of the pixel coordinates (x, y) that have been determined to be flicker, thereby reducing misrecognition.

[0116] Figure 14 is a diagram illustrating another configuration example related to flicker detection. It differs from the imaging system 1 described in Figure 6 in that the event number difference calculation unit 44, flicker intensity calculation unit 46, and flicker stability calculation unit 48 of the first signal processing unit 25 are located on the processing unit 20 side. Thus, it is also possible to mainly place the processing unit related to event detection on the imaging device 10 side and the processing unit related to signal processing on the processing unit 20 side.

[0117] Figure 15 illustrates yet another configuration example related to flicker detection. It differs from the imaging system 1 described in Figure 6 in that all of the processing units of the first signal processing unit 25 are located on the imaging device 10 side. Thus, it is also possible to place the processing unit related to flicker detection on the imaging device 10 side.

[0118] Figure 16 is a flowchart showing an example of the flicker detection process. The control unit 13 sets the period N for performing the flicker detection and the regions sa1 to sa8 for determining the flicker, and sets the evaluation value count and frame number n to their initial value of 0 (step S100).

[0119] Next, the event count unit 42 uses the event information (x, y, t, p) that occurred up to period N to determine the total number of positive events and the total number of negative events that occurred in t = Δ × n for each region sa1 to sa8. Then, the event difference calculation unit 44 takes one of the total number of positive events and the total number of negative events as negative and adds them to calculate the event signal x(n) (step S102).

[0120] Next, the flicker intensity calculation unit 46 calculates the intermediate flicker intensity Amp for each region sa1 to sa8 according to equations (1) and (3). n Calculate (step S104).

[0121] Next, the flicker stability calculation unit 48 calculates the intermediate value of the flicker intensity Amp for n. n The moving average value is calculated and an evaluation value count is generated (step S106). The control unit 13 determines whether n is less than N-1 (S108). If the control unit 13 determines that n is less than N-1 (Yes in S108), it adds 1 to n (S110) and repeats the process from step S102. On the other hand, if the control unit 13 determines that n is N-1 or greater (No in S108), the flicker stability calculation unit 48 supplies the evaluation value count for N-1 for each region sa1 to sa8 to the flicker determination unit 50 (S110).

[0122] Next, the flicker detection unit 50 determines whether the evaluation value count is above a predetermined threshold for each region sa1 to sa8 (S114). If it is above the predetermined threshold (Yes in S114), it determines that flicker is occurring (step S114) and terminates the process. On the other hand, if the evaluation value count is below a predetermined threshold (No in S114), the flicker detection unit 50 determines that flicker is not occurring (step S118) and terminates the process.

[0123] Figure 17 is a flowchart showing another example of flicker detection processing. Here, we explain an example in which the flicker detection unit 50 adds yet another detection criterion to the flicker detection.

[0124] After the control unit 13 has performed the processing from step S100 to S112, the flicker determination unit 50 determines the intermediate flicker intensity Amp when n=N-1. N-1 The system determines whether the value is above a predetermined second threshold (step S120), and if it is above the predetermined second threshold (Yes in S120), it determines that flicker is occurring (step S114), and the process ends.

[0125] On the other hand, the flicker determination unit 50 determines that no flicker has occurred (step S116) if the evaluation value count is less than a predetermined threshold (No. in S120), and terminates the process.

[0126] Thus, the median flicker intensity Amp when n=N-1 N-1 By adding this to the judgment, it becomes possible to suppress the detection of flicker caused by weak light that has little effect on the naked eye. In this way, the flicker detection unit 50 can also perform flicker detection using multiple judgment criteria.

[0127] Figure 18 is a flowchart showing an example of imaging control by the control unit 13. As shown in Figure 18, first the control unit 13 sets the period N for the first mode and the period for the second mode (step S200).

[0128] Next, the control unit 13 causes the imaging device 10 and the second signal processing unit 26 to perform the first mode, which is flicker detection processing (step S202). Then, the control unit 13 causes the recording unit 12 to record information about the area where flicker is occurring as pixel coordinates (x, y).

[0129] Next, the control unit 13 causes the imaging device 10 and the second signal processing unit 26 to perform the imaging process, which is the second mode (step S204). In the second mode, the control unit 13 stops the transfer unit 335 from transmitting event information of pixel coordinates (x, y) that have been determined to be flicker to the readout area selection unit 23. In addition, in the second mode, the control unit 13 controls the subsequent processing unit 60 so as not to use the event information of pixel coordinates (x, y) that have been determined to be flicker.

[0130] Next, the control unit 13 determines whether the number of imaging cycles in the second mode has reached a predetermined number (step S206). If the control unit 13 does not determine that the number of imaging cycles has reached a predetermined number (No. in step S206), it repeats the process from step S204.

[0131] On the other hand, if the control unit 13 determines that the number of imaging cycles has reached a predetermined number (Yes in step S206), it determines whether or not to terminate the overall control process (step S208). If the control unit 13 determines that it will not terminate the control process (No in step S206), it repeats the process from step S202. On the other hand, if the control unit 13 determines that it will terminate the control process (Yes in step S206), it terminates the overall control process.

[0132] In this way, it becomes possible to repeat the process of executing the first mode followed by a predetermined number of imaging cycles in the second mode. This makes it possible to periodically determine the flicker region, and even if the flicker region fluctuates, the effects of flicker can be reduced in the images captured in the second mode and in post-processing.

[0133] As described above, according to this embodiment, the event number difference calculation unit 44 generates a first event signal xp(n) indicating the number of times the absolute value of the positive change amount by which the electrical signal generated by the photoelectric conversion element 311 increases exceeds a predetermined threshold value for each time in a time series, and a second event signal xn(n) indicating the number of times the absolute value of the negative change amount by which the electrical signal decreases exceeds a predetermined threshold value for each time in a time series, and generates a time series event signal x(n) in which one of them is inverted in positive and negative. Then, the flicker determination unit 27 determines whether the event signal x(n) is caused by flicker based on the monotonic increase of the value (vector V1 n , equation (1)) regarding the convolution of the flicker detection signal determined by the flicker detection target frequency f with respect to the event signal x(n) and the sampling frequency of the EVS.

[0134] The amplitude component V2 n (V1 n −V1 n－１ ) has the same phase as V1 n－１ when the time series event signal x(n) has periodicity (see FIG. 9). Therefore, by using the value regarding the phase of the amplitude component (for example, vector V1 n , equation (1)), it is possible to determine whether it is caused by flicker. That is, the flicker intensity calculation unit 46 generates, for the time series event signal x(n), the real part Rn of the convolution operation result of the cosine wave of the flicker detection signal composed of a cosine wave and a sine wave having an angular velocity ω, and the imaginary part In of the convolution operation result of the sine wave, and generates the square root of the sum of the squares of the real part R n and the imaginary part I n respectively (see equation (3)). The flicker determination unit 27 determines that the vector V2 n (V1 n −V1 n ) has the same phase as V1 n－１ when the flicker intensity intermediate value Amp n－１ (equation (2)) monotonically increases as the number of frames increases, and can objectively determine that the time series event signal x(n) is caused by flicker. n－１

[0135] Note that the present technology can adopt the following configuration.

[0136] (1) An imaging system comprising: a signal generation unit that generates a time-series event signal in which the absolute value of the positive change amount of the pixel signal generated by a pixel having a photoelectric conversion element exceeds a predetermined threshold in a time-series manner, and a second event signal in which the absolute value of the negative change amount of the pixel signal decreases exceeds a predetermined threshold in a time-series manner, and the positive and negative values ​​of the second event signal are reversed at the baseline; and a determination unit that determines whether or not the event signal is caused by flicker based on a value relating to the phase in the convolution calculation value of the event signal and a flicker detection signal having the same frequency as the flicker frequency to be detected.

[0137] (2) The imaging system according to (1), further comprising: (2) The system performs a convolution operation on the time-series event signal with respect to the flicker detection signal, which consists of a cosine wave and a sine wave having the same frequency as the flicker frequency to be detected, and generates the convolution result of the cosine wave as the real part of the amplitude component and the convolution value of the sine wave as the imaginary part of the amplitude component, and generates an absolute value obtained by adding the real part and the imaginary part of the amplitude component, respectively; and the determination unit determines that the event signal is caused by flicker when the absolute value obtained by adding the real part and the imaginary part of the amplitude component increases monotonically with increasing frame count.

[0138] (3) The imaging system according to (1), wherein the flicker frequency to be detected is the frequency of the light source that generates the flicker.

[0139] (4) The imaging system according to (3), wherein the determination unit determines whether the event signal is caused by flicker when the absolute value in a predetermined number of frames exceeds a predetermined value.

[0140] (5) The imaging system according to (3), further comprising an evaluation processing unit that generates a second absolute value that suppresses random fluctuations of the absolute value, wherein the determination unit determines that the event signal is caused by flicker when the second absolute value increases monotonically in accordance with the increase in the second absolute value of the number of frames.

[0141] (6) The evaluation processing unit calculates the moving average of the absolute value from the second absolute value with the random fluctuations suppressed.n The imaging system described in (5) generates as follows.

[0142] (7) The imaging system according to (6), wherein the evaluation processing unit increases the evaluation value indicating stability when the moving average values ​​of the adjacent frames are larger on the side where the number of frames was increased, and decreases the evaluation value when the moving average values ​​of the adjacent frames are smaller on the side where the number of frames was increased, and the determination unit determines that the event signal is caused by flicker when the evaluation value exceeds a predetermined value.

[0143] (8) The imaging system according to (7), wherein the absolute value is the square root of the sum of the squares of the real sums obtained by adding the real parts of each of the amplitude components and the squares of the imaginary sums obtained by adding the imaginary parts of each of the amplitude components.

[0144] (9) The imaging system according to (1), further comprising: a plurality of photoelectric conversion elements, each of which converts incident light into electrical signals; a plurality of detection units, each corresponding to the plurality of photoelectric conversion elements, which generate event information having a first polarity when the absolute value of the positive change in the voltage corresponding to the electrical signals generated by the plurality of photoelectric conversion elements exceeds a predetermined threshold, and a negative polarity when the absolute value of the negative change in the voltage corresponding to the pixel signal exceeds a predetermined threshold; coordinate information based on the photoelectric conversion element that exceeds the threshold; and time information at the time when the threshold is exceeded; and an event signal generation unit that generates the first event signal and the second event signal using at least one of the event information generated by the plurality of detection units.

[0145] (10) The imaging system according to (2), wherein the flicker intensity calculation unit performs convolution integrals on sine waves and cosine waves obtained by sequentially increasing the rotational phase of the event signal over a predetermined period of time, and generates the real and imaginary parts of the amplitude components at the different rotational phases.

[0146] (11) The imaging system according to (9), wherein the event signal generation unit generates the first event signal and the second event signal using event information generated by one or more detection units.

[0147] (12) The imaging system according to (9), further comprising an analog-to-digital converter that converts pixel signals corresponding to electrical signals generated by each of the plurality of photoelectric conversion elements into digital image signals.

[0148] (13) The imaging system according to (10), further comprising a region selection unit that, when the event information is supplied, causes the analog-to-digital converter to selectively output an electrical signal generated by the photoelectric conversion element, which is identified by the coordinate information contained in the event information.

[0149] (14) The imaging system according to (13), further comprising a control unit that controls the supply of event information from the detection unit used to generate the event signal to the region selection unit when the determination unit determines that the event signal is caused by flicker.

[0150] (15) The imaging system according to (13), further comprising a recognition processing unit that performs recognition processing on an area identified by coordinate information included in the event information in imaging data based on the digital image signal, wherein when the determination unit determines that the event signal is caused by flicker, it performs the recognition processing without using the event information supplied from the detection unit used to generate the event signal.

[0151] (16) The imaging system according to (2), wherein the determination unit determines whether the event signal is caused by flicker based on the phase of the amplitude component, when the convolution result of the flicker detection signal, which is composed of a cosine wave and a sine wave, is used as the real part of the amplitude component and the convolution result of the event signal and the sine wave is used as the imaginary part of the amplitude component of the convolution result of the event signal and the sine wave.

[0152] (17) The imaging system according to (3), wherein the determination unit determines that the event signal is caused by flicker when the phase of each of the amplitude components is within a predetermined range.

[0153] (18) The imaging system according to (17), wherein the determination unit determines that the phase of each amplitude component is within a predetermined range when the magnitude of the complex vector composed of the real and imaginary parts of the amplitude component increases monotonically in accordance with the increase in the number of frames.

[0154] (19) An imaging method comprising: a signal generation step that generates a time-series event signal in which a first event signal indicates in time series the number of times the absolute value of the positive change amount of the pixel signal generated by a pixel having a photoelectric conversion element exceeds a predetermined threshold, and a second event signal indicates in time series the number of times the absolute value of the negative change amount of the pixel signal decreases exceeds a predetermined threshold, and the two event signals are reversed in sign at the baseline; and a determination step that determines whether the event signal is caused by flicker based on a value relating to the phase in the multiplication value of the event signal and a flicker detection signal having the same frequency as the frequency to be detected.

[0155] The aspects of this disclosure are not limited to the individual embodiments described above, but include various modifications that a person skilled in the art could conceive, and the effects of this disclosure are not limited to those described above. In other words, various additions, modifications, and partial deletions are possible, as long as they do not depart from the conceptual idea and spirit of this disclosure derived from the claims and their equivalents.

[0156] 1: Imaging system, 10: Imaging device, 20: Processing unit, 13: Control unit, 33: Address event detection unit (detection unit), 42: Event count unit (event signal generation unit), 44: Event difference calculation unit (signal generation unit), 46: Flicker intensity calculation unit (amplitude component generation unit), 48: Flicker stability calculation unit (evaluation processing unit), 50: Flicker determination unit (determination unit), 60: Subsequent processing unit (recognition processing unit), 311: Photodetector (photoelectric conversion element)

Claims

1. An imaging system comprising: a signal generation unit that generates a time-series event signal in which the first event signal, which shows in a time-series manner the number of times the absolute value of the positive change in the pixel signal generated by a pixel having a photoelectric conversion element exceeds a predetermined threshold, and the second event signal, which shows in a time-series manner the number of times the absolute value of the negative change in the pixel signal exceeds a predetermined threshold, are reversed in sign at the baseline; and a determination unit that determines whether or not the event signal is caused by flicker based on a value relating to the phase in the convolution calculation of the event signal and a flicker detection signal having the same frequency as the flicker frequency to be detected.

2. The imaging system according to claim 1, further comprising: a flicker intensity calculation unit that performs a convolution operation on the time-series event signal with respect to the flicker detection signal composed of a cosine wave and a sine wave having the same frequency as the flicker frequency to be detected, generates the convolution result of the cosine wave as the real part of the amplitude component and the convolution value of the sine wave as the imaginary part of the amplitude component, and generates an absolute value by adding the real part and the imaginary part of the amplitude component, respectively; and a determination unit that determines that the event signal is caused by flicker when the absolute value obtained by adding the real part and the imaginary part of the amplitude component increases monotonically with increasing frame count.

3. The imaging system according to claim 1, wherein the flicker frequency to be detected is the frequency of the light source that generates the flicker.

4. The imaging system according to claim 3, wherein the determination unit determines whether the event signal is caused by flicker when the absolute value in a predetermined number of frames exceeds a predetermined value.

5. The imaging system according to claim 3, further comprising an evaluation processing unit that generates a second absolute value that suppresses random fluctuations of the absolute value, wherein the determination unit determines that the event signal is caused by flicker when the second absolute value increases monotonically in accordance with the increase in the second absolute value of the number of frames.

6. The evaluation processing unit calculates the moving average value of the absolute value from the second absolute value with the random fluctuations suppressed. n The imaging system according to claim 5, which generates as follows.

7. The imaging system according to claim 6, wherein the evaluation processing unit increases the evaluation value indicating stability when the moving average values ​​of the adjacent frames are larger on the side where the number of frames was increased, and decreases the evaluation value when the moving average values ​​of the adjacent frames are smaller on the side where the number of frames was increased, and the determination unit determines that the event signal is caused by flicker when the evaluation value exceeds a predetermined value.

8. The imaging system according to claim 7, wherein the absolute value is the square root of the sum of the squares of the real-part sums obtained by adding the real parts of each of the amplitude components and the squares of the imaginary-part sums obtained by adding the imaginary parts of each of the amplitude components.

9. The imaging system according to claim 1, further comprising: a plurality of photoelectric conversion elements, each of which converts incident light into electrical signals; a plurality of detection units, each corresponding to the plurality of photoelectric conversion elements, which generate event information having a first polarity when the absolute value of the positive change in the voltage of a pixel signal corresponding to the electrical signals generated by each of the plurality of photoelectric conversion elements exceeds a predetermined threshold, and a negative polarity when the absolute value of the negative change in the voltage of a pixel signal exceeds a predetermined threshold; coordinate information based on the photoelectric conversion element that exceeded the threshold; and time information at the time the threshold was exceeded; and an event signal generation unit that generates the first event signal and the second event signal using at least one of the event information generated by the plurality of detection units.

10. The imaging system according to claim 2, wherein the flicker intensity calculation unit performs convolution integrals on sine waves and cosine waves with rotational phases sequentially increased in predetermined time units for the event signal over a predetermined period, and generates the real and imaginary parts of the amplitude components at the different rotational phases.

11. The imaging system according to claim 9, wherein the event signal generation unit generates the first event signal and the second event signal using event information generated by one or more detection units.

12. The imaging system according to claim 9, further comprising an analog-to-digital converter that converts pixel signals corresponding to electrical signals generated by each of the plurality of photoelectric conversion elements into digital image signals.

13. The imaging system according to claim 10, further comprising a region selection unit that, when the event information is supplied, causes the analog-to-digital converter to selectively output an electrical signal generated by the photoelectric conversion element, which is identified by the coordinate information contained in the event information.

14. The imaging system according to claim 13, further comprising a control unit that controls the supply of event information from the detection unit used to generate the event signal to the region selection unit when the determination unit determines that the event signal is caused by flicker.

15. The imaging system according to claim 13, further comprising a recognition processing unit that performs recognition processing on an area identified by coordinate information included in the event information in imaging data based on the digital image signal, wherein when the determination unit determines that the event signal is caused by flicker, it performs the recognition processing without using the event information supplied from the detection unit used to generate the event signal.

16. The imaging system according to claim 2, wherein the determination unit determines whether the event signal is caused by flicker based on the phase of the amplitude component, when the convolution result of the flicker detection signal, which is composed of a cosine wave and a sine wave, is used as the real part of the amplitude component, and the convolution result of the event signal and the sine wave is used as the imaginary part of the amplitude component.

17. The imaging system according to claim 3, wherein the determination unit determines that the event signal is caused by flicker when the phase of each of the amplitude components is within a predetermined range.

18. The imaging system according to claim 17, wherein the determination unit determines that the phase of each amplitude component is within a predetermined range when the magnitude of the complex vector composed of the real and imaginary parts of the amplitude component increases monotonically in accordance with the increase in the number of frames.

19. An imaging method comprising: a signal generation step that generates a time-series event signal in which a first event signal, which indicates in time series the number of times the absolute value of the positive change amount of the pixel signal generated by a pixel having a photoelectric conversion element exceeds a predetermined threshold, and a second event signal, which indicates in time series the number of times the absolute value of the negative change amount of the pixel signal decreases exceeds a predetermined threshold, are inverted in positive and negative signs at the baseline; and a determination unit determination step that determines whether the event signal is caused by flicker based on a value relating to the phase in the multiplication value of the event signal and a flicker detection signal having the same frequency as the frequency to be detected.

Images