Imaging device

Abstract

This imaging device outputs image data derived by performing a process using spatial filters 12, 12a, 13, 13a and temporal filters 14, 14a on respective pixel values of a plurality of pixels. The plurality of pixels are divided into a plurality of pixel groups, and different spatial filters 12, 12a, 13, 13a and different temporal filters 14, 14a are used in processing the pixel values of the pixels according to the pixel groups to which the pixels belong. It is possible to switch between the spatial filters 12, 12a, 13, 13a and the temporal filters 14, 14a used for each of the plurality of pixels.

Description

Imaging device

[0001] The present invention relates to an imaging device that performs predetermined processing on the pixel values of each pixel.

[0002] Recently, event-driven cameras (event-based cameras) designed inspired by the visual systems of living organisms have attracted attention as being useful in the fields of computer vision and robotics. An event-driven camera detects only pixels whose luminance has changed by a predetermined amount or more and outputs the data asynchronously, which is different from a conventional camera that detects the luminance of each of all pixels and outputs it as a frame image.

[0003] Here, it is important that the visual system, like other systems, can be realized with limited hardware resources. For example, in the field of robotics, there are restrictions such as the placement of sensors, computing power, and communication wiring for connecting each component, and within these restrictions, the corresponding visual system must be designed. In this regard, in a conventional event-driven camera, when processing a specific visual event in real time, the number of spikes (events) increases, and a large amount of resources are allocated to the processing and transmission of those spikes.

[0004] The visual system of a living organism has central vision and peripheral vision that perform different processes. Specifically, central vision focuses on the central region within the entire visual field and processes visual information with high resolution, while peripheral vision targets the peripheral region around the central region and performs processing for quickly detecting the movement of objects and changes in light in that region. That is, in each region, a large amount of resources are used for the necessary functions, and the resources allocated to other functions are restricted, and while suppressing the overall resources, the required visual functions are realized.

[0005] Japanese Unexamined Patent Application Publication No. 2024-016542

[0006] However, while conventional imaging devices, including conventional event-driven cameras, exist that focus on reproducing the central visual acuity of biological visual systems or that only detect changes in brightness, none have performed different processing on different pixels using a single imaging means. The present invention has been made in view of these circumstances and aims to provide an imaging device that can suppress the overall number of spikes by performing different processing on different pixels.

[0007] An imaging device according to the present invention, which is in line with the above objective, is an imaging device that outputs image data derived by processing the pixel value of each of a plurality of pixels using a spatial filter and a temporal filter, wherein the plurality of pixels are divided into a plurality of pixel groups, and different spatial filters and different temporal filters are used to process the pixel value of each pixel depending on the pixel group to which the pixel belongs.

[0008] The imaging device according to the present invention divides multiple pixels into multiple pixel groups, and uses different spatial filters and different temporal filters to process the pixel values ​​of pixels depending on the pixel group to which the pixel belongs. This allows different processing to be performed on different pixels, thereby suppressing the overall number of spikes.

[0009] This is an explanatory diagram of an imaging device according to one embodiment of the present invention. This is an explanatory diagram of a pixel group. This is an explanatory diagram of an individual processing circuit.

[0010] Next, with reference to the attached drawings, an embodiment of the present invention will be described to facilitate understanding of the present invention. As shown in Figures 1, 2, and 3, an imaging device 10 according to one embodiment of the present invention is a device that outputs image data derived by processing the pixel values ​​of each of a plurality of pixels 11 using spatial filters 12, 12a, 13, 13a and temporal filters 14, 14a.

[0011] In this embodiment, the imaging device 10 includes an optical lens 15, an image sensor 16, and a signal processing means 17, as shown in Figure 1. The image sensor 16 has a plurality of light-receiving elements that each receive light that has passed through the optical lens 15, and each light-receiving element converts the received light into an electrical signal (i.e., a pixel value). As shown in Figure 2, each light-receiving element corresponds to each pixel 11, and in this embodiment, each pixel 11 (light-receiving element) is arranged so that the plurality of pixels 11 together form a rectangle.

[0012] The pixel values ​​derived at each pixel 11 are converted into digital data by an A / D conversion circuit and then provided to the signal processing means 17. The signal processing means 17 can be configured, for example, by an FPGA (field programmeable gate array). The signal processing means 17 may also receive analog pixel values ​​and convert those pixel values ​​into digital data internally. It goes without saying that the signal processing means 17 can be configured with something other than an FPGA.

[0013] The signal processing means 17 comprises a plurality of individual processing circuits 20, each configured as shown in Figure 3, with spatial filters 12, 12a, 13, 13a, time filters 14, 14a, an ON path section 18, and a spike conversion section 19. There is a one-to-one relationship between the individual processing circuit 20 and the pixels 11, and the digitized pixel value corresponding to the nth pixel 11 is given to the nth individual processing circuit 20, where predetermined processing is performed and an output is produced.

[0014] Spatial filters 12, 12a, 13, and 13a are low-pass filters that perform spatial filtering on a given value, smoothing the pixel value of the pixel 11 based on the pixel values ​​of the pixel 11 in question and the multiple pixels 11 arranged around that pixel 11 in spatial coordinates. Temporal filters 14 and 14a are filters that perform temporal filtering on a given value. Specifically, temporal filter 14 is a low-pass filter that performs smoothing based on the time-varying pixel value of the pixel 11 in question, while temporal filter 14a is a band-pass filter that performs contrast enhancement.

[0015] The ON path section 18 corresponds to the ON path in the visual system of living organisms. It goes without saying that instead of the ON path section 18, or in addition to the ON path section 18, an OFF path section corresponding to the OFF path in the visual system of living organisms may be provided. The spike conversion section 19 converts the values ​​output from the ON path section 18 (or the OFF path section if one is provided) into event pixel (spike) data. The event pixel data is output from the individual processing circuit 20.

[0016] Furthermore, as shown in Figure 3, the individual processing circuit 20 includes a switching unit 22 that switches whether to use spatial filters 12 or 12a for processing a given pixel value, a switching unit 23 that switches whether to use spatial filters 13 or 13a for the same processing, and a switching unit 24 that switches whether to use temporal filters 14 or 14a for the same processing.

[0017] If the individual processing circuit 20 is using the spatial filters 12, 13 and the time filter 14, the following processing is performed sequentially on the pixel values ​​given to the individual processing circuit 20.

[0018] 1) Filtering is performed by spatial filter 12. 2) The values ​​output from spatial filter 12 are sent to two paths at the branching point, and the values ​​sent to one path are further filtered by spatial filter 13 (the values ​​sent to the other path are not filtered by spatial filter 13). 3) A subtraction operation is performed to subtract the value derived by processing only by spatial filter 12 from the value derived by processing both spatial filters 12 and 13.

[0019] 4) The value calculated by the subtraction process is filtered by the time filter 14. 5) The value derived by the filtering process by the time filter 14 passes through the ON path section 18 and the spike conversion section 19 in order and is output from the individual processing circuit 20.

[0020] The values ​​corresponding to each pixel 11 output from the individual processing circuit 20 are output from the signal processing means 17 along with information on which pixel 11 the value corresponds to, and are provided to the information processing terminal 25 (see Figure 1) connected to the signal processing means 17. The information processing terminal 25 generates an image by performing predetermined calculations based on the values ​​provided by the signal processing means 17.

[0021] In this embodiment, the multiple pixels 11 are divided into multiple (two in this embodiment) pixel groups 27 and 28, as shown in Figure 2. In other words, each of the multiple pixels 11 is divided to belong to one of the multiple pixel groups 27 and 28. The individual processing circuit 20 corresponding to each pixel 11 belonging to pixel group 27 is set to use spatial filters 12 and 13 and a time filter 14. In contrast, the individual processing circuit 20 corresponding to each pixel 11 belonging to pixel group 28 is set to use spatial filters 12a and 13a and a time filter 14a.

[0022] Therefore, in this embodiment, different spatial filters 12, 12a, different spatial filters 13, 13a, and different temporal filters 14, 14a are used to process the pixel value of pixel 11 depending on the pixel groups 27 and 28 to which pixel 11 belongs. In this embodiment, spatial filters 12 and 13 are filters that perform processing that increases spatial sensitivity compared to spatial filters 12a and 13a, respectively, and temporal filter 14 is a filter that performs processing that decreases temporal sensitivity compared to temporal filter 14a.

[0023] Therefore, the signal processing means 17 performs processing on the pixel values ​​of each pixel 11 belonging to pixel group 27, comparing them with the pixel values ​​of each pixel 11 belonging to pixel group 28, to increase spatial sensitivity and decrease temporal sensitivity. In other words, in a single image corresponding to multiple pixels 11, by providing regions where the spatial resolution is sparse and the temporal resolution is dense, and regions where the spatial resolution is dense and the temporal resolution is sparse, it is possible to derive the necessary information for each region, suppress the overall number of spikes (number of events), and reduce the computational load of post-processing (processing by the information processing terminal 25 in this embodiment) and the amount of communication when transferring data to post-processing.

[0024] Furthermore, this embodiment includes a switching unit 22 for switching the spatial filters 12, 12a used for each of the multiple pixels 11, a switching unit 23 for switching the spatial filters 13, 13a, and a switching unit 24 for switching the temporal filters 14, 14a. Therefore, by switching each of the switching units 22, 23, and 24, it is possible to group the pixels 11 into pixel groups, making it possible to create, for example, three or more pixel groups.

[0025] Furthermore, the switching by each switching unit 22, 23, and 24 can be performed statically (pre-set) or dynamically (even while the signal processing means 17 is processing data).

[0026] Although embodiments of the present invention have been described above, the present invention is not limited to the above-described forms, and any changes to conditions that do not depart from the gist of the invention are all within the scope of application. For example, a switching unit for switching the spatial filter used or a switching unit for switching the time filter used may not be provided.

[0027] The imaging device according to the present invention uses different spatial and temporal filters for processing the pixel values ​​of pixels belonging to each of the multiple pixel groups, thereby enabling different processing for each pixel group and suppressing the number of spikes. Therefore, it can be applied to real-time visual environment recognition and object tracking in fields such as computer vision, robotics, and IoT embedded technologies.

[0028] 10: Imaging device, 11: Pixel, 12, 12a, 13, 13a: Spatial filter, 14, 14a: Time filter, 15: Optical lens, 16: Image sensor, 17: Signal processing means, 18: ON path unit, 19: Spike conversion unit, 20: Individual processing circuit, 22, 23, 24: Switching unit, 25: Information processing terminal, 27, 28: Pixel group

Claims

1. An imaging device that outputs image data derived by processing the pixel values ​​of multiple pixels using a spatial filter and a temporal filter, wherein the multiple pixels are divided into multiple pixel groups, and different spatial filters and different temporal filters are used to process the pixel values ​​of each pixel depending on the pixel group to which the pixel belongs.

2. The imaging apparatus according to claim 1, further comprising a switching unit for switching the spatial filter and the temporal filter used for each of the plurality of pixels.

Images