Data processing device and data processing system

Abstract

The present disclosure relates to a data processing device and a data processing system that make it possible to reduce the load of processing event data. Data drop processing is executed, in which event data output from an EVS pixel is selected for prescribed units on the basis of a drop rate, and an addition operation corresponding to the selected event data is performed to generate intermediate data. An event occurrence rate, which indicates the rate of occurrence of an event detected by the EVS pixel, is obtained by statistical calculation using the intermediate data, a drop rate according to the event occurrence rate is set in a first data drop processing unit, and the event data is framed using the intermediate data per prescribed cycle. The present technology can be applied to, for example, a data processing device that processes event data.

Description

Data processing apparatus and data processing system

[0001] The present disclosure relates to a data processing apparatus and a data processing system, and particularly to a data processing apparatus and a data processing system capable of reducing the processing load on event data.

[0002] In recent years, the development of an EVS (Event based Vision Sensor) that detects luminance changes for each pixel in real time as an event along with the movement of a subject and outputs event data has been progressing. For example, event data is stream data consisting of the coordinates of a pixel and the polarity of the generated event (positive event or negative event), and a considerable memory read / write bandwidth is required to handle the event data.

[0003] For example, Patent Document 1 discloses a detection system capable of reducing the processing load in a detection unit by separately providing a preprocessing unit that extracts auxiliary information from event data and a detection unit that detects a detection target (or the state of the detection target).

[0004] Japanese Patent Application Laid-Open No. 2023-179970

[0005] However, Patent Document 1 does not disclose a reduction in the processing load of general-purpose event data in the preprocessing unit, and a countermeasure for reducing the processing load on event data more than before is required.

[0006] The present disclosure has been made in view of such a situation, and aims to reduce the processing load on event data.

[0007] A data processing device in one aspect of the present disclosure comprises: a first data drop processing unit that performs data drop processing to select event data output from EVS pixels in predetermined units based on a drop rate indicating the frequency of dropping the event data; an addition calculation unit that generates intermediate data by performing an addition calculation according to the event data selected by the first data drop processing unit; a statistical calculation unit that obtains an event occurrence rate indicating the rate at which events detected by the EVS pixels occur by statistical calculation using the intermediate data, and sets the drop rate according to the event occurrence rate to the first data drop processing unit; and a framing unit that frames the event data using the intermediate data at predetermined intervals.

[0008] One aspect of the data processing system of this disclosure includes an event sensor in which EVS pixels are arranged in an array on the sensor surface; a first data drop processing unit that performs data drop processing to select event data output from the EVS pixels in predetermined units based on a drop rate indicating the frequency of dropping the event data; an addition calculation unit that generates intermediate data by performing an addition calculation according to the event data selected by the first data drop processing unit; a statistical calculation unit that obtains an event occurrence rate indicating the rate at which events detected by the EVS pixels occur by statistical calculation using the intermediate data, and sets the drop rate according to the event occurrence rate to the first data drop processing unit; and a framing unit that frames the event data using the intermediate data at predetermined intervals.

[0009] In one aspect of this disclosure, event data output from EVS pixels is subjected to a data drop process that selects event data at predetermined intervals based on a drop rate indicating the frequency of dropping event data. Intermediate data is generated by performing an addition operation according to the event data selected by the data drop process. An event occurrence rate, which indicates the rate at which events detected by the EVS pixels occur, is determined by statistical calculations using the intermediate data. A drop rate is set according to the event occurrence rate, and the event data is framed using intermediate data at predetermined intervals.

[0010] This is a block diagram illustrating an example configuration of the first embodiment of a data processing system to which this technology is applied. This is a diagram illustrating an example of a frame format used for transmitting event data. This is a diagram illustrating an example of the packet structure of event data. This is a diagram illustrating an example of processing in the frame processing unit. This is a diagram illustrating an example of the event occurrence rate. This is a block diagram illustrating an example of the selector configuration. This is a block diagram illustrating an example configuration of the first variation of the data processing system in Figure 1. This is a block diagram illustrating an example configuration of the second variation of the data processing system in Figure 1. This is a block diagram illustrating an example configuration of the third variation of the data processing system in Figure 1. This is a block diagram illustrating an example configuration of the second embodiment of a data processing system to which this technology is applied. This is a diagram illustrating an example of a line header attached to event data. This is a block diagram illustrating an example configuration of the first variation of the data processing system in Figure 10. This is a block diagram illustrating an example configuration of the second variation of the data processing system in Figure 10. This is a diagram illustrating an example of the exposure period of an RGB pixel. This is a diagram illustrating an example of a sensor output format. This is a block diagram illustrating an example configuration of the third embodiment of a data processing system to which this technology is applied. This is a block diagram illustrating an example configuration of the fourth embodiment of a data processing system to which this technology is applied. This is a block diagram illustrating an example configuration of the first variation of the data processing system in Figure 17. This is a block diagram illustrating an example configuration of the fifth embodiment of a data processing system to which this technology is applied. This is a diagram illustrating the effect of statistical calculations.

[0011] The following describes in detail a specific embodiment of this technology, with reference to the drawings.

[0012] <First Configuration Example of a Data Processing System> Referring to Figures 1 to 9, a first configuration example of a data processing system to which this technology is applied will be described.

[0013] Figure 1 is a block diagram showing an example configuration of a first embodiment of a data processing system to which this technology is applied.

[0014] As shown in Figure 1, the data processing system 11 is configured to include an event sensor 21 and a post-processor 22.

[0015] The event sensor 21 is configured to include EVS pixels 31, an encoding unit 32, and a transmission unit 33, with multiple EVS pixels 31 arranged in an array on the sensor surface. The post-processor 22 is configured to include a receiving unit 41, a decoding unit 42, a pre-processing unit 43, a frame processing unit 44, and a post-processing unit 45.

[0016] The EVS pixel 31 compares the change in light brightness with a threshold and outputs event data indicating that an event has occurred when the change in light brightness exceeds the threshold. For example, if the EVS pixel 31 detects that the change in brightness in the bright direction exceeds the threshold, it outputs event data indicating that a positive event has occurred, and if it detects that the change in brightness in the dark direction exceeds the threshold, it outputs event data indicating that a negative event has occurred.

[0017] The encoding unit 32 encodes the event data output from the EVS pixels 31 using a predetermined encoding method and supplies it to the transmission unit 33.

[0018] The transmitting unit 33 transmits the event data encoded in the encoding unit 32 to the post-processor 22 in a frame format commonly used in image sensors, such as the one shown in Figure 2.

[0019] The receiving unit 41 receives event data transmitted from the transmitting unit 33 and supplies it to the decoding unit 42.

[0020] The decoding unit 42 decodes the event data supplied from the receiving unit 41 according to the encoding method used by the encoding unit 32 and supplies it to the preprocessing unit 43.

[0021] The preprocessing unit 43 performs necessary preprocessing on the event data supplied from the decoding unit 42 before frame processing, and supplies the preprocessed event data to the frame processing unit 44 in predetermined units (for example, line units, packet units, or region units).

[0022] For example, Figure 3 shows an example of the packet structure of event data supplied from the preprocessor 43 to the frame processing unit 44. As shown in the figure, one packet of event data consists of an X start coordinate, a Y start coordinate, time information, polarity information, and a data portion. For example, a polarity information of 0 indicates a negative event, and a polarity information of 1 indicates a positive event. The data portion consists of k bits according to the number of EVS pixels 31 per packet. For example, if the X start coordinate is xi, the Y start coordinate is yj, and the data portion is 32 bits, then for the 32 EVS pixels 31 at coordinates (xi, yj), (xi+1, yj), ..., (xi+31, yj), the data portion is set to 1 if an event has occurred, and to 0 if no event has occurred.

[0023] As shown in the figure, the frame processing unit 44 is configured to include a selector 51, an adder 52, an intermediate data holding unit 53, a statistical calculation unit 54, and a framing unit 55.

[0024] The selector 51 stores event data in predetermined units, which is input to the frame processing unit 44, in a buffer 56 (for example, a FIFO (First-In First-Out)). The selector 51 is also set by the statistical calculation unit 54 to have a drop rate that indicates the frequency of dropping event data. The selector 51 then performs a process (hereinafter referred to as data drop processing) to select and discard event data in predetermined units to be supplied to the adder 52 according to the drop rate. Specifically, when the selector 51 uses event data with a packet structure as shown in Figure 3 above, it performs a process to apply a mask to the data portion that will be targeted for event data dropping (for example, a process to replace the event data with 0).

[0025] The adder 52 generates intermediate data that is commonly used in the statistical calculation unit 54 and the framing unit 55 by performing addition operations according to the event data selected by the selector 51.

[0026] For example, the adder 52 reads intermediate data held in the intermediate data holding unit 53 according to the coordinates of the event data supplied from the selector 51. The adder 52 then updates (generates) the intermediate data by adding the event data to the intermediate data read from the intermediate data holding unit 53, and has the updated intermediate data held in the intermediate data holding unit 53. For example, the adder 52 can obtain intermediate data based on various addition methods such as counting systems and timestamp systems, as described later. The adder 52 also instructs the framing unit 55 to perform framing at a timing according to the frame rate (predetermined period) at which the frame processing unit 44 frames the event data.

[0027] The intermediate data holding unit 53 holds the intermediate data obtained by the adder 52 for each coordinate of the event data.

[0028] The statistical calculation unit 54 uses the intermediate data held in the intermediate data holding unit 53 to statistically calculate the event occurrence rate detected by the EVS pixels 31, and sets a variable drop rate in the selector 51 according to the event occurrence rate. For example, if the statistical calculation unit 54 finds through statistical calculation that the event occurrence rate is excessively high, it performs feedback control to increase the drop rate set in the selector 51.

[0029] When the framing unit 55 receives an instruction from the adder 52 to perform framing, it frames the event data using the intermediate data read from the intermediate data holding unit 53 and supplies the framed event data to the post-processing unit 45.

[0030] The post-processing unit 45 performs necessary post-processing on the event data framed by the frame processing unit 44 after the frame processing has been completed.

[0031] An example of processing in the frame processing unit 44 will be described with reference to Figures 4 and 5.

[0032] On the left side of Figure 4, an example of event data for coordinates (xi to xi+5, yj) during the period T from time tk to time tk+T is shown, which is input to the adder 52 via the selector 51. In the example shown, event data indicating that an event has occurred is hatched, and the event data for coordinates (xi+4, yj) at time tk, time tk+2, time tk+3, and time tk+T indicates that an event has occurred.

[0033] In the center of Figure 4, an example of an image of the intermediate data held in the intermediate data holding unit 53 is shown. For example, the intermediate data is held for each coordinate of the event data. As described above, if the event data for coordinate (xi+4, yj) indicates that an event has occurred, the intermediate data for each time point is obtained, as shown on the right side of Figure 4.

[0034] For example, when intermediate data is obtained using a counting system, the adder 52 uses the event count value obtained by counting up from 0 during the period T as intermediate data if the event data supplied from the selector 51 indicates that an event has occurred. If the event count value exceeds the set value n, the adder 52 sets the event count value from that point onward to n. Then, when the period T has elapsed, the adder 52 instructs the framing unit 55 to perform framing of the event data, and then resets the event count value to 0.

[0035] At this time, the framing unit 55 can frame the event count value as is, or it can normalize the event count value by a set value n and frame it, or it can binarize the event count value to 0 or 1 or more (=Binary Image) and frame it.

[0036] Furthermore, when intermediate data is obtained using a timestamp system, the adder 52 uses the event count value obtained using the three channels Ch1 to Ch3 as the intermediate data. For example, if the event data supplied from the selector 51 indicates that an event has occurred, the adder 52 sets channel Ch1 to 1, moves the event count value of channel Ch1 prior to that timing to channel Ch2, and moves the event count value of channel Ch2 prior to that timing to channel Ch3. Then, at each time step, the adder 52 counts up if the event count value of channels Ch1 to Ch3 is not 0, and if the event count value exceeds a predetermined value tn, it assigns 0 to the event count value. Then, when the period T has elapsed, the adder 52 instructs the framing unit 55 to perform framing of the event data, and then the event count value may or may not be reset.

[0037] At this time, the framing unit 55 can frame by multiplying the event count value by a predetermined time step, frame by obtaining the Log of the event count value, frame by normalizing the event count value with a specified value tn, or frame by normalizing the event count values of channels Ch1 to Ch3 with a specified value tn and a predetermined bin size, clipping them, and then obtaining the sum.

[0038] When the intermediate data is obtained in a counting system, the statistical calculation unit 54 can use, as the event occurrence rate, the average value of the intermediate data for each EVS pixel 31, the intermediate data for each EVS pixel 31 with the value per time, etc.

[0039] When the intermediate data is obtained in a time stamp system, for example, the statistical calculation unit 54 can use, as the event occurrence rate, the average value for each EVS pixel 31 obtained by the following formula (1).

[0040]

[0041] Fig. 5 shows an example in which the statistical calculation unit 54 divides the sensor surface of the event sensor 21 into a plurality of regions (in the illustrated example, nine regions 1 to 9) and represents the event occurrence rate by the density of dot hatching. For example, as shown in Fig. 5, when the event occurrence rate is high (the dot hatching is dense) around region 5, it can be determined that a moving subject is captured in region 5, or it can be determined that there is a high possibility that a moving subject is captured in region 5. Therefore, the statistical calculation unit 54 can utilize the analysis result that the moving subject continues to be captured around region 5 and event data is likely to occur frequently at the next time in the drop rate set for the selector 51.

[0042] Here, the statistical calculation unit 54 can predict a tendency for event data to increase at the next time by obtaining the event occurrence rate through statistical calculation.

[0043] That is, when the event rate of events in the entire viewing angle is high for the event data at the most recent time, the statistical calculation unit 54 can predict that the entire viewing angle is in a moving state or a state where high noise is generated in a dark place, and it is highly likely that the same state will occur at the next time. Or, when the distribution of the event data at the most recent time is biased towards a part of the viewing angle, the statistical calculation unit 54 can predict that there are moving subjects, blinking objects, etc. in a specific area, and it is highly likely that data derived from the same moving subjects, blinking objects, etc. will be observed at the next time.

[0044] The statistical calculation unit 54 predicts such a tendency by calculating the density with respect to the intermediate data for time stamp calculation, and may introduce a procedure for obtaining a coefficient I1 and a coefficient Idiff for adjusting the drop rate of event data at the next time as described below.

[0045] For example, the statistical calculation unit 54 sets the first and second channels of the intermediate data for time stamp calculation as ch1 and ch2, and sets the threshold values of the prescribed time values as τ and τdiff in advance. Then, the statistical calculation unit 54 obtains the coefficient I1 according to the following formula (2).

[0046]

[0047] Next, the statistical calculation unit 54 sets the density (average value) of the event data in the entire viewing angle of I1 as N all and sets the variance value of the density value Ni of each region when the entire viewing angle of I1 is divided into a K×K region as σ. Then, the statistical calculation unit 54 obtains the entropy using the density value Ni of each region when the entire viewing angle of I1 is divided into a K×K region according to the following formula (3).

[0048]

[0049] Furthermore, the statistical calculation unit 54 obtains the coefficient Idiff according to the following formula (4).

[0050]

[0051] Then, the statistical calculation unit 54 sets the density N of the event data in the entire viewing angle of I1 allI1: The variance value σ of the density value Ni of each region when the entire field of view is divided into K × K regions, and Idiff: the data density N of the entire field of view. all_diff Using this, the drop rate at the next time step can be calculated according to the following equation (5).

[0052]

[0053] Note that in equation (5), λ all , λ α , and λ all_diff These are predetermined coefficients. Furthermore, the method for calculating the drop rate for the next time step described here is just one example, and the statistical calculation unit 54 may calculate the drop rate for the next time step by including other statistics and algorithms.

[0054] Figure 6 is a block diagram illustrating another configuration example of the selector 51.

[0055] As shown in Figure 6A, the selector 51 may have a built-in memory 57, such as SRAM (Static Random Access Memory), or a memory located near the selector 51, and may be configured to store event data in the memory 57. Such a configuration can speed up processing by the selector 51.

[0056] As shown in Figure 6B, the selector 51 may employ a configuration that stores event data in system memory 58, such as DDR (Double Data Rate memory). Of course, various types of memory other than DDR can be used as system memory 58.

[0057] Alternatively, a selector 51 with a configuration that combines the configuration shown in Figure 6A and the configuration shown in Figure 6B may be adopted.

[0058] As described above, the data processing system 11 is configured such that the frame processing unit 44 accepts event data input at predetermined units and reduces the processing load on the event data in the framing unit 55 through data drop processing by the selector 51. For example, in the frame processing unit 44, the statistical calculation unit 54 calculates the event occurrence rate of events previously detected by the EVS pixels 31 through statistical calculations, and the selector 51 adjusts the frequency of dropping the event data for the next time step according to a drop rate that is variable according to the event occurrence rate, thereby reducing the processing load on the framing unit 55. Furthermore, in the frame processing unit 44, the intermediate data used by the statistical calculation unit 54 to calculate the event occurrence rate through statistical calculations and the intermediate data used by the framing unit 55 to frame the event data are the same.

[0059] Therefore, the data processing system 11 only needs to prepare intermediate data corresponding to the number of pixels (Height × Width × Channel) bits of the framed event data output from the frame processing unit 44. Furthermore, when writing intermediate data to the intermediate data holding unit 53, the data processing system 11 sets the sum of each intermediate data to one of the following: the count value of the event data input to the frame processing unit 44, the elapsed time since the previous writing of intermediate data, and the convolution of the event and its surroundings, or a combination thereof. This allows the system to generate intermediate data in a data representation corresponding to the event count value and timestamp value in the aforementioned period T when framing the event data. In addition, the data processing system 11 allows the statistical calculation unit 54 to quickly perform statistical calculations by using the number of bits of the intermediate data directly as the input rate, thereby improving the efficiency of the entire pipeline.

[0060] In the explanation above, the drop rate to be set in selector 51 is determined based on the amount of data present in the intermediate data. Alternatively, for example, processing could be started with a high drop rate set in advance, and a predetermined coefficient could be set in advance according to the degree of increase or decrease in the capacity of the intermediate data, and the drop rate could be reduced based on this coefficient. Even in this case, the control to increase or decrease the drop rate according to the amount of intermediate data would essentially be repeated.

[0061] As described above, the data processing system 11 can reduce the processing load on event data compared to conventional systems. As a result, the data processing system 11 can enable the execution of applications even with limited memory and bandwidth in systems that utilize event data.

[0062] Figure 7 is a block diagram showing an example configuration of the first variation of the data processing system 11. In the data processing system 11-1 shown in Figure 7, components common to the data processing system 11 in Figure 1 are denoted by the same reference numerals, and their detailed explanations are omitted.

[0063] As shown in Figure 7, the data processing system 11-1 is configured to include an event sensor 21-1 and a post-processor 22-1. The data processing system 11-1 has the same configuration as the data processing system 11 in Figure 1, in that the event sensor 21-1 includes the EVS pixels 31 and the transmission unit 33, and the post-processor 22-1 includes the receiving unit 41.

[0064] Furthermore, the data processing system 11-1 differs from the data processing system 11 in Figure 1 in that the event sensor 21-1 includes a pre-processing unit 43, a frame processing unit 44, a post-processing unit 45, and a communication control unit 61. The communication control unit 61 performs communication control to transmit the framed and post-processed event data to the post-processor 22-1 via the transmission unit 33 and the reception unit 41. For example, the communication control unit 61 can transmit event data from the transmission unit 33 to the reception unit 41 using an existing frame transmission standard such as MIPI CSI-2 (Mobile Industry Processor Interface Camera Serial Interface - 2).

[0065] In other words, while the data processing system 11 in Figure 1 was configured with a post-processor 22 comprising a pre-processing unit 43, a frame processing unit 44, and a post-processing unit 45, the data processing system 11-1 is configured with an event sensor 21-1 comprising a pre-processing unit 43, a frame processing unit 44, and a post-processing unit 45.

[0066] Therefore, the data processing system 11-1 is configured such that event data, which has been framed by the frame processing unit 44 provided in the event sensor 21-1 and post-processed by the post-processing unit 45, is transmitted from the event sensor 21-1 to the post-processor 22-1. In the data processing system 11-1 configured in this way, as with the data processing system 11 in Figure 1, the selector 51 performs data drop processing according to the drop rate set by the statistical calculation unit 54, thereby reducing the processing load on the event data.

[0067] Figure 8 is a block diagram showing an example configuration of a second variation of the data processing system 11. In the data processing system 11-2 shown in Figure 8, components common to the data processing system 11 in Figure 1 are denoted by the same reference numerals, and their detailed explanations are omitted.

[0068] As shown in Figure 8, the data processing system 11-2 is configured to include an event sensor 21-2 and a post-processor 22-2. The data processing system 11-2 has the same configuration as the data processing system 11 in Figure 1, in that the event sensor 21-2 provides the EVS pixels 31 and the transmission unit 33, and the post-processor 22-2 provides the receiving unit 41 and the post-processing unit 45.

[0069] Furthermore, the data processing system 11-2 differs from the data processing system 11 in Figure 1 in that the event sensor 21-2 includes a pre-processing unit 43, a frame processing unit 44, 2, and a communication control unit 61. The communication control unit 61 performs communication control to transmit framed event data to the post-processor 22-2 via the transmission unit 33 and the reception unit 41. For example, the communication control unit 61 can output event data from the transmission unit 33 to the reception unit 41 using an existing frame transmission standard such as MIPI CSI-2.

[0070] In other words, while the data processing system 11 in Figure 1 was configured with a post-processor 22 providing a pre-processing unit 43 and a frame processing unit 44, the data processing system 11-2 is configured with an event sensor 21-2 providing a pre-processing unit 43 and a frame processing unit 44.

[0071] Accordingly, the data processing system 11-2 is configured such that event data framed by the frame processing unit 44 provided in the event sensor 21-2 is transmitted from the event sensor 21-2 to the post-processor 22-2, and post-processing is performed on the framed event data in the post-processor 22-2. In the data processing system 11-2 configured in this way, as with the data processing system 11 in Figure 1, the selector 51 performs data drop processing according to the drop rate set by the statistical calculation unit 54, thereby reducing the processing load on the event data.

[0072] Figure 9 is a block diagram showing an example configuration of a third variation of the data processing system 11. In the data processing system 11-3 shown in Figure 9, components common to the data processing system 11 in Figure 1 are denoted by the same reference numerals, and their detailed explanations are omitted.

[0073] As shown in Figure 9, the data processing system 11-3 is configured to include an event sensor 21 and a post-processor 22-3, with the event sensor 21 having the same configuration as in Figure 1. The data processing system 11-3 has the same configuration as the data processing system 11 in Figure 1, in that the post-processor 22-3 includes a receiving unit 41, a decoding unit 42, and a pre-processing unit 43.

[0074] Furthermore, the data processing system 11-3 differs from the data processing system 11 in Figure 1 in that the post-processor 22-3 includes a frame processing unit 44-3, a post-processing unit 45-3, receiving units 62a to 62c, and sensor data processing units 63a to 63c. The frame processing unit 44-3 has the same configuration as the frame processing unit 44 in Figure 1 in that it includes a selector 51, an adder 52, an intermediate data holding unit 53, a statistical calculation unit 54, and a buffer 56, but differs from the frame processing unit 44 in Figure 1 in that it includes a framing unit 55-3.

[0075] The receiving units 62a to 62c receive sensor data transmitted from external sensors (not shown) (for example, RGB image sensors other than the event sensor 21, or IMUs (Inertial Measurement Units)) and supply them to the respective sensor data processing units 63a to 63c.

[0076] The sensor data processing units 63a to 63c each perform data processing on the sensor data supplied via the receiving units 62a to 62c, and then supply the processed sensor data to the framing unit 55-3 and the post-processing unit 45-3.

[0077] The framing unit 55-3 processes part or all of the sensor data that has been processed in the sensor data processing units 63a to 63c, and can utilize the sensor data in controlling the framing of event data.

[0078] The post-processing unit 45-3 processes part or all of the sensor data that has been processed in the sensor data processing units 63a to 63c, and can utilize the sensor data in controlling the post-processing of the framed event data in the frame processing unit 44.

[0079] As described above, the data processing system 11-3 configured in this way can reduce the processing load on event data, and by utilizing sensor data output from sensors other than the event sensor 21, it can improve the efficiency of framing and post-processing of event data.

[0080] Furthermore, a configuration in which multiple event sensors (not shown) are connected to the post-processor 22-3 may be adopted, and some or all of the event data may be supplied to a common or independent framing unit 55-3.

[0081] <Second Configuration Example of a Data Processing System> A second configuration example of a data processing system to which this technology is applied will be described with reference to Figures 10 to 15.

[0082] Figure 10 is a block diagram showing an example configuration of a second embodiment of a data processing system to which this technology is applied. In the data processing system 11A shown in Figure 10, components common to the data processing system 11 in Figure 1 are denoted by the same reference numerals, and their detailed descriptions are omitted.

[0083] As shown in Figure 10, the data processing system 11A is configured to include an event sensor 21 and a post-processor 22A, with the event sensor 21 having the same configuration as in Figure 1. The data processing system 11A has the same configuration as the data processing system 11 in Figure 1, in that the post-processor 22A includes a receiving unit 41, a decoding unit 42, a pre-processing unit 43, a frame processing unit 44, and a post-processing unit 45.

[0084] Furthermore, the data processing system 11A differs from the data processing system 11 in that the post-processor 22A includes the selector 71 and buffer 72. As shown in the figure, the selector 71 and buffer 72 are provided between the pre-processing unit 43 and the frame processing unit 44.

[0085] The selector 71 receives event data in predetermined units (e.g., line units, packet units, or area units) from the preprocessing unit 43, and the selector 71 stores the event data of the predetermined units in a buffer 72 (e.g., FIFO (First-In First-Out)). Then, according to pre-set rules, the selector 71 performs data drop processing to select event data to be input to the frame processing unit 44, for example, in a high-speed feedforward manner.

[0086] For example, in the data processing system 11A, the selector 71 can make a decision to drop event data based on header information attached to the event data packets supplied from the preprocessing unit 43. Specifically, the selector 71 can perform data drop processing by analyzing one or more pieces of information, such as the row or area number in the event data frame, or metadata associated with the frame.

[0087] Furthermore, the selector 71 may perform data drop processing on the event data on a line-by-line basis, corresponding to each line of the event data frame received by the receiving unit 41.

[0088] For example, as shown in Figure 11, a line header LH is assigned to each line of event data before encoding. The line header LH is basically used to describe the decoding method for each line in the decoding unit 42, but it can also contain other metadata. Therefore, by including metadata in the line header LH for selecting whether or not to drop event data in the selector 71, the selector 71 can perform data drop processing according to that metadata.

[0089] In this case, it is assumed that the event sensor 21 is configured such that a line header LH is added to the event data at the encoding unit 32, or at least at a stage prior to the encoding unit 32. Therefore, it is sufficient that the line header LH is included in either the encoded data or the decoded data, regardless of whether it occurs before or after the decoding unit 42.

[0090] Furthermore, the data processing system 11A can use an error code in the metadata included in the line header LH to indicate that the event sensor 21 failed to output event data from the EVS pixel 31. The selector 71 can then perform data drop processing according to the error code included in the metadata of the line header LH. This allows the data processing system 11A to implement, for example, measures to prevent interference with data output.

[0091] Furthermore, in the data processing system 11A, when event data is transmitted from the event sensor 21 to the post-processor 22A in a compressed state, the event data compression mode can be used in the metadata included in the line header LH. The statistical calculation unit 54 then estimates the density of the event data decoded from the compressed state according to the event data compression mode included in the metadata of the line header LH, and can use the estimation result as the drop rate to be set in the selector 51.

[0092] The data processing system 11A configured in this way can perform event data dropping in a feedforward manner at high speed using the selector 71, and can perform event data dropping while maintaining statistical accuracy using the selector 51. As a result, the data processing system 11A can further improve the system's bandwidth efficiency by, for example, rapidly selecting event data that has been previously determined to be of low importance.

[0093] Figure 12 is a block diagram showing an example configuration of a first variation of the data processing system 11A. In the data processing system 11A-1 shown in Figure 12, components common to the data processing system 11A in Figure 10 are denoted by the same reference numerals, and their detailed explanations are omitted.

[0094] As shown in Figure 12, the data processing system 11A-1 is configured such that the positioning of the selector 71 and buffer 72 in the post-processor 22A-1 differs from that of the post-processor 22A of the data processing system 11A in Figure 10. Specifically, while the post-processor 22A of the data processing system 11A in Figure 10 is configured with the selector 71 and buffer 72 positioned between the pre-processing unit 43 and the frame processing unit 44, the post-processor 22A-1 of the data processing system 11A-1 is configured with the selector 71 and buffer 72 positioned between the receiving unit 41 and the decoding unit 42.

[0095] In the data processing system 11A-1 configured in this way, similar to the data processing system 11A in Figure 10, the selectors 51 and 71 perform data drop processing, thereby further improving bandwidth efficiency.

[0096] Figure 13 is a block diagram showing an example configuration of a second variation of the data processing system 11A. In the data processing system 11A-2 shown in Figure 13, components common to the data processing system 11A in Figure 10 are denoted by the same reference numerals, and their detailed explanations are omitted.

[0097] As shown in Figure 13, the data processing system 11A-2 is configured to include an event sensor 21A and a post-processor 22A, the post-processor 22A having the same configuration as in Figure 10. The data processing system 11A-2 has the same configuration as the data processing system 11A in Figure 10 in that the event sensor 21A includes the EVS pixels 31 and the transmission unit 33.

[0098] Furthermore, the data processing system 11A-2 differs from the data processing system 11A in Figure 10 in that the encoding unit 32A and the RGB pixels 34 are provided by the event sensor 21A.

[0099] The RGB pixels 34 capture an RGB image consisting of, for example, red, green, and blue, and output RGB image data.

[0100] The encoding unit 32A encodes the event data output from the EVS pixel 31 and the RGB image data output from the RGB pixel 34 and supplies them to the transmission unit 33.

[0101] Here, as shown in Figure 14, the data processing system 11A-2 controls the row-direction scanning timing of the RGB pixels 34, and the period from the completion of reading out one pixel signal to the start of reading out the next pixel signal is set as the exposure period. Therefore, the data processing system 11A-2 may include an exposure period flag in the line header LH that indicates whether or not the RGB pixels 34 are in the exposure period, and the exposure period flag can be used in the data drop processing by the selector 71.

[0102] For example, the selector 71 can determine whether each row or region of event data packet has an RGB pixel 34 within or outside the exposure period by referring to the exposure period flag, and can perform data drop processing based on the importance of whether it is within or outside the exposure period.

[0103] Referring to Figure 15, the sensor output format including metadata will be explained.

[0104] For example, in a data processing system 11A in which the post-processor 22 includes a frame processing unit 44, it is useful to add the following metadata to the payload output from the event sensor 21 (for example, a frame header FH or line header H as shown in Figure 15A).

[0105] Specifically, metadata can include the frame rate of event data output from the event sensor 21 (number of events per frame or per unit time), and a flag representing the density of events according to the event data (a flag indicating 1 if the number of events is above a threshold, and 0 otherwise). Metadata can also include information specifying the encoding method and the amount of data in a single line that expands or contracts due to encoding. Furthermore, in the case of the data processing system 11A-2 shown in Figure 13, a flag indicating scanning interference between the EVS pixels 31 and RGB pixels 34 can be used as metadata.

[0106] Alternatively, as shown in Figure 15B, metadata for a frame may be added to the frame header FH provided in the event data of a single frame, or a single frame may be divided into multiple regions (four regions 0 to 3 in the example shown in Figure 15B) and metadata for each region may be added.

[0107] Similarly, as shown in Figure 15C, metadata for a line may be added to the line header LH provided in the event data of a single line, or a single line may be divided into multiple groups (four groups, Group 0 to Group 3, in the example shown in Figure 15C) and metadata for each group may be added.

[0108] <Third Configuration Example of a Data Processing System> Referring to Figure 16, a third configuration example of a data processing system to which this technology is applied will be described.

[0109] Figure 16 is a block diagram showing an example configuration of a third embodiment of a data processing system to which this technology is applied. In the data processing system 11B shown in Figure 16, components common to the data processing system 11 in Figure 1 are denoted by the same reference numerals, and their detailed descriptions are omitted.

[0110] As shown in Figure 16, the data processing system 11B is configured to include an event sensor 21B and a post-processor 22B. The data processing system 11B has the same configuration as the data processing system 11 in Figure 1, in that the event sensor 21B includes the EVS pixels 31 and the transmission unit 33, and the post-processor 22B includes the receiving unit 41 and the pre-processing unit 43.

[0111] Furthermore, the data processing system 11B differs from the data processing system 11 in Figure 1 in that the event sensor 21B includes an encoding unit 32B and RGB pixels 34, while the post-processor 22B includes a decoding unit 42B, a frame processing unit 44B, a post-processing unit 45B, a pre-processing unit 81, and a compression processing unit 82.

[0112] The RGB pixels 34 capture an RGB image consisting of, for example, red, green, and blue, and output RGB image data.

[0113] The encoding unit 32B encodes the event data output from the EVS pixel 31 and the RGB image data output from the RGB pixel 34 and supplies them to the transmission unit 33.

[0114] The decoding unit 42B decodes the event data and RGB image data encoded by the encoding unit 32B, and supplies the event data to the preprocessing unit 43 and the RGB image data to the preprocessing unit 81.

[0115] The frame processing unit 44B has the same configuration as the frame processing unit 44 in Figure 1, in that it includes an adder 52, an intermediate data holding unit 53, a statistical calculation unit 54, a framing unit 55, and a buffer 56, but it has a different configuration from the frame processing unit 44 in Figure 1, in that it includes a selector 51B. The selector 51B can perform data drop processing for each region divided by the region division unit 83, in addition to the drop rate set by the statistical calculation unit 54, according to the divided region information supplied from the region division unit 83 of the compression processing unit 82.

[0116] The post-processing unit 45B performs post-processing on the event data framed in the frame processing unit 44, and supplies the post-processed event data to the compression unit 84 of the compression processing unit 82.

[0117] The preprocessing unit 81 performs preprocessing on the RGB image data supplied from the decoding unit 42B, and then supplies the preprocessed RGB image data to the compression processing unit 82.

[0118] The compression processing unit 82 includes a region division unit 83 and a compression unit 84, and can output a compressed video of the RGB image data supplied from the preprocessing unit 81 (for example, to either the internal or external location of the post-processor 22B).

[0119] When the compression processing unit 82 performs compression processing, the region division unit 83 divides the RGB image data into fixed intervals in the spatial direction, or time-divides the RGB image data into intervals according to a predetermined rule, and supplies the RGB image data for each divided region to the compression unit 84. The region division unit 83 can then supply the frame processing unit 44B with divided region information indicating each of the divided regions of the RGB image data.

[0120] The compression unit 84 compresses the RGB image data for each divided region supplied from the region division unit 83 according to a predetermined compression method (for example, a codec such as H264 / H265). At this time, the compression unit 84 can compress the RGB image data by referring to event data supplied from the post-processing unit 45B.

[0121] In the data processing system 11B configured in this way, the selector 51B performs data drop processing according to the divided region information supplied from the region division unit 83 of the compression processing unit 82, in addition to the drop rate set by the statistical calculation unit 54, thereby further reducing the processing load on event data.

[0122] <Fourth Configuration Example of a Data Processing System> Referring to Figures 17 and 18, a configuration example of a fourth embodiment of a data processing system to which this technology is applied will be described.

[0123] Figure 17 is a block diagram showing an example configuration of a fourth embodiment of a data processing system to which this technology is applied. In the data processing system 11C shown in Figure 17, components common to the data processing system 11 in Figure 1 are denoted by the same reference numerals, and their detailed descriptions are omitted.

[0124] As shown in Figure 17, the data processing system 11C is configured to include an event sensor 21 and a post-processor 22C, with the event sensor 21 having the same configuration as in Figure 1. The data processing system 11C has the same configuration as the data processing system 11 in Figure 1, in that the post-processor 22C includes a receiving unit 41, a decoding unit 42, and a pre-processing unit 43.

[0125] Furthermore, the data processing system 11C has a different configuration from the data processing system 11 in Figure 1, in that the post-processor 22C includes a frame processing unit 44C and a post-processing unit 45C. The frame processing unit 44C has the same configuration as the frame processing unit 44 in Figure 1, in that it includes an adder 52, an intermediate data holding unit 53, a statistical calculation unit 54, a framing unit 55, and a buffer 56, but it has a different configuration from the frame processing unit 44 in Figure 1, in that it includes a selector 51C.

[0126] For example, the post-processing unit 45C can be configured with a neural network that analyzes the framed event data output from the frame processing unit 44C, and the results of the neural network's analysis of the event data can be fed back to the selector 51C. The selector 51C can then perform data drop processing according to the analysis results of the event data supplied by the post-processing unit 45C, in addition to the drop rate set by the statistical calculation unit 54.

[0127] This allows the data processing system 11C to further reduce the processing load on event data.

[0128] Figure 18 is a block diagram showing an example configuration of the first variation of the data processing system 11C. In the data processing system 11C-1 shown in Figure 18, components common to the data processing system 11C in Figure 17 are denoted by the same reference numerals, and their detailed explanations are omitted.

[0129] As shown in Figure 18, the data processing system 11C-1 is configured to include an event sensor 21 and a post-processor 22C-1, with the event sensor 21 having the same configuration as in Figure 1. The data processing system 11C-1 has the same configuration as the data processing system 11C in Figure 17, in that the post-processor 22C-1 includes a receiving unit 41, a decoding unit 42, and a pre-processing unit 43.

[0130] Furthermore, the data processing system 11C-1 differs from the frame processing unit 44 in Figure 1 in that the post-processor 22C includes a frame processing unit 44C-1 and a post-processing unit 45C-1. The frame processing unit 44C-1 has the same configuration as the frame processing unit 44 in Figure 1 in that it includes a selector 51, an adder 52, an intermediate data holding unit 53, a statistical calculation unit 54, and a buffer 56, but differs from the frame processing unit 44 in Figure 1 in that it includes a framing unit 55C.

[0131] For example, the post-processing unit 45C-1 can be configured with a neural network that analyzes event data, and the results of the neural network's analysis of the event data can be fed back to the framing unit 55C. The framing unit 55C can then use, for example, the results of the event data analysis supplied from the post-processing unit 45C-1 to select a framing method.

[0132] Thus, in the data processing system 11C-1, the framing unit 55C can utilize the results of the post-processing unit 45C-1's analysis of event data to determine which framing method to select.

[0133] <Fifth Configuration Example of a Data Processing System> Referring to Figure 19, a fifth configuration example of a data processing system to which this technology is applied will be described.

[0134] Figure 19 is a block diagram showing an example configuration of a fifth embodiment of a data processing system to which this technology is applied. In the data processing system 11D shown in Figure 19, components common to the data processing system 11 in Figure 1 are denoted by the same reference numerals, and their detailed descriptions are omitted.

[0135] As shown in Figure 19, the data processing system 11D is configured to include an event sensor 21D and a post-processor 22D. The data processing system 11D has the same configuration as the data processing system 11 in Figure 1, in that the event sensor 21D comprises an encoding unit 32 and a transmission unit 33, and the post-processor 22D comprises a receiving unit 41, a decoding unit 42, a pre-processing unit 43, and a post-processing unit 45.

[0136] The data processing system 11D differs from the data processing system 11 in Figure 1 in that the event sensor 21D includes the EVS pixel 31D, the receiving unit 93, and the communication control unit 94, while the post-processor 22D includes the frame processing unit 44D, the communication control unit 91, and the transmission unit 92. The frame processing unit 44D has the same configuration as the frame processing unit 44 in Figure 1, including the selector 51, the adder 52, the intermediate data holding unit 53, the framing unit 55, and the buffer 56, but differs from the frame processing unit 44 in Figure 1 in that it includes the statistical calculation unit 54D.

[0137] The statistical calculation unit 54D obtains a value obtained by multiplying the drop rate set in the selector 51 by a predetermined coefficient, for example, as data output amount adjustment information, and supplies it to the communication control unit 91.

[0138] The communication control unit 91 performs communication control to transmit data output amount adjustment information supplied from the statistical calculation unit 54D to the event sensor 21D via the transmission unit 92 and the reception unit 93.

[0139] The transmitting unit 92 transmits data output amount adjustment information in accordance with the communication control by the communication control unit 91.

[0140] The receiving unit 93 receives data output amount adjustment information in accordance with the communication control by the communication control unit 94 and supplies it to the communication control unit 94.

[0141] The communication control unit 94 performs communication control to receive data output amount adjustment information transmitted from the post-processor 22D via the transmission unit 92 and the reception unit 93, and sets the data output amount adjustment information in the EVS pixel 31D.

[0142] The EVS pixel 31D adjusts the output amount of event data by controlling a threshold used to determine whether or not an event has occurred based on a comparison with a change in light brightness, according to the data output amount adjustment information set by the communication control unit 94. The EVS pixel 31D may also use the data output amount adjustment information set by the communication control unit 94 to control the noise coefficient.

[0143] The data processing system 11D configured in this way can be equipped with a function to perform interrupt feedback control from the post-processor 22D to the event sensor 21D. For example, in the data processing system 11D, if the drop rate in the selector 51 increases as the event data increases, the output amount of event data output from the EVS pixel 31D is adjusted to decrease, and if the drop rate in the selector 51 decreases as the event data decreases, the output amount of event data output from the EVS pixel 31D is adjusted to increase. In this way, in the data processing system 11D, feedback control is performed by the post-processor 22D performing interrupt communication with the event sensor 21D.

[0144] Furthermore, the event sensor 21D can perform feedback control using data output amount adjustment information for the entire sensor surface, or for each row or region of the sensor surface.

[0145] <Effects of Statistical Calculation> Referring to Figure 20, the effects of the selector 51 performing data drop processing according to the drop rate determined by statistical calculations performed by the statistical calculation unit 54 will be explained.

[0146] Figure 20A shows an example of a VFI (Video Frame Interpolation) neural network. VFI is an image processing technique that takes low-frame-rate RGB video as input and generates and outputs high-frame-rate RGB video. In this example, in addition to low-frame-rate RGB video (e.g., 15 FPS RGB image data), a high-frame-rate event stream (framed event data) is input to the VFI neural network, and a high-frame-rate RGB video (e.g., 60 FPS RGB image data) is output.

[0147] Figure 20B shows experimental results comparing high-frame-rate RGB video generated by a VFI neural network with ground-truth RGB video, and calculating the PSNR (Peak Signal to Noise Ratio) for each frame of the video. In Figure 20B, the horizontal axis represents the frame number of the video output from the VFI neural network, and the vertical axis represents the evaluation index, PSNR [dB]. A higher PSNR indicates that the high-frame-rate RGB video generated by the VFI neural network is closer to the ground-truth RGB video, and therefore yields better results.

[0148] Furthermore, in Figure 20B, the "No Selection" graph shows the PSNR of a high-frame-rate RGB video generated using all event data without dropping any. The "Selected Based on Next Time Event Rate Prediction" graph shows the PSNR of a high-frame-rate RGB video generated using event data from which data drop processing has been performed according to the drop rate determined by the statistical calculation described above to reduce data bandwidth (for example, adjusted to reduce by 25%). The "Selected According to Memory Capacity" graph shows the PSNR of a high-frame-rate RGB video generated using event data from which event data has been randomly dropped according to the memory capacity value of the intermediate data holding unit 53 to reduce data bandwidth (for example, adjusted to reduce by 25%), rather than using the drop rate determined by the statistical calculation described above.

[0149] As shown in the graphs in Figure 20, when data drop processing is performed on the event stream input to the VFI neural network according to the drop rate determined by the statistical calculation described above to reduce data bandwidth, it was confirmed that the same level of accuracy as when using all uncropped data (i.e., high bandwidth) can be maintained. On the other hand, when the same amount of data is dropped randomly instead of using the drop rate determined by the statistical calculation described above, it was confirmed that the same level of accuracy as when using all uncropped data cannot be maintained.

[0150] Therefore, by having the selector 51 perform data drop processing according to the drop rate determined by statistical calculations performed by the statistical calculation unit 54, accuracy can be maintained and bandwidth can be reduced.

[0151] Furthermore, the framed event data generated by this technology can be applied to RGB image codecs and VFIs as described above, as well as to various other image processing tasks such as recognition and detection.

[0152] For example, this technology can be applied to enhancing motion images, including VFI, and to fusing RGB and EVS images, including HDR (High Dynamic Range) image formation and image processing such as deblurring. Furthermore, this technology can be applied, for example, to the compression processing unit 82 in Figure 16, to the process of selecting keyframes using framed event data or the results of post-processing applied to framed event data. This technology can also be applied to techniques such as pose estimation, self-localization estimation, SLAM (Simultaneous Localization and Mapping) which performs self-localization estimation and environment mapping simultaneously, and eye tracking.

[0153] <Examples of Configuration Combinations> The technology can also take the following configurations: (1) A data processing device comprising: a first data drop processing unit that performs data drop processing to select event data output from an EVS (Event Based Vision Sensor) pixel for each predetermined unit based on a drop rate indicating the frequency of dropping the event data; an addition unit that generates intermediate data by performing an addition operation according to the event data selected by the first data drop processing unit; a statistical calculation unit that determines an event occurrence rate indicating the rate at which events detected by the EVS pixel occur by statistical calculation using the intermediate data, and sets the drop rate according to the event occurrence rate to the first data drop processing unit; and a framing unit that frames the event data using the intermediate data at predetermined intervals. (2) The data processing device according to (1) above, wherein the statistical calculation unit analyzes the density of event data occurring within a predetermined period and determines the drop rate by using the intermediate data having one or more channels that are updated by an addition operation by the addition unit at predetermined time intervals. (3) The data processing device according to (1) or (2) above, further comprising a second data drop processing unit that performs data drop processing for selecting event data for each predetermined unit based on header information attached to the event data packet, prior to the first data drop processing unit that performs data drop processing based on the drop rate according to the event occurrence rate. (4) The data processing device according to (3) above, wherein the second data drop processing unit performs data drop processing based on one or more pieces of information, which include information indicating the row or region number in the event data frame, or metadata attached to the event data frame.(5) The data processing device according to (4) above, wherein RGB image data output from an RGB pixel is supplied along with the event data, the second data drop processing unit refers to an exposure period flag indicating whether the RGB pixel is in the exposure period, determines whether the event data in a predetermined unit is within or outside the exposure period of the RGB pixel, and executes the data drop processing based on the importance of whether the RGB pixel is within or outside the exposure period. (6) The data processing device according to (4) above, wherein the second data drop processing unit uses an error code included in the metadata indicating that the output of the event data from the EVS pixel failed to execute the data drop processing. (7) The data processing device according to any one of (1) to (6) above, wherein if the event data is supplied in a compressed state, the metadata attached to the frame of the event data includes the compression mode of the event data, the statistical calculation unit estimates the density of the event data decoded from the compressed state according to the compression mode of the event data included in the metadata, and uses the estimation result as the drop rate to be set in the first data drop processing unit. (8) A data processing device according to any one of (1) to (7) above, wherein at least one of the following is used as metadata attached to the frame of event data output from an event sensor in which the EVS pixels are arranged in an array on the sensor surface: the frame rate of the event data and a flag representing the density of events according to the event data. (9) A data processing device according to any one of (1) to (8) above, wherein when RGB image data encoded together with the event data is supplied and compression processing is performed on the RGB image data, the RGB image data is divided into fixed intervals in the spatial direction, or the RGB image data is time-divided into intervals according to a predetermined rule, and the first data drop processing unit performs the data drop processing according to the division region information indicating each of the divided regions of the RGB image data, in addition to the drop rate.(10) A data processing device according to any one of (1) to (9) above, wherein the event data framed by the framing unit is post-processed by a neural network that analyzes the event data, and the first data drop processing unit performs the data drop processing according to the analysis results by the neural network in addition to the drop rate. (11) A data processing device according to any one of (1) to (10) above, wherein the statistical calculation unit obtains a value obtained by multiplying the drop rate set in the first data drop processing unit by a predetermined coefficient as data output amount adjustment information, and performs interrupt feedback control to the EVS pixels using the data output amount adjustment information. (12) A data processing system comprising: an event sensor in which EVS (Event Based Vision Sensor) pixels are arranged in an array on the sensor surface; a first data drop processing unit that performs data drop processing to select event data output from the EVS pixels in predetermined units based on a drop rate indicating the frequency of dropping the event data; an addition calculation unit that generates intermediate data by performing an addition calculation according to the event data selected by the first data drop processing unit; a statistical calculation unit that determines an event occurrence rate indicating the rate at which events detected by the EVS pixels occur by statistical calculation using the intermediate data, and sets the drop rate according to the event occurrence rate to the first data drop processing unit; and a framing unit that frames the event data using the intermediate data at predetermined intervals.

[0154] It should be noted that this embodiment is not limited to the embodiment described above, and various modifications are possible without departing from the spirit of this disclosure. Furthermore, the effects described herein are merely illustrative and not limiting, and other effects may also exist.

[0155] 11 Data processing system, 21 Event sensor, 22 Post-processor, 31 EVS pixel, 32 Encoding unit, 33 Transmission unit, 34 RGB pixel, 41 Receiving unit, 42 Decoding unit, 43 Pre-processing unit, 44 Frame processing unit, 45 Post-processing unit, 51 Selector, 52 Adder, 53 Intermediate data holding unit, 54 Statistical calculation unit, 55 Framing unit, 56 Buffer, 57 Memory, 58 System memory, 61 Communication control unit, 62 Receiving unit, 63 Sensor data processing unit, 71 Selector, 72 Buffer, 81 Pre-processing unit, 82 Compression processing unit, 83 Region division unit, 84 Compression unit, 91 Communication control unit, 92 Transmission unit, 93 Receiving unit, 94 Communication control unit

Claims

A first data drop processing unit performs a data drop process that selects event data output from an EVS (Event Based Vision Sensor) pixel for each predetermined unit based on a drop rate indicating the frequency of dropping the event data. An addition unit that generates intermediate data by performing an addition operation corresponding to the event data selected by the first data drop processing unit, A statistical calculation unit determines the event occurrence rate, which indicates the rate at which events detected by the EVS pixels occur, by statistical calculation using the intermediate data, and sets the drop rate according to the event occurrence rate in the first data drop processing unit. A framing unit that frames the event data using the intermediate data at predetermined intervals. A data processing device equipped with the following features. 　 The statistical calculation unit uses the intermediate data, which has one or more channels and is updated at predetermined time intervals by addition calculations performed by the addition calculation unit, to analyze the density of the event data occurring within a predetermined period and determine the drop rate. The data processing device according to claim 1. 　 Prior to the first data drop processing unit which executes the data drop processing based on the drop rate according to the event occurrence rate, a second data drop processing unit which executes data drop processing for each predetermined unit, selecting the event data based on the header information attached to the event data packet. The data processing apparatus according to claim 1, further comprising: 　 The second data drop processing unit executes the data drop process based on one or more pieces of information, which include information indicating the row or region number in the event data frame, or metadata attached to the event data frame. The data processing device according to claim 3. 　 Along with the event data, RGB image data output from the RGB pixels is supplied. The second data drop processing unit refers to an exposure period flag indicating whether the RGB pixel is in the exposure period, determines whether the event data in a predetermined unit is within or outside the exposure period of the RGB pixel, and executes the data drop processing based on the importance of whether the RGB pixel is within or outside the exposure period. The data processing device according to claim 4. 　 The second data drop processing unit executes the data drop process using an error code included in the metadata that indicates that the output of the event data from the EVS pixel failed. The data processing device according to claim 4. 　 If the event data is supplied in a compressed state, the metadata attached to the frame of the event data includes the compression mode of the event data. The statistical calculation unit estimates the density of the event data decoded from the compressed state according to the compression mode of the event data included in the metadata, and uses the estimation result as the drop rate to be set in the first data drop processing unit. The data processing device according to claim 1. 　 At least one of the following is used as metadata attached to the frame of event data output from an event sensor in which the EVS pixels are arranged in an array on the sensor surface: the frame rate of the event data and a flag representing the density of events according to the event data. The data processing device according to claim 1. 　 When RGB image data encoded together with the event data is supplied and compression processing is performed on the RGB image data, the RGB image data is divided into fixed intervals in the spatial direction, or the RGB image data is time-divided into intervals according to a predetermined rule. The first data drop processing unit executes the data drop process according to the drop rate and the divided region information indicating each region into which the RGB image data has been divided. The data processing device according to claim 1. 　 Post-processing of the event data is performed on the event data by a neural network that analyzes the event data framed by the framing unit. The first data drop processing unit executes the data drop process according to the analysis results by the neural network, in addition to the drop rate. The data processing device according to claim 1. 　 The statistical calculation unit obtains a value as data output amount adjustment information by multiplying the drop rate set in the first data drop processing unit by a predetermined coefficient, and uses this data output amount adjustment information to perform interrupt feedback control for the EVS pixels. The data processing device according to claim 1. An event sensor in which EVS (Event Based Vision Sensor) pixels are arranged in an array on the sensor surface, A first data drop processing unit performs a data drop process that selects event data output from the EVS pixels for each predetermined unit based on a drop rate indicating the frequency of dropping the event data, An addition unit that generates intermediate data by performing an addition operation corresponding to the event data selected by the first data drop processing unit, A statistical calculation unit determines the event occurrence rate, which indicates the rate at which events detected by the EVS pixels occur, by statistical calculation using the intermediate data, and sets the drop rate according to the event occurrence rate in the first data drop processing unit. A framing unit that frames the event data using the intermediate data at predetermined intervals. A data processing device having A data processing system equipped with the following features.

Images