Signal processing circuit, signal processing method, and program

Abstract

Provided is a signal processing circuit for processing event signals generated by an event-based vision sensor (EVS), the signal processing circuit comprising a memory for storing a program code, and a processor for executing an operation according to the program code. The operation includes: detecting lines formed by a set of positions of the event signals in blocks obtained by dividing a detection area of the EVS, the event signals being generated in the blocks; and generating a group including the lines if the lines detected in the blocks adjacent to each other satisfy a prescribed relationship.

Description

Signal processing circuit, signal processing method, and program

[0001] The present invention relates to a signal processing circuit, a signal processing method, and a program.

[0002] An event-based vision sensor (EVS: Event-based Vision Sensor) in which pixels that detect changes in the intensity of incident light generate signals asynchronously over time is known. The EVS is also called an EDS (Event Driven Sensor), an event camera, or a DVS (Dynamic Vision Sensor), and includes a sensor array composed of sensors including light receiving elements. When the EVS 110 detects a change in the intensity of light incident on the sensor, more specifically, a change in the luminance of the object surface, it generates an event signal including a time stamp, identification information of the sensor, and information on the polarity of the luminance change. The EVS is advantageous in that it can operate at low power and high speed compared to frame-type vision sensors that scan all pixels at a predetermined period, specifically image sensors such as CCDs and CMOSs. Technologies related to such EVSs are described in, for example, Patent Document 1, Patent Document 2, and Patent Document 3. Patent Document 3 describes detecting line segments and curves from event signals generated in blocks obtained by dividing the detection area of the EVS.

[0003] Japanese Patent Publication No. 2014-535098, Japanese Unexamined Patent Application Publication No. 2018-85725, International Publication No. 2023 / 209843

[0004] By treating the event signals generated by the EVS as line segments and curves detected for each block as in Patent Document 3 above, the event signals can be processed more quickly, and the high-speed operation of the EVS can be fully utilized. On the other hand, since the edges of the object to be detected by the event signals exist over a number of blocks, a large amount of resources are still required for calculations for clustering the lines detected in each block as representing a common edge.

[0005] Therefore, the present invention aims to provide a signal processing circuit, a signal processing method, and a program that can further speed up processing when the target of processing is a line detected from an event signal generated by an EVS.

[0006] According to one aspect of the present invention, a signal processing circuit is provided for processing event signals generated by an event-based vision sensor (EVS), comprising a memory for storing program code and a processor for performing operations according to the program code, wherein the operation includes detecting lines formed by a set of positions of the event signals generated in blocks obtained by dividing the detection area of ​​the EVS, and generating groups containing the lines when the lines detected in adjacent blocks satisfy a predetermined relationship.

[0007] According to another aspect of the present invention, a signal processing method is provided for processing event signals generated by an event-based vision sensor (EVS), the method comprising: detecting lines formed by a set of positions of the event signals generated in blocks obtained by dividing the detection area of ​​the EVS, and generating groups containing the lines when the lines detected in adjacent blocks satisfy a predetermined relationship, by an operation performed by a processor in accordance with program code stored in memory.

[0008] In yet another aspect of the present invention, a program is provided for processing event signals generated by an event-based vision sensor (EVS), wherein the operation performed by a processor according to the program includes detecting lines formed by a set of positions of the event signals within blocks obtained by dividing the detection area of ​​the EVS, and generating groups of lines if the lines detected in adjacent blocks satisfy a predetermined relationship.

[0009] This figure schematically shows the configuration of a signal processing circuit according to one embodiment of the present invention. This is a schematic diagram showing examples of blocks and events. This figure illustrates an example of straight line detection in the example shown in Figure 1. This figure conceptually explains the process of grouping detected lines. This figure conceptually explains the process of grouping detected lines. This figure specifically explains the procedure for grouping lines. This figure specifically explains the procedure for grouping lines. This figure specifically explains the procedure for grouping lines. This figure illustrates the relationship between the direction of lines detected in a block and the search target during grouping. This figure shows an example of tracking edge movement in time series using a group of detected lines (LSG). This figure shows an example of calculating disparity using a group of detected lines (LSG). This figure illustrates an example of curve detection in the example shown in Figure 1.

[0010] Figure 1 is a schematic diagram showing the configuration of a signal processing circuit according to one embodiment of the present invention. The signal processing circuit 200, which processes event signals generated by the event-based vision sensor (EVS) 100, is composed of a processor or processing circuit such as a CPU (Central Processing Unit), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), and / or an FPGA (Field-Programmable Gate Array). The signal processing circuit 200 includes a memory 210 composed of various ROMs (Read Only Memory) and / or RAMs (Random Access Memory). The signal processing circuit 200 performs operations as described below according to the program code stored in the memory 210. Note that part or all of the post-processing 226 may be performed by the signal processing circuit 200, or it may be performed by a device or circuit other than the signal processing circuit 200.

[0011] Event signals generated by the EVS 100 are temporarily held in buffer 221 and distributed by splitter 222 to block event buffers (BEBs) 223A, 223B, ... (hereinafter collectively referred to as BEB223). Here, splitter 222 distributes event signals generated in each of the grid-like blocks 310A, 310B, ... (hereinafter collectively referred to as block 310) that divide the detection area of ​​the EVS 100, as shown in Figure 2, to the corresponding BEB223A, 223B, .... BEB223 is predefined as a buffer that temporarily holds the event signals corresponding to each of the grid-like blocks 310 that divide the detection area of ​​the EVS 100. If the settings of block 310 are dynamically changed, as in the example described later, the definition of BEB223 is also dynamically changed in accordance with the settings of block 310. The event signal includes information such as the position x, y within the detection area and the time t at which it was generated. The splitter 222 determines which BEB 223 to distribute the event signal to by referring to information indicating the positions x and y. As shown in the example described later, the splitter 222 may duplicate the event signal and distribute it to two or more BEB 223s.

[0012] The BEB223 holds the event signals generated in each block 310. When an event signal is assigned to any of the BEB223A, 223B, ... the line detector 224 detects a line, i.e., a straight line or curve, from the set of position x, y of the event signal held in that BEB223. Here, generally, straight lines and curves do not necessarily have endpoints, but in this embodiment, the straight lines or curves detected within a block 310 have endpoints at the boundary of the periphery of the block 310. Therefore, the straight lines detected by the line detector 224 can also be called line segments. For example, if an event occurs due to the movement of an object's edge within a certain block 310, the set of position x, y of the event signal forms a straight line or curve. The edges of an object are not necessarily straight lines, but by setting an appropriate size for the grid-like blocks 310, the edges of an object, which are curves, can also be represented as a set of straight lines. Alternatively, the line detector 224 may be capable of detecting curves, as in the example described later. The line detector 224 detects lines within each block using methods such as the Hough transform or a technique that minimizes the sum of the distances from the position of each event signal to the straight line.

[0013] The line detector 224 outputs block line parameters (BLPs) 225A, 225B, ... (hereinafter collectively referred to as BLP225) that indicate the detected line. BLP225A is information indicating the line detected by the line detector 224 from the event signal generated in block 310A and held in BEB323A, and the same applies to BLP225B and subsequent parameters. Note that BLP225A, 225B, ... are not necessarily output synchronously, but are output asynchronously by the processing that the line detector 224 executes when an event signal is distributed to any of the BEB223, as described above. The outputted BLP225 is used in the post-process 226 as information indicating the detection result of the EVS100. Post-processing 226 may include, for example, tracking the movement of object edges over time, as described later, or estimating the depth of an object from the parallax between two EVS 100s with different viewpoints. However, it is not limited to these examples, and various processes using line information detected from event signals may be performed.

[0014] Figure 3 is a diagram illustrating an example of line detection in the example shown in Figure 1. As described above, in this embodiment, when an event signal is distributed to a block event buffer (BEB) 223 corresponding to each of the grid-like blocks 310 that divide the detection area of ​​the EVS 100, the line detector 224 detects a line from the set of positions x and y of the event signals. In the example shown in Figure 3, the process of detecting a line when there are schematically five event signals in the BEB 223 (the actual number of event signals may be more) is shown. The event signals E1 to E5 each contain information such as positions x1 to x5, y1 to y5 within the detection area and the time t1 to t5 when they were generated. Since positions x1 to x5 and y1 to y5 all indicate positions within the block 310 being processed, if the size of the block 310 (16 pixels x 16 pixels in the illustrated example) is appropriate, it is not necessary to sort the event information in the spatial and temporal directions, and the line detector 224 can mathematically detect a straight line from the positions x1 to x5 and y1 to y5 of the event signals E1 to E5 held in the BEB223.

[0015] When detecting lines in the line detector 224, for example, an upper limit may be set on the number of event signals held in the BEB 223, and the oldest event signal may be deleted when a new event signal is distributed using a FIFO (First In, First Out) method. Alternatively, a threshold may be set for the time t of the event signal, based on the difference between the processing time or the time t of the latest event signal, and event signals whose difference exceeds the threshold may not be used by the line detector 224 for line detection or may be deleted from the BEB 223. In this way, old event signals may be deleted or conditions may be set regarding the time t of the event signals, but even in these cases, it is not necessary to sort the event signals in the time direction when detecting lines.

[0016] Furthermore, if an event signal at the same position x, y as an event signal already held in BEB223 is newly assigned, the time t of the held event signal may be updated with the time t of the newly assigned event signal to avoid duplication of event signals at the same position x, y within BEB223. In this case, by assuming that event signals at the same position x, y do not overlap, calculations for detecting lines, for example, can be sped up. Alternatively, multiple event signals at the same position x, y but with different times t may be held in BEB223.

[0017] In the example shown in Figure 3 above, the line detector 224 outputs block line parameters (BLP) including angle (θ), distance (r), latest event time (Tnew), and event duration (Duration). The angle (θ) indicates the inclination of the line with respect to the x-axis, and the distance (r) indicates the distance from the upper left corner of the block to the line (length of the perpendicular). However, this is not limited to this example, and any line can be identified according to other known methods (for example, with two parameters indicating the inclination of the line and its relative position to the block). The latest event time (Tnew) is the time corresponding to the most recent event signal used to detect the line. The latest event time (Tnew) may be determined, for example, by extracting the most recent event signal from the times t1 to t5 of the event signals E1 to E5 used to detect the line (in the example in Figure 3, Tnew = t5). Alternatively, since line detection is performed when the latest event signal is distributed to BEB232, the latest event time (Tnew) may be set to the time when the line detector 224 outputs BLP225, or the time when the post-process 226 receives BLP225, without referring to the times of event signals E1 to E5.

[0018] The event duration is the difference between the first and last time points (t1-t5) of the event signals E1-E5 used to detect the line (i.e., Duration = t5-t1 in the example in Figure 3). From the event duration information, it is possible to determine how long the line was detected based on the event signals. For example, if the event duration is extremely long, many event signals detected as noise are used in the line detection, and the reliability of the detected line may be judged as low in post-processing 226. The line detector 224 may also output the variance Var[t] on the time series of the time the event signal was generated. In this case, post-processing 226 can judge that the reliability of the detected line is high if the variance Var[t] is small even if the event duration is long. Conversely, if the event duration is long and the variance Var[t] is large, the reliability of the detected line can be judged as low.

[0019] Figures 4 and 5 are diagrams that conceptually illustrate the process of grouping detected lines. As described above, in this embodiment, lines formed by a set of positions within the event signal block 310 are detected. Since the lines are not associated at the time of detection, for example, if the edge of an object spans multiple blocks 310, multiple lines representing the edge will be detected. As shown in Figure 4, if a group containing lines representing the same edge, specifically an LSG (Line Segment Group), can be generated, then processes such as tracking the movement of an edge in a time series or detecting the parallax of edges detected by two EVS 100 with different viewpoints can be sped up, as in the example described later. In many cases, if the processing is sped up, power consumption is also reduced. As shown in Figure 5, it is not necessarily the case that all lines L representing a series of edges are included in a single LSG; for example, the edges may be divided and grouped into several LSGs. Each of the multiple groups is assigned an ID. In the illustrated example, five LSGs from ID=0 to ID=4 are generated.

[0020] Figures 6A, 6B, and 6C are diagrams that specifically illustrate the procedure for grouping lines. In the example shown in Figure 6A, lines L1 to L7 are detected in each block 310 within a certain portion of the EVS detection area. To group the detected lines, the blocks 310 are searched from left to right and from top to bottom within the detection area. In the search of the top block, no lines are detected, so the grouping procedure is not performed. In the search of the second block, the grouping procedure is performed in the second block from the left, where line L1 is detected. In the grouping procedure, lines detected in the second block adjacent to the first block where line L1 was detected are searched. In the illustrated example, since the direction of line L1 detected in the first block is left-right, the blocks adjacent to the left and right of the block containing line L1 are searched. No lines are detected in the block adjacent to the left, but line L2 is detected in the block adjacent to the right, so this block becomes the second block mentioned above. Once line L2, detected in the second block, is identified, it is determined whether the distance between the respective endpoints of line L1 and line L2 is less than or equal to a threshold. If the distance is less than or equal to the threshold, an LSG containing line L1 and line L2 is generated. Next, the search procedure for line L2, which was the last line added to the LSG, is performed in the same way as for line L1. Since the direction of line L2 is also left-right, the blocks adjacent to the left and right of the block containing line L2 are searched. The block adjacent to the left is the block where line L1, already included in the LSG, was detected, but the block adjacent to the right has line L3, which is not yet included in the LSG, detected, so it is determined whether the distance between the respective endpoints of line L2 and line L3 is less than or equal to a threshold. If the distance is less than or equal to the threshold, line L3 is added to the LSG containing lines L1 and L2.

[0021] Figure 7 is a diagram illustrating the relationship between the direction of a line detected in a block and the search targets during grouping. The direction of a line detected in block 310 can be determined, for example, from the angle (θ) included in the BLP in the example shown in Figure 3. In the illustrated example, if the direction of the line detected in block 310 is the left-right direction a, then blocks 310-1 and 310-2 adjacent to the left and right are searched, and if the direction of the line is the up-down direction b, then blocks 310-3 and 310-4 adjacent to the top and bottom of the block are searched. If the direction of the line is neither left-right nor up-down but diagonal direction c, then all adjacent blocks 310-1 to 310-8 are searched. Regarding diagonal direction c, for example, if the direction of the line is a downward sloping direction as shown, there is a possibility that lines to be grouped exist in block 310-1 on the left and block 310-4 below, or there is a possibility that lines to be grouped exist in block 310-3 above and block 310-2 on the right. Alternatively, as shown in the illustrated example, there may be lines that should be grouped in blocks 310-5 to 310-8, where the edges are not adjacent but are diagonally adjacent. The same applies when the direction of the line is a diagonal downward sloping direction. Therefore, if the direction of the line detected in block 310 is diagonal, blocks adjacent to block 310 in all directions are searched. In either case, blocks where no line has been detected, or blocks where a line already included in the LSG has been detected, are excluded from the search. In other examples, if no line is detected in a block adjacent to the direction of the line detected in a block, or if the detected line does not satisfy the relationship for grouping, the search range may be extended to blocks adjacent in all directions.

[0022] Continuing to refer to Figure 6A, since the direction of line L3 is neither up / down nor left / right, blocks adjacent to the block of line L3 in all directions are searched. Excluding the block of line L2, which is already included in the LSG, line L4 is detected in the block adjacent to the lower side of the block of line L3, so it is determined whether the distance between the respective endpoints of line L3 and line L4 is less than or equal to a threshold. If the distance is less than or equal to the threshold, line L4 is added to the LSG which includes lines L1 to L3. Similarly, the next line is searched for for line L4, and line L5, detected in the block adjacent to the right side of the block of line L4, is added to the LSG. Since the direction of line L5 is left / right, blocks adjacent to the left and right sides of the block of line L5 are searched. The block adjacent to the left is the block where line L4, which is already included in the LSG, was detected, but line L6, which is not yet included in the LSG, is detected in the block adjacent to the right, so it is determined whether the distance between the respective endpoints of line L5 and line L6 is less than or equal to a threshold. The endpoint of line L6 does not lie on an edge common to the block of line L5, and the distance between the respective endpoints of lines L5 and L6 exceeds the threshold. In this case, line L6 is not added to the LSG containing line L5, and the procedure for adding lines to the LSGs (0-1, 0-2, 0-3, 0-4, 0-5) containing lines L1 to L5 is terminated.

[0023] On the other hand, line L6, which was not added to the LSG containing lines L1 to L5, may be included in another LSG, so following the above procedure, the grouping procedure is executed again from the block in which line L6 was detected. Since the direction of line L6 is vertical, blocks adjacent to the upper and lower sides of the block containing line L6 are searched. No line is detected in the block adjacent to the lower side, but line L7 is detected in the block adjacent to the upper side, so it is determined whether the distance between the respective endpoints of lines L6 and L7 is less than or equal to a threshold. If the distance is less than or equal to the threshold, an LSG containing lines L6 and L7 is generated. Similarly, the next line is searched for for line L7, but since no lines other than line L6 are detected in the block adjacent to line L7, the procedure for adding lines to the LSG (1-1, 1-2) containing lines L6 and L7 is also completed. Note that for the block containing line L6, in addition to line L7 in the block adjacent to the upper side, it is also possible that a line has been detected in the block adjacent to the lower side. In that case, after performing the grouping procedure for line L7 as described above, perform the same grouping procedure for the line detected in the lower block on the opposite side.

[0024] In the example above, when determining whether lines detected in adjacent blocks are included in the same LSG, the condition was that the distance between each endpoint of the lines is less than or equal to a threshold. However, the condition for including lines in the same LSG may be that the lines satisfy other predetermined relationships. For example, the condition may be that the lines are in the same direction, or that the difference in the angle of their directions is less than or equal to a threshold.

[0025] Following the above procedure, as shown in Figure 6B, LSG_ID=0 is assigned to the blocks where lines L1 to L5 are detected, and LSG_ID=1 is assigned to the blocks where lines L6 and L7 are detected. By assigning an LSG_ID to each block, the figure represented by the LSG can be identified without referring to the BLP of the lines detected in each block. Also, following the above procedure, as shown in Figure 6C, a flag d is set to indicate that the block has been explored in the top row block, the leftmost block of the second row, and the blocks where lines L1 to L7 are detected. Blocks with this flag d set are skipped in subsequent searches within the same time frame. Therefore, the next search resumes from the fifth block from the left in the second row, which does not have the flag d set. Since no lines were detected in this block, the grouping procedure is not performed. Next, the block with line L7, which has the flag d set, is skipped, and the search is performed from the block adjacent to its right. In the third stage, the search is performed starting from the leftmost block, but blocks on lines L4 to L6 with flag d set are skipped, and the search resumes from the block adjacent to the right of the block on line L6. Since the grouping procedure described above is not necessarily performed in a fixed order such as from left to right or from top to bottom, setting flag d to indicate that a block has already been searched prevents duplicate processing from being performed on blocks that have already been assigned an LSG ID.

[0026] Figure 8 shows an example of tracking edge movement over time using detected line groups (LSGs). The LSGs detected at time frame N (IDs 0-4) and at time frame N+1 (IDs 0'-4') are shown. By matching the shapes represented by the lines contained within these LSGs using methods such as convolution matching or semi-block matching, the LSGs can be associated with each other. By detecting the movement and rotation of the shapes represented by the lines contained within each associated LSG, the movement of the edges of the object represented by those shapes can be tracked over time.

[0027] Figure 9 shows an example of calculating disparity using detected line groups (LSGs). The LSG generated from event signals detected by one of two EVSs positioned at different viewpoints in the left-right direction is shown as (A), and the disparity calculated by comparing it with the LSG generated from the event signals detected by the other is shown as (B). The LSGs generated from event signals detected by each EVS are associated by matching the shapes represented by the lines contained within each LSG, similar to the example described above with reference to Figure 8. The distance between the shapes associated by this matching, more specifically the direction of the viewpoint displacement, i.e., the distance between the shapes in the left-right direction in the illustrated example, is the disparity. If we can assume the shape is a straight line, the disparity of the middle part can be calculated from the disparity at both ends of the straight line without measuring the disparity of each part of the shape. From the disparity, the depth of the object can be estimated. Parallax is difficult to detect for lines extending in the same direction as the parallax, but if, for example, the end of one such line can be connected to the end of another line extending in a different direction, then the parallax of the connected line can be used to identify the parallax of the line extending in the left-right direction and estimate its depth.

[0028] In the process described above with reference to Figures 8 and 9, it is possible to identify the edges of an object by calculating the movement and rotation of lines from the BLP for each block, as shown in the example in Figure 3, and matching lines that have undergone similar movement and rotation. However, because there are many blocks, a lot of resources are required for the matching calculation. When using LSG as the unit, even if the BLP of each line included in the LSG is referenced to identify the shape of the figure, the matching calculation is performed on an LSG basis, thus reducing the resources required for calculation and speeding up the process.

[0029] Figure 10 is a diagram illustrating an example of curve detection in the example shown in Figure 1. In the illustrated example, the line detector 224 shown in Figure 1 detects a circular arc from the set of position x, y of the event signal E. In this case, the line detector 224 outputs block line parameters (BLP) including the position of the circle's center (pos), radius (r), starting angle (θs), ending angle (θe), the time of the latest event (Tnew), and the event duration (Duration). Thus, the line detected by the line detector 224 is not necessarily limited to a straight line, but may also be a curve such as a circular arc, an elliptical arc, or a parabola.

[0030] 100...EVS, 200...Signal processing circuit, 210...Memory, 221...Buffer, 222...Splitter, 223...BEB, 224...Line detector, 225...BLP, 226...Post-processing, 310...Block.

Claims

1. A signal processing circuit for processing event signals generated by an event-based vision sensor (EVS), comprising a memory for storing program code and a processor for performing operations according to the program code, wherein the operation includes detecting lines formed by a set of positions of the event signals within blocks obtained by dividing the detection area of ​​the EVS, and generating groups containing the lines when the lines detected in adjacent blocks satisfy a predetermined relationship.

2. The signal processing circuit according to claim 1, wherein the predetermined relationship includes the distance between each endpoint of the line being less than or equal to a threshold.

3. The signal processing circuit according to claim 1, wherein generating the group includes determining whether a line detected in a second block adjacent to the first block in any direction satisfies a predetermined relationship with the line detected in the first block, depending on the direction of the line detected in the first block.

4. The signal processing circuit according to claim 1, wherein the operation further includes performing a matching between a first figure represented by lines included in the group generated in a first time frame and a second figure represented by lines included in the group generated in a second time frame different from the first time frame, and detecting movement and rotation between the first and second figures associated with each other by the matching.

5. The signal processing circuit according to claim 1, wherein the operation further comprises performing a matching between a first figure represented by lines included in the group generated in a first viewpoint and a second figure represented by lines included in the group generated in a second viewpoint different from the first viewpoint, and determining the disparity between the first viewpoint and the second viewpoint from the distance between the first and second figures associated with each other by the matching.

6. A signal processing method for processing event signals generated by an event-based vision sensor (EVS), comprising: detecting lines formed by a set of positions of event signals within blocks obtained by dividing the detection area of ​​the EVS, and generating groups containing the lines when the lines detected in adjacent blocks satisfy a predetermined relationship, by an operation performed by a processor in accordance with program code stored in memory.

7. A program for processing event signals generated by an event-based vision sensor (EVS), wherein the actions performed by a processor according to the program include detecting lines formed by a set of positions of the event signals generated in blocks obtained by dividing the detection area of ​​the EVS, and generating groups containing the lines when the lines detected in adjacent blocks satisfy a predetermined relationship.

Images