System for determining egomotion and method for the same

Abstract

The current invention relates to a system for determining egomotion wherein use is made of an event camera, multiple motion indicating devices and a processing unit. The motion indicating devices are configured to generate motion signals based on events in an event stream generated by the event camera. The processing device is configured to determine a signal representing the egomotion based on the motion signals. The current invention further relates to a method for determining egomotion wherein use is made of a system according to the invention.

Description

[0001] P37120PC00 / SBI

[0002] SYSTEM FOR DETERMINING EGOMOTION AND METHOD FOR THE SAME

[0003] The current invention relates to a system for determining egomotion wherein use is made of an event camera. The current invention further relates to a method for determining egomotion wherein use is made of a system according to the invention.

[0004] Egomotion may be stated to be the relative motion of oneself, or in this case of the system, with respect to a static background. In other words, egomotion is the motion of the system in a fixed frame.

[0005] Known visual odometry systems rely on capturing consecutive images from a frame based camera and extracting features, such as corners or edges, to track their motion between frames. Algorithms like Simultaneous Localization and Mapping (SLAM) are often used in combination with visual odometry to refine motion estimates and construct a map of the environment. Deep learning and convolutional neural networks (CNNs) may be used to enhance the accuracy of such systems by learning motion patterns and compensating for visual noise or distortions. Further features such as sensor fusion, integrating data from Inertial Measurement Units (IMUs), LiDAR, or GPS are increasingly common in these known systems to increase robustness and / or reliability in motion estimates.

[0006] A downside of known systems is that these systems suffer from processing latency as a result of the heavy processing required to determine the egomotion based on the obtained data. A further downside of known system is that sensors of these systems inherently measure redundant information, i.e. information obtained when there is no motion. A further downside associated with systems based on deep learning algorithms, is that such algorithms may be fragile when encountering unknown data.

[0007] The invention aims to improve upon one or more of the abovementioned downsides.

[0008] The aim of the invention is achieved by the system for determining egomotion according to claim 1. The system may be configured to determine egomotion in one or more directions. For example, the system, when configured to be placed on a car, may be configured to determine egomotion in two directions, a forward / backward direction and a left / right direction. In another example, the system may be configured to determine egomotion for six degrees of motion, for example three linear directions and three rotary directions. In such a case, the system may be mounted on a drone.

[0009] The system comprises an event camera comprising multiple pixels. The pixels may be configured to measure a change in illumination, e.g. a change in light intensity, e.g. compared to an earlier state. The pixels may be ordered in a two-dimensional grid such that each pixel may be characterized by two coordinates, such as an x-coordinate and a y-coordinate. The camera may further comprise auxiliary components such as lenses, mirrors, filters, a camera processing unit and the like. In embodiments, the camera consists of a light receiving unit comprising the pixels and an event output unit for outputting the event stream comprising the events.

[0010] The event camera is configured for producing an event stream comprising events generated by the camera. An event according to the invention may be a change in illumination measurement by a pixel of the event camera. An event in the event stream may be characterized by a pixel coordinate, indicating which pixel registered the event. The event stream may be asynchronous, e.g. such that the events in the event stream do not carry a time stamp, e.g. as a result of the camera not having an internal clock.

[0011] A movement by the system may result in a multitude of events which are put in the event stream. A stationary state of the system may result in no generated events. The event stream comprises a reduced amount of redundant information because no information may be send when no change in illumination is registered by a pixel.

[0012] The system further comprises multiple motion indicating devices. The motion indicating device may be embodied as an application specific circuit or the motion indicating device may be a processing device comprising software configured for processing the event stream and outputting the motion signals. Each motion indicating device is associated with a pixel multiplet of the event camera. The pixel multiplet comprises a first pixel and a second pixel of the event camera. For example, the pixel multiplet may be a pixel pair or a pixel triplet. A single pixel of the event camera may be associated with multiple motion indicating devices. For example, a single pixel may be considered a first pixel of multiple motion indicating devices. A subset of the pixels of the event camera may be associated with motion indicating devices, e.g. some pixels of the event camera may not be associated with a motion indicating device such that events generated by such pixels are not considered for determining the egomotion of the system. Each motion indicating device is configured to output a motion signal when an event associated with the first pixel occurs in the event stream followed, within a predetermined time period, by an event associated with the second pixel. No motion signal may be outputted if an event associated with the first pixel occurs in the event stream followed by an event associated with the second pixel outside of the predetermined time period. The motion indicating devices may be said to process the event stream into a motion signal stream.

[0013] The system further comprises a processing device for outputting a signal representing the egomotion of the system based on motion signals received from the multiple motion indicating devices. The processing device may be embodied as a counter for counting the motion signals, a processor with software for processing the motion signals, e.g. using some form of neural network, or the processing device may be embodied in any other suitable way.

[0014] The system of the invention, e.g. the motion indicating devices and the processing unit thereof, may be embodied as a system on a chip, e.g. as an application specific integrated circuit. For example, the motion indicating devices and processing unit may be embodied as a system on a chip which may connected to an event camera for processing the event stream outputted thereby. In other embodiments, one or more of the event camera, motion indicating devices and processing device may be simulated by a processor to perform the functions thereof.

[0015] Each motion indicating device is associated with one of a number of motion directions of the system based on an orientation and / or direction of the pixel multiplet in the event camera. For example, the motion directions may be associated with a degree of freedom of the system. For example, the motion directions may be associated with an x-direction, y-direction, z- direction and / or rotations around those directions.

[0016] The motion direction that the motion indicating device is associated with may be reflected in the orientation and / or direction of the pixel multiplet of the event camera in relation to the motion direction. For example, if the motion direction is a backward motion, the first and second pixel may be arranged on a pixel grid of the camera such that an arrow may be drawn between the first and second pixel towards a center of the pixel grid of the camera. By grouping all motion indicating devices associated with a backward motion and drawing arrows between the associated pixels, an arrow field may be created on the pixel grid with all arrows pointing towards the center of the pixel grid. In the art, the optical flow refers to a pattern of apparent motion of objects, surfaces, and edges in a visual scene produced by the relative movement between an observer and the environment. Optical flow may be expressed as a continuous velocity field that describes how points in the image intensity landscape shift over time.

[0017] The orientation and / or direction of the pixel multiplets in the event camera may be determined by an expected optical flow pattern of events on the event camera for the corresponding degrees of freedom. For example, an orientation and / or direction of a pixel multiplet may be determined based on a given desired degree of freedom. The determining may be dependent on the field of view of the camera. For example, a wider field of view may produce a more curved and non-linear flow near the edges of the camera influencing the pixel pair multiplets to be determined for a given degree of freedom. For example, a narrower field of view may allow for more linear arrangements of the pixel multiplets.

[0018] In embodiments, for rotational degrees of freedom the optical flow may increase with distance from a center of the event camera. As a result, for rotational degrees of freedom, pixel multiplets may be orientated tangentially around the center of the event camera, preferably with increased spacing between pixels of the mutliplets closer to the periphery.

[0019] In embodiments, for translational degrees of freedom, the optical flow may form a divergence or convergence pattern. For example, forward motion may lead to an outward optical flow from the center, backward motion may lead to inward optical flow towards the center, and lateral translations may produce predominantly horizontal or vertical optical flow in the event camera. The pixel multiplet may be determed to match the dominant optical flow direction associated with the desired degree of freedom.

[0020] In embodiments, sensitivity of the system for determining egomotion may be tuned by adjusting spacing of pixels in pixel multiplets or by weighing contributions of pixel multiplets based on a position of the pixel mutliplets in the field of view of the event camera.

[0021] Each motion indicating device has a complimentary motion indicating device associated with an opposing one of the motion directions. For example, for each motion indicating device with its associated pixels a complimentary motion indicating device may be present with the ordering of the pixels reversed. That is to say, the first pixel of the motion indicating device is the second pixel of the complimentary motion indicating device and vice versa. In the previous example, the complimentary arrow field, associated with forward motion, would look like arrows pointing away from the center of the pixel grid. For example, the opposing motion direction of the x-direction may be the -x direction and the opposing motion direction of clockwise rotation around the x-axis may be counter clockwise rotation around the x-axis.

[0022] The motion indicating devices are grouped in motion direction groups based on the associated motion directions, wherein motion direction groups associated with opposing motion directions are complimentary motion direction groups. Thus, all motion indicating devices associated with a first motion direction may be grouped in a first motion direction group, all motion indicating devices associated with a first complimentary motion direction may be grouped in a first complimentary direction group, and so on for each relevant motion direction of the considered system. Thus, motion signals outputted by motion indicating devices associated with the first motion direction group may all be associated with movement in the first direction and motion signals outputted by complimentary motion indicating devices associated with the complimentary first motion direction group may all be associated with opposite movement in the first direction.

[0023] The motion indicating devices may generate the associated motion signals asynchronously, independently and in parallel with each other. No particular ordering or structure between the different motion indicating devices has to be present.

[0024] The processing device is configured to determine the signal representing egomotion of the system based on comparing motion signals from motion indicating devices in a particular motion direction group to motion signals from motion indicating devices in the complimentary motion direction group thereof. The processing device may further be configured to output the determined signal representing egomotion of the system.

[0025] For example, the processing device may, for a particular motion direction, compare a number of motion signals from a particular motion direction group to a number of motion signals from the complimentary motion direction group to determine if a motion is in the particular motion direction or in the opposite complimentary motion direction. Motion in a particular motion direction may be represented by a number, e.g. indicating an amount of motion in the direction, which number may be continuously updated as new motion signals coming from the motion direction group and complimentary motion direction group are processed by the processing device.

[0026] Thus, in embodiments, the processing device may be a counter counting motion signals to determine if a motion in a particular direction is in the particular direction, in the opposite direction, or if no motion occurs in that direction. The output signal representing the egomotion may be a digital signal, a visual signal or any type of suitable signal.

[0027] The invention is based on the insight that a motion signal may be generated by a motion indicating device as a result of a local motion, e.g. a motion of a background object, instead of an egomotion. However, if motion signals of the motion direction group and complimentary motion direction group are compared, these local motions may be filtered, e.g. as noise, and the dominant egomotion remains. Thus, the system may determine the egomotion.

[0028] In embodiments, the processing device is configured to determine the signal representing egomotion of the system by comparing a number of motion signals from motion determining devices in the particular motion direction group to a number of motion signals from motion determining devices in the complimentary motion direction group.

[0029] In embodiments, the processing device is configured to compare motion signals and / or number of motion signals by adding and / or subtracting motion signals and / or number of motion signals from the particular motion direction group to motion signals and / or number of motion signals from the complimentary motion direction group. For example, motion signals associated with the motion direction group may give a positive contribution to a signal representing egomotion in an associated direction, whereas motion signals associated with the complimentary group may give a negative contribution. In this way, addition and subtraction may be used to compare the signals. In embodiments, more the motion signals and / or number of motion signals may be compared in more complicated ways, for example based on multiplication and / or division operations.

[0030] In embodiments, each motion direction is associated with a translation or a rotation of the system, e.g. wherein opposing motion direction groups are associated with translations or rotations in opposing directions. For example, the translations may be related to Cartesian coordinate directions, and the rotations to rotations around the Cartesian axis. However, other coordinate systems, such as polar coordinates, may be used as a basis for determining the motion directions.

[0031] In embodiments, the system is further configured to determine a velocity thereof by assigning a velocity to each motion signal produced by the motion indicating devices based on, for each motion indicating device, a time difference between an event associated with the first pixel and an event associated with the second pixel, wherein the processing device is configured to output a signal representing the velocity of the system based on comparing velocities from motion indicating devices in a particular motion direction group to velocities from motion indicating devices in the complimentary motion direction group thereof. The time between the events associated with the first and second pixel may be associated with a velocity. Similarly to determining the egomotion, by comparing the determined velocity of motion indicating devices in particular motion direction groups and complimentary motion direction groups, the velocity in the motion direction may be determined.

[0032] In further embodiments, a lower time difference between events associated with the first and second pixel is associated with a higher velocity.

[0033] In embodiments, the motion indicating devices and processing device are embodied as a single system on a chip.

[0034] In embodiments, each motion indicating device is embodied as a circuit configured to generate a decaying internal voltage if an event associated with the first pixel occurs in the event stream, and to output the motion signal if an event associated with the second pixel occurs in the event stream while the decaying internal voltage is above a threshold voltage. In these embodiments, the decaying voltage may decay according to a predetermined pattern based on circuit characteristics. Further, if two events follow each other closely, the motion signal outputted by the circuit may have a larger amplitude, e.g. in voltage and / or current compared to when the two events are further apart such that the amplitude of the motion signal carries velocity information.

[0035] In embodiments, each motion indicating device is configured to output a signal representing a time difference between the event associated with the first pixel and the event associated with the second pixel based on a magnitude of the decaying internal voltage when the event associated with the second pixel occurs in the event stream.

[0036] The invention further relates to a method for determining egomotion wherein use is made of a system according to the invention.

[0037] In embodiments, method comprises:

[0038] - producing, with the event camera, an event stream which comprises events measured with the event camera;

[0039] - producing motion signals from the motion indicating devices when, for a particular motion indicating device, an event associated with the associated first pixel occurs in the event stream followed, within a predetermined time period, by an event associated with the associated second pixel;

[0040] - outputting a signal representing the egomotion of the system based on the motion signals produced by the motion indicating devices, wherein the signal representing the egomotion is determined by comparing motion signals from motion indicating devices in a particular motion direction group to motion signals from motion indicating devices in the complimentary motion direction group thereof.

[0041] In embodiments, method further comprises determining the signal representing egomotion of the system by comparing a number of motion signals from motion determining devices in the particular motion direction group to a number of motion signals from motion determining devices in the complimentary motion direction group

[0042] In embodiments, the method further comprises comparing motion signals and / or number of motion signals by adding and / or subtracting motion signals and / or number of motion signals from the particular motion direction group to motion signals and / or number of motion signals from the complimentary motion direction group.

[0043] In embodiments, the method further comprises determining a velocity of the system by assigning a velocity to each motion signal produced by the motion indicating devices based on, for each motion indicating device, a time difference between an event associated with the first pixel and an event associated with the second pixel, and outputting a signal representing the velocity of the system based on comparing velocities from motion indicating devices in a particular motion direction group to velocities from motion indicating devices in the complimentary motion direction group thereof.

[0044] The invention will now be explained below with reference to the drawing in which:

[0045] Fig.1 depicts a vehicle which may be provided with the system for determining egomotion;

[0046] Fig. 2 depicts a portion of a pixel grid of an event camera;

[0047] Fig. 3 depicts possible motion direction groups; and

[0048] Fig. 4 depicts a system for determining egomotion.

[0049] Figure 1 depicts a vehicle, in this case a drone, which may be provided with the system 1 for determining egomotion. The system 1 may generally be configured to determine egomotion in one or more directions depending on a purpose of the system 1. For example in the case of figure 1 , the system 1 for determining egomotion may be configured to determine the egomotion in the x-, y-, and z-directions as well as in the associated rotations depicted in the figure. If, instead of a drone, the vehicle is for example a car the relevant directions may be the x- and y-directions. If, in another example, the vehicle is a boat, the relevant directions may be the x- and y-directions as well as the associated rotations. For certain embodiments, other coordinate systems and associated directions may be used. For example, polar coordinates may be used.

[0050] Figure 2 depicts a portion of a pixel grid of an event camera 2, wherein the first pixel, pixel 1, and the second pixel, pixel 2, are indicated together with an arrow indicating that the two pixels are associated with a motion indicating device 3 such that if an event associated with pixel 1 occurs in the event stream, followed within a predetermined time period by an event associated with pixel 2, the associated motion indicating device 3 outputs a motion signal. By grouping multiple motion indicating devices 3 based on their associated motion directions, the system 1 , e.g. the processing device 4 thereof, may determine the egomotion thereof.

[0051] Figure 3 depicts possible motion direction groups represented by arrows in a pixel grid of the event camera 2. Figure 3 depicts four possible motion direction groups. A forward motion direction group and its complimentary backward motion direction group and a clockwise motion direction group and its complimentary anti-clockwise motion direction group. The motion direction groups are represented by arrows on a pixel grid of the event camera 2 such that a tail end of each arrow represents a first pixel of an associated motion indicating device 3 and a point of each arrow represents an associated second pixel. In this way, each arrow represents an associated motion indicating device 3. As may be appreciated, the associated motion direction may be inferred from the arrow fields in the pixel grid.

[0052] Figure 4 depicts a system 1 for determining egomotion thereof. The system 1 comprises an event camera 2 comprising multiple pixels and configured for producing an event stream, wherein the event stream comprises events generated by the event camera. The pixels of the event camera 2 may be ordered a pixel grid as depicted in figure 3.

[0053] The system 1 further comprises multiple motion indicating devices 3. Each motion indicating device is associated with a pixel multiplet, e.g. a pixel pair as shown in figure 2, comprising a first pixel and a second pixel. The motion indicating devices 3 are associated with motion direction groups, e.g. as shown in figure 3. The motion indicating devices 3 are configured to output a motion signal when an event associated with the first pixel occurs in the event stream followed, within a predetermined time period, by an event associated with the second pixel. The system 1 further comprises a processing device 4 for outputting a signal representing the egomotion of the system 1 based on motion signals received from the multiple motion indicating devices 3 by comparing motion signals from motion indicating devices 3 in a particular motion direction group to motion signals from motion indicating devices in the complimentary motion direction group thereof.

Claims

CLAIMS1. System for determining egomotion thereof, wherein the system comprises:- an event camera comprising multiple pixels and configured for producing an event stream, wherein the event stream comprises events generated by the event camera;- multiple motion indicating devices, wherein each motion indicating device is associated with a pixel multiplet comprising at least a first pixel and a second pixel of the event camera, and wherein each motion indicating device is configured to output a motion signal when an event associated with the first pixel occurs in the event stream followed, within a predetermined time period, by an event associated with the second pixel; and- a processing device for outputting a signal representing the egomotion of the system based on motion signals received from the multiple motion indicating devices, wherein each motion indicating device is associated with one of a number of motion directions of the system, e.g. wherein each motion direction is associated with a degree of freedom of the system, based on an orientation and / or distribution of the associated pixel multiplet in the event camera, wherein each motion indicating device has a complimentary motion indicating device associated with an opposing one of the motion directions, e.g. based on a reversed orientation and / or distribution of the associated pixel multiplet, wherein the motion indicating devices are grouped in motion direction groups based on the associated motion directions, wherein motion direction groups associated with opposing motion directions are complimentary motion direction groups, wherein the processing device is configured to determine the signal representing egomotion of the system based on comparing motion signals from motion indicating devices in a particular motion direction group to motion signals from motion indicating devices in the complimentary motion direction group thereof.

2. System according to claim 1, wherein the processing device is configured to determine the signal representing egomotion of the system by comparing a number of motion signals from motion determining devices in the particular motion direction group to a number of motion signals from motion determining devices in the complimentary motion direction group.

3. System according to one or more of the preceding claims, wherein the processing device is configured to compare motion signals and / or number of motion signals by adding and / or subtracting motion signals and / or number of motion signals from the particular motion direction group to motion signals and / or number of motion signals from the complimentary motion direction group.

4. System according to one or more of the preceding claims, wherein each motion direction is associated with a translation or a rotation of the system, e.g. wherein opposing motion direction groups are associated with translations or rotations in opposing directions.

5. System according to one or more of the preceding claims, wherein the system is further configured to determine a velocity thereof by assigning a velocity to each motion signal produced by the motion indicating devices based on, for each motion indicating device, a time difference between an event associated with the first pixel and an event associated with the second pixel, wherein the processing device is configured to output a signal representing the velocity of the system based on comparing velocities from motion indicating devices in a particular motion direction group to velocities from motion indicating devices in the complimentary motion direction group thereof.

6. System according to claim 5, wherein a lower time difference between events associated with the first and second pixel is associated with a higher velocity.

7. System according to one or more of the preceding claims, wherein the motion indicating devices and processing device are embodied as a single system on a chip.

8. System according to one or more of the preceding claims, wherein each motion indicating device is embodied as a circuit configured to generate a decaying internal voltage if an event associated with the first pixel occurs in the event stream, and to output the motion signal if an event associated with the second pixel occurs in the event stream while the decaying internal voltage is above a threshold voltage.

9. System according to claim 8, wherein each motion indicating device is configured to output a signal representing a time difference between the event associated with the first pixel and the event associated with the second pixel based on a magnitude of the decaying internal voltage when the event associated with the second pixel occurs in the event stream.

10. Method for determining egomotion wherein use is made of a system according to one or more of the claims 1 - 9.

11. Method according to claim 10, wherein the method comprises:- producing, with the event camera, an event stream which comprises events measured with the event camera;- producing motion signals from the motion indicating devices when, for a particular motion indicating device, an event associated with the associated first pixel occurs in the event stream followed, within a predetermined time period, by an event associated with the associated second pixel;- outputting a signal representing the egomotion of the system based on the motion signals produced by the motion indicating devices, wherein the signal representing the egomotion is determined by comparing motion signals from motion indicating devices in a particular motion direction group to motion signals from motion indicating devices in the complimentary motion direction group thereof.

12. Method according to one or more of the claims 10 - 11 , wherein the method further comprises determining the signal representing egomotion of the system by comparing a number of motion signals from motion determining devices in the particular motion direction group to a number of motion signals from motion determining devices in the complimentary motion direction group13. Method according to one or more of the claims 10 - 12, wherein the method further comprises comparing motion signals and / or number of motion signals by adding and / or subtracting motion signals and / or number of motion signals from the particular motion direction group to motion signals and / or number of motion signals from the complimentary motion direction group.

14. Method according to one or more of the claims 10 - 13, wherein the method further comprises determining a velocity of the system by assigning a velocity to each motion signal produced by the motion indicating devices based on, for each motion indicating device, a time difference between an event associated with the first pixel and an event associated with the second pixel, and outputting a signal representing the velocity of the system based on comparing velocities from motion indicating devices in a particular motion direction group to velocities from motion indicating devices in the complimentary motion direction group thereof.

Images