Event and Image Sensor Fusion for 3D Trajectory Resolution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional machine vision systems, relying on 2D cameras or 3D point clouds, face challenges in accurately capturing continuous surface representations of moving objects, especially when using event sensors which provide less data and lower pixel density compared to image sensors, making it difficult to combine data effectively for precise robotic tasks.
Innovation Solution
A multi-camera sensing system that combines event sensors and image sensors to capture scene features by scanning multiple paths with a signal generator, generating events and images that are used to determine trajectories and centroids, enhancing resolution through the distribution of signal intensity and timestamps, allowing for super-resolution scanning and improved object detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If event sensors are used to capture moving objects, then the response speed is improved, but the data quantity and pixel density deteriorate
Solution Approach 1:
The patent combines event sensor data and image sensor data into a unified representation. Event sensors provide temporal precision for motion detection while image sensors provide spatial detail and data completeness. The system merges these complementary data streams to achieve both fast response and rich information capture.
Solution Approach 2:
The patent introduces a temporal dimension to the data representation by using timestamps from event sensors alongside spatial information from image sensors. This creates a four-dimensional representation (x, y, z, t) that captures both the spatial detail of image sensors and the temporal precision of event sensors, resolving the contradiction between speed and information quantity.
2Speed
If event sensors are used to capture moving objects, then the response speed is improved, but the pixel density deteriorates
Solution Approach 1:
The patent merges the high temporal resolution of event sensors with the high spatial resolution of image sensors. Event sensors trigger rapid responses to motion while image sensors provide dense pixel coverage. The combined system achieves both fast response speed and high pixel density by leveraging the strengths of each sensor type.
Solution Approach 2:
The patent applies different sensor types to different aspects of the capture task: event sensors are used locally for temporal precision in motion detection, while image sensors are used for spatial detail. This local specialization allows each sensor to excel at its optimal function, achieving both speed and density without compromise.
3Shape
If 3D point clouds are used to represent objects, then the spatial information is improved, but the continuous surface representation deteriorates
Solution Approach 1:
The patent transitions from discrete 3D point cloud representation to a continuous 4D spatio-temporal representation by incorporating timestamps. This additional temporal dimension allows the system to represent continuous surface evolution over time, transforming discrete spatial points into continuous spatio-temporal trajectories that maintain both spatial accuracy and surface continuity.
Solution Approach 2:
The patent makes the representation dynamic by using time-stamped event data to track surface changes continuously. Instead of static 3D point clouds, the system creates dynamic spatio-temporal representations that adapt to object motion, maintaining continuous surface representation even as objects move and change shape.
Data Source
AI summary
Embodiments are directed to perceiving scene features using event sensors and image sensors. Paths may be scanned across objects in a scene with one or more beams. Images of the scene may be captured with frame cameras. Events may be generated based on detection of beam reflections that correspond to the objects. Trajectories may be based on the paths and the events. A distribution of intensity associated with the trajectories may be determined based on the energy associated with the traces in the images. Centroids for the trajectories may be determined based on the distribution of the intensity of energy, a resolution of the frame cameras, or timestamps associated with the events. Enhanced trajectories may be generated based on the centroids such that the enhanced trajectories may be provided to a modeling engine that executes actions based on the enhanced trajectories and the objects.