Event Camera Tracking for AR VR Controllers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing AR/VR systems face increased latency and power consumption due to the high data processing requirements of conventional frame-based cameras used for determining correspondences between head-mounted devices (HMDs) and secondary devices, despite offering high precision.
Innovation Solution
The use of event cameras to determine correspondences between HMDs and secondary devices by obtaining light intensity data from pixel events, identifying optical sources based on defined illumination parameters, and mapping location data to determine the correspondence between the two devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional frame-based cameras with high resolution (10-20 megapixels) are used to determine correspondence between HMD and secondary device, then measurement precision is improved, but use of energy and loss of time increase due to substantial data processing requirements
Solution Approach 1:
The patent extracts only the essential information needed for tracking by using event cameras that detect and report only pixel changes exceeding a threshold. Instead of processing complete high-resolution frames with all pixel data, the system extracts only the subset of pixel events that contain meaningful information about secondary device movement, thereby reducing data volume and processing power requirements while maintaining tracking precision
Solution Approach 2:
The system applies partial action by processing only the necessary portion of visual data. Event cameras generate data sparsely only when changes occur, rather than continuously processing all pixels in high-resolution frames. This partial processing approach maintains sufficient tracking precision while significantly reducing the power budget required for data processing
2Measurement precision
If conventional frame-based cameras with high resolution (10-20 megapixels) are used to determine correspondence between HMD and secondary device, then measurement precision is improved, but loss of time increases due to substantial data processing requirements
Solution Approach 1:
The patent extracts only the essential information needed for tracking by using event cameras that detect and report only pixel changes exceeding a threshold. Instead of processing complete high-resolution frames with all pixel data, the system extracts only the subset of pixel events that contain meaningful information about secondary device movement, thereby reducing data volume and processing time while maintaining tracking precision
Solution Approach 2:
The system skips unnecessary processing steps by using event-based data that directly encodes meaningful changes. Rather than processing entire frames and filtering out unchanged regions, the event camera approach rushes through only the relevant data changes, significantly reducing latency in determining correspondence between HMD and secondary device
3Measurement precision
If conventional frame-based cameras are used to track secondary device, then measurement precision is improved, but device complexity increases due to substantial data processing requirements
Solution Approach 1:
The patent extracts only the essential information needed for tracking by using event cameras that detect and report only pixel changes exceeding a threshold. Instead of processing complete high-resolution frames with all pixel data, the system extracts only the subset of pixel events that contain meaningful information about secondary device movement, thereby reducing data volume and processing complexity while maintaining tracking precision
Solution Approach 2:
The system changes the fundamental parameter of data representation from continuous high-resolution frames to discrete pixel events with associated metadata (timestamp, pixel coordinates, polarity). This parameter change simplifies the data structure and processing algorithms, reducing device complexity while maintaining the ability to achieve precise tracking measurements
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
This approach reduces data processing requirements and latency by focusing on changes in light intensity, enabling efficient and precise tracking of secondary devices without the need for high-resolution, frame-based camera data processing.
Implementation Method 1
Each respective pixel event is generated in response to a particular pixel sensor detecting a change in light intensity that exceeds a comparator threshold
Implementation Method 2
At least a portion of the light is emitted by a plurality of optical sources disposed on the secondary device
Data Source
AI summary
In one implementation, a method involves obtaining light intensity data from a stream of pixel events output by an event camera of a head-mounted device (“HMD”). Each pixel event is generated in response to a pixel sensor of the event camera detecting a change in light intensity that exceeds a comparator threshold. A set of optical sources disposed on a secondary device that are visible to the event camera are identified by recognizing defined illumination parameters associated with the optical sources using the light intensity data. Location data is generated for the optical sources in an HMD reference frame using the light intensity data. A correspondence between the secondary device and the HMD is determined by mapping the location data in the HMD reference frame to respective known locations of the optical sources relative to the secondary device reference frame.