3D Object Tracking With Interleaved Depth Frames for Faster Reporting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems struggle to maximize the reporting rate of 3D coordinates of landmarks in high frame rate tracking applications like virtual reality and augmented reality, limiting smooth visual experiences.
Innovation Solution
An object tracking system that interleaves 2D and 3D image frames, allowing the processor to calculate two 3D locations per frame period by reusing depth image frames from previous periods, reducing memory usage and access time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the system captures interleaved 2D and 3D image frames at sensor frame rate, then the reporting rate of 3D locations is doubled, but the memory usage and access time increase
Solution Approach 1:
The patent extracts and reuses only the necessary 3D depth image frames from previous periods rather than storing all frames. Specifically, it retrieves the N-th 3D depth image frame from memory to calculate the second 3D location, minimizing memory usage while maintaining the doubled reporting rate
Solution Approach 2:
The system performs preliminary capture of 3D depth image frames during the first period and stores them in memory for later reuse. This preliminary action allows the frames to be readily available when needed for calculating the second 3D location in the second period, reducing access time
2Productivity
If the system calculates two 3D locations per frame period by reusing depth image frames, then the reporting rate increases, but the processing complexity increases
Solution Approach 1:
The patent uses the N-th 3D depth image frame as a copy that can be reused for multiple calculations. The same depth frame is used to calculate both the first 3D location (with N-th 2D image frame) and the second 3D location (with (N+1)-th 2D image frame), simplifying processing by avoiding the need to capture and process new depth frames for each calculation
3Stability of the object's composition
If the system uses interleaved capture of 2D and 3D frames, then the visual experience smoothness is improved, but the sensor frame rate requirements increase
Solution Approach 1:
The patent maintains continuous tracking by interleaving 2D and 3D frame capture at the sensor frame rate. This continuous interleaved capture ensures that both 2D image frames and 3D depth image frames are consistently available for processing, providing smooth visual experience without requiring the sensor to operate at a higher overall frame rate
Data Source
AI summary
A 3D location reporting method includes capturing a plurality of 2D image frames having a sensor frame rate and a plurality of 3D depth image frames having the sensor frame rate, wherein the plurality of 2D image frames and the plurality of 3D depth image frames are interleaved, and a first 2D image frame of the plurality of 2D image frames is captured earlier than a first 3D depth image frame of the plurality of 3D depth image frames; calculating a first 3D location according to an N-th 2D image frame of the plurality of 2D image frames and an N-th 3D depth image frame of the plurality of 3D depth image frames; and calculating a second 3D location according to the N-th 3D depth image frame and an (N+1)-th 2D image frame of the plurality of 2D image frames.