Depth Rate Up-conversion via Contour-Based Depth Dragging
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing techniques for providing electronic content based on images of a physical environment and depth information are inaccurate and inefficient due to differences in capture rates between light intensity and depth cameras, leading to unsynchronized data and undesirable occlusions.
Innovation Solution
The method involves generating additional depth frames by adjusting depth values based on light intensity camera data, using contour images and occlusion masks to synchronize the depth frames with the light intensity frames, effectively up-converting the depth frame rate to match the light intensity frame rate.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If depth camera operates at a lower frame rate to capture depth information, then processing load and power consumption are reduced, but synchronization with light intensity frames deteriorates causing inaccurate occlusions
Solution Approach 1:
The patent applies preliminary action by generating a contour image from the light intensity frame before depth up-conversion. This contour image serves as a template that guides the subsequent depth value adjustment process, ensuring that the synthesized depth frames are pre-aligned with the temporal requirements of the light intensity sequence, thereby maintaining synchronization without requiring the depth camera to operate at full frame rate.
Solution Approach 2:
The patent changes the temporal parameter of depth information by up-converting the depth frame rate. Instead of capturing depth frames at the same high rate as light intensity frames, the system captures depth frames at a lower rate and then synthesizes additional depth frames by adjusting depth values based on contour image changes. This parameter transformation maintains synchronization while reducing the original capture rate requirements.
2Device complexity
If depth camera operates at a lower frame rate, then device complexity and processing requirements are reduced, but measurement precision of depth information deteriorates due to temporal misalignment
Solution Approach 1:
The patent introduces a contour image as an intermediary element between the light intensity frames and the depth frames. This contour image, generated from the light intensity data, serves as a mediator that carries temporal and spatial information to guide the adjustment of depth values. By using this intermediary, the system can synchronize depth information with light intensity frames without requiring the depth camera to operate at the same high frame rate, thus reducing processing requirements while maintaining precision.
Solution Approach 2:
The patent applies asymmetry by treating light intensity frames and depth frames differently in the processing pipeline. Instead of requiring symmetric frame rates for both sensors, the system processes light intensity frames at the original high frame rate while processing depth frames at a lower rate, then uses asymmetric operations (contour image generation and depth value adjustment) to synchronize them. This asymmetric approach reduces overall processing complexity while maintaining synchronization accuracy.
3Ease of manufacture
If depth values are simply repeated from previous frames for up-conversion, then implementation simplicity is maintained, but accuracy of occlusion display deteriorates due to lack of motion adaptation
Solution Approach 1:
The patent applies dynamics by making the depth value adjustment process adaptive rather than static. Instead of simply repeating depth values from previous frames, the system dynamically adjusts depth values based on changes detected in the contour image. This dynamic adjustment allows the depth information to adapt to motion in the scene, improving occlusion display accuracy while maintaining relative implementation simplicity through the use of contour-based guidance.
Data Source
AI summary
Various implementations disclosed herein include devices, systems, and methods that create additional depth frames where a depth camera runs at a lower frame rate than a light intensity camera. Rather than upconverting the depth frames by simply repeating a previous depth camera frame, additional depth frames are created by adjusting some of the depth values of a prior frame based on the RGB camera data (e.g., by “dragging” depths from their positions in the prior depth frame to new positions for a new frame). Specifically, a contour image is generated, and changes in the contour image are used to determine how to adjust (e.g., drag) the depth values for the additional depth frames. The contour image may be based on a mask (e.g., occlusions masks identifying where the hand occludes the virtual cube).