Hybrid Camera System for High-Speed Depth Sensing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current depth cameras have a limited update rate of no higher than thirty frames per second, which restricts their application for high-speed motion sensing.
Innovation Solution
A system comprising a conventional depth camera and a high-speed video camera, where the video camera guides temporal up-sampling of depth frames through optical flow measurement and perspective depth correction, resulting in an RGB-D video stream with an update rate matching the high-speed camera.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If a conventional depth camera is used, then depth information can be captured, but the frame rate is limited to no higher than thirty frames per second
Solution Approach 1:
The patent combines a high-speed video camera (capturing at 200 Hz) with a depth camera (capturing at 30 Hz) into a hybrid system. The video camera provides high temporal resolution while the depth camera provides accurate depth measurements. By merging these two sensing modalities and using optical flow from the video camera to guide temporal up-sampling of depth frames, the system achieves high-speed depth sensing with both high frame rate and accurate depth information.
2Speed
If the frame rate of depth sensing is increased, then high-speed motion sensing capability is improved, but the system complexity increases due to multiple cameras and processing requirements
Solution Approach 1:
The patent uses optical flow computation as an intermediary mechanism to bridge the high-speed video camera and the depth camera. The optical flow captures temporal motion information from the video stream, which then guides the temporal up-sampling process to generate intermediate depth frames at the high frame rate. This intermediary approach allows the system to achieve high-speed depth sensing without requiring a high-speed depth camera, thus managing system complexity.
3Speed
If temporal up-sampling is performed using only video camera data, then frame rate is improved, but depth accuracy deteriorates due to lack of direct depth measurements
Solution Approach 1:
The patent applies perspective depth correction as a preliminary action before final depth frame generation. Using depth information from the depth camera and optical flow from the video camera, the system pre-computes correction factors based on perspective geometry. This preliminary correction ensures that when temporal up-sampling generates intermediate depth frames, the depth values remain accurate by accounting for perspective changes due to camera motion, thus maintaining depth accuracy while achieving high frame rates.
Data Source
AI summary
In a method, a first video stream is received from a video camera, and a second video stream is received from a depth camera. A pixel mapping between the video camera and the depth camera is known. The video camera has an update rate greater than that of the depth camera. Optical flow in successive frames of the first video stream is measured, and a portion of the optical flow attributable to depth change is extracted. A scaling factor is calculated for each pixel in successive frames of the first video stream to determine whether a depth change has occurred. A perspective depth correction is applied to each pixel having a depth change. The perspective depth correction is based upon the depth of the corresponding pixel in the most recent frame from the second video stream. A combined video stream having an update rate of the video camera and depth information from the depth camera is obtained.