Depth Camera Augmented Pixels Time-of-Flight Motion Blur
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional depth sensing methods, such as triangulation and time-of-flight, face challenges in achieving high performance, low computational power, and compact form factors while providing accurate and efficient depth information in artificial reality systems, often resulting in high computational costs, limited lateral resolution, and motion blur when capturing moving objects.
Innovation Solution
A depth camera assembly (DCA) with augmented pixels that project pulses of light and image the local area, utilizing a plurality of gates with local storage locations to capture and store image data during synchronized exposure intervals, allowing for the generation of depth information with reduced motion blur and improved precision.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional time-of-flight methods are used for depth sensing, then depth information can be obtained, but lateral resolution is limited due to the limited number of pixels in conventional sensors
Solution Approach 1:
Each pixel in the sensor array is segmented into multiple independently controllable photo detectors, with each photo detector capable of capturing photons during specific time intervals. This segmentation allows the system to achieve higher lateral resolution by effectively creating multiple virtual pixels from each physical pixel location, thereby improving depth sensing precision without increasing the physical sensor size.
Solution Approach 2:
The patent introduces a time dimension to the traditional spatial pixel array by enabling each pixel to capture photons at multiple distinct time intervals. This temporal dimensionality transformation allows the system to achieve enhanced lateral resolution through time-resolved photon counting, effectively adding a fourth dimension (time) to the conventional two-dimensional pixel array.
2Illumination intensity
If multiple image frames with different exposures are captured to generate HDR image, then high dynamic range imaging is achieved, but latency increases due to the need to read out multiple frames
Solution Approach 1:
The system continuously captures photons across multiple time intervals within a single exposure period, rather than requiring separate exposure captures. The photo detectors continuously monitor incoming photons and sort them by arrival time, enabling HDR imaging to be achieved in a continuous, uninterrupted manner that eliminates the latency associated with sequential frame capture and readout.
Solution Approach 2:
The patent implements preliminary time-stamping of photons as they are detected, organizing them into predetermined time intervals before the exposure is complete. This preliminary organization of photon data by time interval during the exposure itself, rather than after capture, allows for immediate HDR image generation without requiring additional readout operations, thereby reducing latency.
3Measurement precision
If conventional triangulation methods are used for depth sensing, then depth map can be generated, but computational cost is high due to rectification and searching for corresponding points
Solution Approach 1:
The patent replaces the complex computational mechanics of triangulation-based depth mapping with a direct time-of-flight measurement approach. Instead of requiring rectification and correspondence searching between stereo images, the system directly measures the time of flight of photons to calculate depth, substituting heavy computational processing with straightforward temporal measurement and calculation.
Solution Approach 2:
The time-resolved photon counting system inherently provides depth information through the time-stamping mechanism built into the detection process itself. Each photon's arrival time automatically encodes depth information, eliminating the need for separate computational steps to extract depth data from image pairs, thereby reducing computational power requirements while maintaining depth accuracy.
4Measurement precision
If time-of-flight methods capture moving objects, then depth information is obtained, but motion blur occurs due to the relatively high number of image frames required
Solution Approach 1:
The system uses periodic time intervals to sort and count photons, with each time interval corresponding to a specific depth range. This periodic temporal sampling allows the system to capture depth information from moving objects without requiring multiple sequential frames, as each photon's arrival time directly indicates its depth at the moment of detection, thereby eliminating motion blur while maintaining depth measurement precision.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
The DCA effectively determines depth information with reduced motion blur and improved precision, achieving high dynamic range imaging and efficient depth sensing, suitable for artificial reality systems, while maintaining low computational power and a compact form factor.
Implementation Method 1
depth determination based on time-of-flight using a camera assembly with augmented pixels
Implementation Method 2
The camera assembly is configured to image a portion of the local area illuminated with the pulses of light
Data Source
AI summary
A camera assembly for determining depth information for a local area includes a light source assembly, a camera assembly, and a controller. The light source assembly projects pulses of light into the local area. The camera assembly images a portion of the local area illuminated with the pulses. The camera assembly includes augmented pixels, each augmented pixel having a plurality of gates and at least some of the gates have a respective local storage location. An exposure interval of each augmented pixel is divided into intervals associated with the gates, and each local storage location stores image data during a respective interval. The controller reads out, after the exposure interval of each augmented pixel, the image data stored in the respective local storage locations of each augmented pixel to generate image data frames. The controller determines depth information for the local area based in part on the image data frames.