SPAD Array Imaging for Aligned Depth and Intensity in HMDs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional mixed-reality systems face challenges in generating parallax-corrected pass-through views due to temporal or spatial offsets between depth and intensity information, leading to artifacts like motion blur and low signal-to-noise ratios, especially under low light conditions.
Innovation Solution
The use of Single Photon Avalanche Diode (SPAD) arrays in head-mounted displays (HMDs) for intensity image sensing, which perform interleaved active and passive imaging operations at the sub-frame level, ensuring temporally aligned intensity and depth information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Illumination intensity
If conventional CMOS or CCD image sensors are used for intensity imaging, then the system can capture images in standard conditions, but the signal-to-noise ratio deteriorates significantly under low light conditions due to read noise
Solution Approach 1:
The patent combines active imaging (time-of-flight depth sensing with modulated light source) and passive imaging (ambient light intensity sensing) into a single SPAD array sensor. This merging allows the sensor to operate in photon-counting mode for both active and passive modes, achieving superior low-light performance by accumulating photon counts over integration periods while rejecting read noise through statistical processing of multiple samples.
Solution Approach 2:
The patent changes the operating parameters of the image sensor by using single-photon detection with variable integration times and frame rates. The system can dynamically adjust the integration period and number of frames to accumulate sufficient photon counts for reliable intensity measurement in low-light conditions, transforming the sensor's response characteristics from analog voltage (CMOS/CCD) to digital photon counting.
2Measurement precision
If separate cameras are used for depth and intensity information, then each camera can be optimized for its specific function, but spatial offset and temporal offset occur between the two types of information
Solution Approach 1:
The patent merges depth sensing and intensity imaging functions into a single SPAD array sensor. The same pixel array detects both the timing (for depth via time-of-flight) and intensity (via photon count accumulation) of reflected light, completely eliminating spatial offset between depth and intensity data. Temporal offset is also eliminated because both measurements are taken simultaneously from the same sensor during the same integration period.
Solution Approach 2:
The SPAD array sensor performs multiple functions: it simultaneously provides depth information through time-of-flight measurement and intensity information through photon counting, while also enabling motion detection through temporal comparison of successive frames. This multi-functionality from a single sensor eliminates the need for separate cameras and ensures perfect spatial and temporal alignment of all measurement types.
3Ease of operation
If conventional image sensors are used for pass-through imaging, then the system can provide basic visibility, but motion blur and artifacts appear due to temporal offset between depth and intensity capture
Solution Approach 1:
The patent combines depth and intensity capture into a single simultaneous measurement process using the SPAD array. Since both depth (via time-of-flight) and intensity (via photon counting) are measured at the same moment from the same sensor, there is no temporal offset between the two data streams, completely eliminating motion blur and registration artifacts in the generated pass-through images.
Solution Approach 2:
The patent replaces the mechanical/system-level solution of using separate synchronized cameras with a single-photon detection-based simultaneous measurement system. The SPAD array inherently captures both depth and intensity information in the same measurement cycle, substituting the need for complex mechanical synchronization and multi-camera calibration with a simpler single-sensor physics-based approach.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
This approach reduces motion blur and improves signal-to-noise ratio, enabling clearer and more accurate pass-through images even in low light environments, enhancing user experience and safety in mixed-reality applications.
Implementation Method 1
a first image sensor of the plurality of image sensors is a single photon avalanche diode (SPAD) array comprising a plurality of SPAD pixels
Implementation Method 2
each pixel is configured to generate electron-hole pairs in response to detected photons
Data Source
AI summary
An HMD includes a single photon avalanche diode (SPAD) array comprising a plurality of SPAD pixels. The HMD also includes a display positioned to display images for viewing by an eye of a user. The HMD also includes one or more processors and one or more hardware storage devices storing instructions that are executable by the one or more processors to configure the HMD to perform various acts associated with using the SPAD array to capture an image frame of an environment for display to the user.