Multi-Camera Hole Filling via Intrinsic Parameter Normalization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The warping process in multi-camera systems introduces holes (occlusions and disocclusions) due to the difference in points of view between the forward-facing image sensor and the user's perspective, which affects the user experience in extended reality environments.
Innovation Solution
A method for multi-camera hole filling that utilizes multiple image sensors to generate an occlusion mask, normalizes images based on intrinsic camera parameters, and applies diffusion and feathering processes to fill these holes, accounting for depth and displacement.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If image warping is applied to account for POV difference between image sensor and user eye, then user comfort and POV accuracy are improved, but holes (occlusions and disocclusions) are introduced in the warped images
Solution Approach 1:
The patent introduces an intermediary hole-filling process that uses depth information and neighboring pixel data to fill in the holes created by warping. This mediator recovers the lost image information by synthesizing pixel values based on spatial relationships and depth cues, thus resolving the contradiction between maintaining POV accuracy and preserving image completeness
Solution Approach 2:
The patent performs preliminary depth estimation and hole detection before final image rendering. By identifying occlusion and disocclusion regions in advance and preparing fill data from multiple sources (neighboring pixels, depth buffers), the system proactively addresses the information loss that will occur during warping, thus maintaining both POV accuracy and image completeness
2Loss of information
If multiple image sensors are used to fill holes, then image completeness is improved, but device complexity increases
Solution Approach 1:
The patent makes the additional image sensors serve multiple functions: they provide both the primary imaging data and the supplementary data needed for hole filling. The same hardware resources are utilized for both standard image capture and occlusion/disocclusion recovery, thus improving image completeness without proportionally increasing system complexity
Solution Approach 2:
The patent merges the hole-filling function with the existing image processing pipeline. Rather than adding a separate complex subsystem, the patent integrates depth-based hole detection and pixel synthesis into the existing rendering flow, combining multiple functions (imaging, depth estimation, hole filling) into a unified processing architecture that minimizes additional complexity
Data Source
AI summary
The method includes: obtaining a first image of an environment from a first image sensor associated with first intrinsic parameters; performing a warping operation on the first image according to perspective offset values to generate a warped first image in order to account for perspective differences between the first image sensor and a user of the electronic device; determining an occlusion mask based on the warped first image that includes a plurality of holes; obtaining a second image of the environment from a second image sensor associated with second intrinsic parameters; normalizing the second image based on a difference between the first and second intrinsic parameters to produce a normalized second image; and filling a first set of one or more holes of the occlusion mask based on the normalized second image to produce a modified first image.