Extended Focal Plane Generation for Large Viewpoint Changes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current multi-focal-plane (MFP) displays struggle to fully preserve information, especially behind occluding objects, due to limitations in capturing and processing light field data for large viewpoint changes.
Innovation Solution
The method involves capturing a set of texture images with varying focal lengths using a large aperture, generating corresponding depth maps, and applying focal weight maps to form extended focal plane images that include information behind occluding objects.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If a small number of focal planes (4-6 planes) are used with depth blending, then the display is technically feasible and easier to implement, but information behind occluding objects is lost and disocclusions occur during viewpoint changes
Solution Approach 1:
The patent performs preliminary action by capturing multiple focus stack images at different focal distances before display. These images are processed to create extended focal planes that contain information from multiple depths, including areas behind occluding objects. This pre-captured data is then used during viewpoint changes to fill in disoccluded regions without requiring real-time computation or additional hardware complexity.
Solution Approach 2:
The patent introduces an intermediary processing step that combines multiple focus stack images to create extended focal plane representations. This intermediary data structure serves as a mediator between the limited focal planes and the full light field information, allowing viewpoint synthesis to access information behind occlusions without requiring a complete light field capture system.
2Loss of information
If a large number of focal planes (14 or more) are used without depth blending, then information behind occluding objects is preserved, but the device complexity and processing requirements increase significantly
Solution Approach 1:
The patent merges multiple focus stack images captured at different focal distances into extended focal plane representations. By combining these images through image processing algorithms, the system consolidates information from what would traditionally require many discrete focal planes into a compressed set of extended planes, reducing the effective number of planes needed while preserving information behind occlusions.
Solution Approach 2:
The patent changes the parameter representation from discrete focal plane indices to continuous focus stack parameters. Instead of treating each focal plane as a separate entity, the system uses focus stack imagery with varying focal lengths as a continuous parameter space, allowing efficient interpolation and synthesis of intermediate views without requiring a large discrete number of focal planes.
3Adaptability or versatility
If focal planes are shifted for large viewpoint changes, then virtual reality and augmented reality applications are enhanced, but disocclusions and holes appear in the synthesized views
Solution Approach 1:
The patent performs preliminary action by pre-capturing focus stack images that contain information from multiple depths and angles. When viewpoint changes occur, the system can synthesize new views by combining information from these pre-captured images, filling in disoccluded regions that would otherwise be missing. This preliminary data collection enables robust viewpoint transformation without information loss.
Solution Approach 2:
The patent adds a depth dimension to the traditional 2D image synthesis by utilizing focus stack information from multiple focal distances. This extra dimensional information allows the system to fill in disoccluded regions during viewpoint changes by projecting information from different depth planes into the synthesized view, effectively adding a third dimension (depth) to the viewpoint synthesis process.
Data Source
Figure 1A
Figure 1B
Figure 2
AI summary
Methods and systems are described for capturing and displaying content for multiple focal plane (MFP) displays. In one example, a plurality of texture images of a scene are captured using a large-aperture camera, each texture image having a different focal distance. For each texture image, a focal plane image is generated. To generate the focal plane image, in some embodiments, each pixel in each texture image is multiplied by a respective weight value. The weight value may be based on a measured depth of the respective pixel (e.g. as determined using a captured depth map) and/or on a level of focus (or defocus) of the respective pixel as determined through filtering. The focal plane images may be displayed on a multifocal-plane display and may be used to generate a virtual viewpoint.