Focal Stack Alignment Using FoV and Flow Fields for Depth Estimation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing Depth from Focus (DfF) technologies in commercial cameras, such as smartphones, suffer from inaccuracies due to focal breathing caused by structural movements during focus changes, which are not accounted for in neural network-based depth estimation methods.
Innovation Solution
A method that aligns focal stacks by calculating relative Field of View (FoV) using metadata, primarily aligns images based on this FoV, determines a flow field using radial, horizontal, and vertical motion features, and secondarily aligns the stack to correct for hardware limitations, followed by extracting focal features and creating a depth map using a neural network.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If neural network-based depth estimation is applied to focal stacks from commercial cameras, then depth estimation can be performed, but accuracy deteriorates due to focal breathing and structural movements during focus changes
Solution Approach 1:
The patent applies preliminary alignment actions before depth estimation by first calculating relative FoV for each image, then performing primary alignment based on FoV, and subsequently determining flow fields and performing secondary alignment. This preliminary alignment process corrects focal breathing and structural movements before the neural network performs depth estimation, thereby improving both alignment reliability and depth estimation accuracy.
Solution Approach 2:
The patent implements feedback mechanisms through iterative alignment processes. The flow field determination uses radial, horizontal, and vertical motion features to adjust image positions, and the secondary alignment step refines the results based on the primary alignment outcomes. This feedback loop continuously improves alignment accuracy, addressing the reliability issue before final depth estimation.
2Measurement precision
If focal stack alignment is performed without considering hardware limitations, then the alignment process is simple, but depth estimation accuracy deteriorates due to uncorrected focal breathing
Solution Approach 1:
The patent segments the alignment process into distinct stages: calculating relative FoV, primary alignment based on FoV, determining flow fields using radial/horizontal/vertical motion features, and secondary alignment. This segmentation allows each stage to address specific aspects of alignment, making the complex task manageable and improving accuracy while maintaining clear process structure.
Solution Approach 2:
The patent changes key parameters during the alignment process: it calculates and applies relative FoV values, determines flow fields based on motion features, and adjusts image positions accordingly. By dynamically changing these parameters to account for hardware limitations like focal breathing, the system improves depth estimation accuracy while managing complexity through systematic parameter adjustment.
3Measurement precision
If images are aligned without correcting for structural movements, then the alignment process is fast, but position errors and focal breathing remain in the focal stack
Solution Approach 1:
The patent performs preliminary alignment actions using relative FoV calculation and primary alignment before the more complex secondary alignment. This preliminary action quickly addresses the majority of position errors and focal breathing, reducing the time needed for subsequent processing while maintaining high position accuracy.
Solution Approach 2:
The patent applies partial alignment corrections in the primary stage using FoV-based scaling, then refines with secondary alignment using flow fields. This partial-excessive action approach ensures that most position errors are corrected early (saving time), while additional refinement handles remaining errors, achieving both speed and accuracy.
Data Source
AI summary
The present disclosure relates to a method of aligning images included in a focal stack and estimates depth information of an object in the images by extracting features of the aligned images, using a neural network model. A focal stack alignment method according to an embodiment of the present disclosure includes: calculating a relative Field of View (FoV) of a focal stack photographed by a photographing device using metadata of the photographing device; primarily aligning the focal stack on the basis of the relative FoV; determining a flow field of the focal stack using radial motion, horizontal motion, and vertical motion features of each of primarily aligned images; and secondarily aligning the focal stack using the flow field.