Light Field Capture via Keypoint Convex Hull Merging
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional digital media formats, such as 2D flat images, limit the ability to recreate memories and events with high fidelity and require significant additional data for interpolation or extrapolation, leading to inefficiencies in processing speed and transfer rates.
Innovation Solution
A method for capturing an unstructured light field using keypoint detection and tracking, with convex hull merging and IMU data compensation, to generate artificially rendered images between captured frames, allowing for efficient interpolation and extrapolation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional 2D flat images are used for digital media, then the format is simple and easy to store, but the ability to reproduce memories and events with high fidelity is limited
Solution Approach 1:
The patent transitions from traditional 2D flat images to light field representation which captures 4D information (x, y, z, theta, phi dimensions). This dimensional expansion enables high-fidelity reproduction of scenes by capturing light ray directions and positions, allowing virtual views from any angle without requiring additional physical cameras.
2Measurement precision
If dense depth maps or optical flow maps are used for interpolation, then the scene structure is described densely, but processing speed and transfer rates are significantly reduced
Solution Approach 1:
The patent extracts only the essential geometric structure information (convex hulls of keypoints) rather than storing dense per-pixel depth maps or optical flow maps. By representing scene structure through sparse keypoint convex hulls, the system maintains accurate geometric relationships while dramatically reducing data volume for interpolation operations.
Solution Approach 2:
The patent segments the scene representation into discrete keypoint convex hulls for each view rather than treating the entire scene as a continuous dense field. This segmentation allows independent processing of each view's geometric structure, enabling efficient interpolation by operating on separate convex hull representations that can be merged computationally.
3Adaptability or versatility
If multiple images are captured for panorama or 3D reconstruction, then the viewing experience is enhanced, but the amount of additional data required increases significantly
Solution Approach 1:
The patent creates virtual copies of scene views through computational rendering from captured light field data and keypoint convex hulls, rather than storing multiple actual captured images. This allows generation of unlimited virtual views from a minimal set of captured images, significantly reducing the quantity of stored data while maintaining viewing experience quality.
Data Source
AI summary
This present disclosure relates to systems and processes for capturing an unstructured light field in a plurality of images. In particular embodiments, a plurality of keypoints are identified on a first keyframe in a plurality of captured images. A first convex hull is computed from all keypoints in the first keyframe and merged with previous convex hulls corresponding to previous keyframes to form a convex hull union. Each keypoint is tracked from the first keyframe to a second image. The second image is adjusted to compensate for camera rotation during capture, and a second convex hull is computed from all keypoints in the second image. If the overlapping region between the second convex hull and the convex hull union is equal to, or less than, a predetermined size, the second image is designated as a new keyframe, and the convex hull union is augmented with the second convex hull.