Decoder-Based Bundle Adjustment for Smoother 3D Reconstruction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing 3D reconstruction methods using camera arrays result in videos with significant jitter and shaking due to inaccuracies in key point location, camera calibration, and re-projection errors, leading to poor subjective quality, especially when scenes have moving objects.
Innovation Solution
Utilize decoder data from decoded 2D videos to refine 3D point coordinates during bundle adjustment by minimizing the difference between original and re-projected 2D image points, incorporating motion vectors and macroblock locations to improve the subjective quality of the reconstructed video.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If traditional 3D reconstruction methods are used, then the reconstruction process is simple, but the video quality suffers from significant jitter and shaking
Solution Approach 1:
The patent applies preliminary action by performing decoder-based bundle adjustment before final 3D reconstruction. The method pre-refines 3D point locations using decoder data from multiple views to minimize reprojection errors, thereby improving the accuracy of subsequent rendering and reducing jitter in the reconstructed video without adding complexity to the overall reconstruction pipeline
Solution Approach 2:
The patent introduces decoder data as an intermediary element between traditional bundle adjustment and final 3D reconstruction. This decoder data serves as a mediator that provides refined 2D point correspondences across multiple views, enabling more accurate 3D point estimation without requiring direct modification of the core bundle adjustment algorithm
2Manufacturing precision
If more refinement steps are added to improve 3D point accuracy, then video quality improves, but processing time increases
Solution Approach 1:
The patent changes the parameters used in bundle adjustment by incorporating decoder data from multiple views instead of relying solely on traditional feature matching. This parameter change enables the system to achieve higher 3D point accuracy by minimizing reprojection errors across views, while the efficient use of decoder data prevents excessive processing time increases
3Productivity
If traditional bundle adjustment is used, then processing is fast, but jitter and shaking are visible in the reconstructed video
Solution Approach 1:
The patent applies universality by making the bundle adjustment process multi-functional: it simultaneously optimizes for both processing efficiency and motion accuracy. The decoder-based approach processes decoder data from multiple views in parallel, maintaining fast processing speeds while the multi-view optimization ensures high motion accuracy and reduces visible jitter in the reconstructed video
Data Source
AI summary
Example systems, articles and methods to improve subjective quality of 3D reconstruction are disclosed. An example method disclosed herein includes obtaining image data of decoded video frame sequences of a same scene from multiple perspectives. The example method also includes generating a three dimensional (3D) structure of 3D points based on the image data. The example method further includes refining locations of the 3D points in the 3D structure based on a decoder-based bundle adjustment, the decoder-based bundle adjustment to refine the locations based on decoder prediction data.