Volumetric Video Deformation Fields for Temporal Coherence
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for generating volumetric video often fail to maintain temporal coherence over multiple frames, leading to discontinuities and the need for frequent re-storing of new topologies and atlases, making it difficult to integrate into animation pipelines and requiring repeated artifact fixing and augmentation.
Innovation Solution
A system and method using neural networks to generate deformation fields between frames by building texture implicit functions and training neural networks with matching losses, allowing for the generation of temporally coherent sequences with shared topologies and atlases.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional frame-by-frame meshing and dragging techniques are used, then temporal coherence can be maintained for a limited number of frames, but the technique fails after a relatively small number of frames requiring frequent re-initialization of topology and atlas
Solution Approach 1:
The system performs preliminary actions by establishing a global deformation field that spans the entire volumetric video sequence from the beginning. Instead of reacting to coherence breakdown after a few frames, the invention proactively creates a comprehensive deformation model that anticipates and maintains coherence throughout the entire video duration, preventing the need for frequent re-initialization.
Solution Approach 2:
The deformation field serves multiple functions simultaneously: it maintains temporal coherence across all frames, enables consistent topology throughout the sequence, and provides a unified framework for rendering. This multi-functional approach eliminates the need for separate processing for each frame or re-initialization events, allowing the system to handle the entire volumetric video as a single coherent sequence.
2Reliability
If new topology and atlas are stored periodically, then temporal coherence can be reset, but storage requirements increase and discontinuities are introduced in the volumetric video
Solution Approach 1:
The invention merges the topology and atlas information into a single consistent global deformation field that applies across all frames. Instead of storing separate topology and atlas data for each re-initialization point, the system combines these elements into one unified representation, significantly reducing storage requirements while maintaining temporal coherence throughout the entire sequence.
Solution Approach 2:
The global deformation field ensures continuous temporal coherence across the entire volumetric video without interruption or discontinuity. By maintaining a consistent deformation model from the first frame to the last, the system eliminates the need for periodic resets that would otherwise introduce discontinuities and require additional storage for multiple topology versions.
3Reliability
If frequent re-initialization of topology occurs, then temporal coherence is restored, but integration into animation pipelines becomes difficult and repeated artifact fixing is required
Solution Approach 1:
The system performs all necessary topology establishment and coherence setup in advance through the global deformation field, before the volumetric video processing begins. This preliminary configuration eliminates the need for repeated artifact fixing and pipeline re-initialization during processing, as the coherence framework is already in place for the entire sequence.
Solution Approach 2:
The global deformation field serves as a universal framework that works consistently across all frames and integrates seamlessly with animation pipelines. This single coherent system replaces the need for multiple separate processing stages and artifact fixing operations, simplifying pipeline integration while maintaining temporal coherence throughout.
Data Source
AI summary
Systems and methods for generating a volumetric video in which all frames are temporally coherent. Method for generating a temporally coherent plurality of frames in a volumetric video, including finding a deformation field transforming a source frame of the volumetric video into a target frame of the volumetric video including building a texture implicit function for said target frame; training a neural network to generate said deformation field between said source frame and said target frame, said texture implicit function for said target frame being a texture matching loss for said neural network; applying said deformation field to said source frame, thereby generating a deformed source frame; selecting a plurality of frames in said volumetric video; selecting a first source frame in said plurality of frames; repeating the steps of finding said deformed source frame for each target frame, said finding of said deformed source frame following one of the following paths: finding a first deformed source frame from said first source frame; and each subsequent source frame being a deformed source frame from a previous pair of frames; or each of said target frames being a deformation of said first source frame.