Dynamic Scene View Synthesis Using Segmented MLP Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing novel view synthesis methods struggle to handle dynamic scenes with moving objects, as they assume static scenes and lack the ability to model temporal changes, making it difficult to render views from different camera poses and times.
Innovation Solution
A deep neural network (MLP) represents dynamic scenes as a continuous function of both space and time, using two separate MLPs to optimize static and dynamic elements, with blending weights to combine their outputs for rendering, and employs volume tracing for rendering images from novel views and times.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If existing novel view synthesis methods are used, then rendering from multiple synchronized views is achieved, but they cannot handle dynamic scenes with moving objects
Solution Approach 1:
The scene representation is segmented into two separate MLP networks: one dedicated to static scene elements and another to dynamic objects. This segmentation allows each network to specialize in its respective domain, enabling accurate handling of dynamic scenes while maintaining reliable scene representation through division of functional responsibilities.
Solution Approach 2:
The patent introduces temporal dimension and scene flow fields to transform the static NeRF representation into a dynamic one. By incorporating time-dependent scene flow that predicts 3D motion of objects between frames, the system adapts to handle moving objects while maintaining geometric consistency, thus improving adaptability without sacrificing reliability.
2Device complexity
If a single MLP represents the entire dynamic scene, then simplicity is maintained, but accuracy in representing both static and dynamic elements is reduced
Solution Approach 1:
The scene representation is segmented into two separate MLP networks: one dedicated to static scene elements and another to dynamic objects. This segmentation allows each network to specialize in its respective domain, enabling accurate handling of dynamic scenes while maintaining reliable scene representation through division of functional responsibilities.
3Productivity
If per-frame depth prediction is used, then processing speed is maintained, but ability to leverage temporal information across frames is lost
Solution Approach 1:
The system performs preliminary action by predicting 3D scene flow fields that describe object motion between frames. This preliminary motion prediction enables the model to anticipate and compensate for temporal changes, allowing it to leverage temporal information across frames while maintaining efficient processing through the continuous function representation.
Solution Approach 2:
The patent establishes continuity of useful action by representing the scene as a continuous function of space and time. This continuous representation allows temporal information to be leveraged across frames seamlessly, enabling novel time synthesis while maintaining processing efficiency through the unified NeRF framework that handles both spatial and temporal dimensions coherently.
Data Source
AI summary
Embodiments of the technology described herein, provide a view and time synthesis of dynamic scenes captured by a camera. The technology described herein represents a dynamic scene as a continuous function of both space and time. The technology may parameterize this function with a deep neural network (a multi-layer perceptron (MLP)), and perform rendering using volume tracing. At a very high level, a dynamic scene depicted in the video may be used to train the MLP. Once trained, the MLP is able to synthesize a view of the scene at a time and/or camera pose not found in the video through prediction. As used herein, a dynamic scene comprises one or more moving objects.