Multi-View Interactive Digital Media Representation Generation

The patent segments the 3D model generation process into two distinct phases: an offline training phase where a neural network learns from dense data (depth maps, optical flow), and an online inference phase where the trained network generates 3D models from sparse 2D images. This segmentation allows the system to benefit from dense data training without requiring dense data during actual operation, thus reducing device complexity while maintaining manufacturing precision.

The patent performs preliminary action by training the neural network offline using dense data descriptions before deployment. The pre-trained network encapsulates the complex processing requirements, allowing the runtime system to operate with reduced complexity while still achieving high 3D model accuracy through the pre-learned patterns and features.

The patent replaces traditional mechanical 3D reconstruction algorithms (which involve complex geometric computations, mesh generation, and texture mapping) with a neural network-based system. This substitution leverages parallel processing capabilities of neural networks, significantly improving processing speed while maintaining 3D model quality through the network's learned understanding of scene geometry and appearance.

The patent changes the fundamental parameters of the 3D generation process by transitioning from deterministic algorithmic parameters to probabilistic neural network parameters. The network learns optimal parameters for 3D reconstruction from training data, enabling faster processing while maintaining or improving model quality through data-driven parameter optimization rather than fixed algorithmic rules.

The patent applies partial action by capturing images from a limited set of viewpoints rather than completing a full 360-degree capture. The neural network compensates for the missing views by inferring unseen portions from the captured images, allowing the system to achieve comprehensive adaptability with reduced capture time by performing only partial capture actions.

The patent uses the neural network to create virtual copies of viewpoints that were not physically captured. By learning the scene structure from limited captured images, the network generates synthetic images representing uncaptured angles, effectively copying the appearance and geometry of unseen portions without requiring physical capture, thus maintaining adaptability while reducing capture time.