Neural Rendering for Real-Time Character Animation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current techniques lack effective methods for adapting high-definition, high-fidelity characters from feature animation films to real-time applications like computer-based games and previsualization, as real-time rendering engines support only linear blend skinning and lower-resolution models, requiring manual conversion which is time-consuming and labor-intensive.
Innovation Solution
A computer-implemented method using a trained machine learning model that translates 2D or 3D control points to rendered images, allowing high-resolution characters to be controlled without requiring full resolution geometry or proprietary rigs, with a perceptual loss training approach that converges more effectively than traditional methods.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If high-resolution models and textures are used for feature animation films, then character fidelity and quality are improved, but compatibility with real-time rendering engines deteriorates
Solution Approach 1:
The patent creates a copy of the high-resolution character model by training a neural network to generate high-fidelity renderings from low-resolution inputs. The neural network learns to replicate the visual appearance of high-resolution characters while being compatible with real-time rendering engines, effectively copying the visual quality without requiring the original high-resolution assets.
Solution Approach 2:
The patent transforms the character representation by changing parameters from traditional geometric models to neural network-based feature representations. This allows the system to maintain high visual fidelity while adapting to the constraints of real-time rendering engines through parameter transformation rather than direct geometric conversion.
2Manufacturing precision
If proprietary rigs and deformation algorithms are used for feature animation films, then character animation quality is improved, but compatibility with real-time rendering engines deteriorates
Solution Approach 1:
The patent replaces traditional mechanical rigging and deformation algorithms with a neural network-based system. Instead of using proprietary mechanical deformation tools, the system uses learned patterns from training data to generate animated character renderings, substituting mechanical systems with intelligent algorithms that achieve similar or superior results while being engine-agnostic.
Solution Approach 2:
The neural network learns to copy the animation effects achieved by proprietary rigs by training on rendered images generated with those rigs. This allows the system to replicate high-quality animation outcomes without requiring access to or compatibility with the original proprietary deformation algorithms.
3Adaptability or versatility
If low-resolution assets are created manually for real-time applications, then compatibility with rendering engines is improved, but production time and labor are increased
Solution Approach 1:
The system performs preliminary action by pre-training the neural network on high-resolution character data before real-time application. This preprocessing step captures the essence of high-fidelity characters in the neural network's parameters, so that during real-time rendering, high-quality results are achieved without manual asset creation, dramatically improving production efficiency.
Solution Approach 2:
The system enables self-service by allowing the neural network to automatically generate high-fidelity character renderings without requiring manual intervention for asset creation or conversion. The automated pipeline eliminates the need for artists to manually recreate characters, letting the system serve itself in generating production-ready assets.
Data Source
AI summary
Techniques are disclosed for learning a machine learning model that maps control data, such as renderings of skeletons, and associated three-dimensional (3D) information to two-dimensional (2D) renderings of a character. The machine learning model may be an adaptation of the U-Net architecture that accounts for 3D information and is trained using a perceptual loss between images generated by the machine learning model and ground truth images. Once trained, the machine learning model may be used to animate a character, such as in the context of previsualization or a video game, based on control of associated control points.