Film Object Modification Using Synthetic Models for Visual Dubbing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The filmmaking process is time-consuming and expensive, and reshooting scenes is rarely feasible due to high costs and interdependence of tasks, leading to foreign language versions of films losing nuance and quality.
Innovation Solution
A computer-implemented method using a machine learning model, such as a deep neural network, to isolate and modify objects within video frames, enabling photorealistic editing and applications like visual dubbing or performance transposition.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If foreign language versions are produced using text subtitles or audio dubbing, then language localization is achieved, but nuance and quality are lost
Solution Approach 1:
The patent creates a digital twin (synthetic model) of the actor's face that can be rendered in different languages while preserving the original performance nuances. This virtual copy allows quality preservation by maintaining the actor's original expressions and emotions while enabling language adaptation through parameter modification of the synthetic model.
Solution Approach 2:
The patent modifies parameters of the synthetic model (such as mouth shape, facial expressions) to match the target language's phonetic requirements while preserving the original performance. This allows quality and nuance preservation by making targeted parameter adjustments rather than complete reperformance.
2Manufacturing precision
If scenes are reshoot for foreign language versions, then quality and nuance are maintained, but time and cost increase significantly
Solution Approach 1:
The patent creates a digital twin (synthetic model) of the actor's face that can be rendered in different languages while preserving the original performance nuances. This virtual copy allows quality preservation by maintaining the actor's original expressions and emotions while enabling language adaptation through parameter modification of the synthetic model.
Solution Approach 2:
The patent performs preliminary capture of the actor's performance and creates a synthetic model during the original production phase. This preliminary action enables rapid generation of foreign language versions without requiring additional reshooting, thus preserving quality while avoiding time loss.
3Ease of manufacture
If conventional VFX are used to modify objects in film, then some level of editing is achieved, but photorealistic quality is not attained
Solution Approach 1:
The patent replaces conventional VFX mechanical processes with a machine learning-based synthetic model rendering system. This substitution enables photorealistic quality by using trained neural networks to generate highly realistic modifications that adapt to lighting, angles, and environmental conditions automatically.
Solution Approach 2:
The patent modifies parameters of the synthetic model (such as mouth shape, facial expressions) to match the target language's phonetic requirements while preserving the original performance. This allows quality and nuance preservation by making targeted parameter adjustments rather than complete reperformance.
Data Source
AI summary
A computer-implemented method of processing video data comprising a first sequences of image frames containing a first instance of an object. The method includes isolating said first instance of the object within the first sequence of image frames, determining, using the isolated first instance of the object, first parameter values for a synthetic model of the object, modifying the first parameter values for the synthetic model of the object, rendering a modified first instance of the object using a trained machine learning model and the modified first parameter values for the synthetic model of the object, and replacing at least part of the first instance of the object within the first sequence of image frames with a corresponding at least part of the modified first instance of the object.