Real-Time Video Matting With ML Depth and Mask Compositing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for combining live-action video with computer-generated elements, such as green screens and motion capture, require expensive equipment, complex setups, and significant processing power, limiting real-time adjustments during filming.
Innovation Solution
A real-time matting system using machine learning models to generate image masks and depth estimates, allowing for real-time composition of live-action video with computer-generated elements on a single computing device.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If green screen or motion capture is used to compose live-action video with computer-generated elements, then the compositing quality is improved, but the equipment cost and setup complexity increase significantly
Solution Approach 1:
The patent replaces mechanical/optical systems (green screens, motion capture equipment) with a machine learning-based computational system. The ML model processes video frames to automatically generate masks and depth estimates, eliminating the need for physical green screens and complex motion capture setups while achieving comparable or superior compositing results
Solution Approach 2:
The system creates a computational copy of the scene understanding process through trained machine learning models. Instead of using physical props (green screens) or specialized equipment (motion capture volumes), the system learns to infer scene structure, segmentation, and depth from standard video input, replicating the functionality of complex hardware systems through software
2Manufacturing precision
If green screen or motion capture is used for compositing, then the compositing quality is improved, but the processing time and computational resources increase significantly
Solution Approach 1:
The machine learning models are trained in advance on large datasets to learn scene understanding, segmentation, and depth estimation. This preliminary training enables the system to perform real-time compositing during filming without requiring lengthy post-processing computations, as the heavy lifting of learning scene structure has already been completed during model training
Solution Approach 2:
The system transforms the compositing problem from traditional parameter-based approaches (requiring precise calibration of green screen colors, motion capture markers, etc.) to a learning-based approach where the model automatically adapts to different scenes and lighting conditions, enabling faster processing without sacrificing quality
3Ease of operation
If software applications are used to add computer-generated elements to video captured on personal devices, then the ease of use is improved, but the ability to integrate with professional video equipment and production pipelines is reduced
Solution Approach 1:
The patent creates a universal compositing system that can process video from multiple sources including personal devices and professional video equipment. The machine learning model accepts standard video input formats and outputs standardized masks and depth estimates that can be integrated into various production pipelines, making the system adaptable to both consumer and professional applications without requiring separate specialized software for each device type
Data Source
AI summary
The disclosed matting technique comprises receiving a video feed comprising a plurality of temporally ordered video frames as the video frames are captured by a video capture device, generating, using one or more machine learning models, an image mask corresponding to each video frame included in the video feed and a depth estimate corresponding to each video frame included in the video feed, and, for each video frame in the video feed, transmitting, in real-time, the video frame, the corresponding image mask, and the corresponding depth estimate to a compositing system. The compositing system composites a computer-generated element with the video frame based on the corresponding image mask and the corresponding depth estimate.