Video Game Engine Compositing for Consistent Multi-Camera Backgrounds

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Traditional chroma keying techniques face issues with background consistency when filming at multiple angles, and filming against LED video screens is limited to single-camera setups, causes pixelization, and complicates post-production, especially when combining foreground and background elements.

Innovation Solution

A multi-camera virtual studio system is used to film actors in a studio with a green screen, while simultaneously capturing background plates at the same angles and positions, which are then composited with real-time adjustments using a video game engine to generate consistent and realistic backgrounds, allowing for seamless integration of foreground and background elements.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If traditional chroma keying techniques are used to replace backgrounds, then background flexibility is improved, but background consistency deteriorates when filming at multiple angles

Engineering Contradiction:

Improvebackground flexibilityVSAvoidbackground consistency

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The patent uses background plates captured from the same multi-camera setup to create consistent backgrounds. Instead of relying on chroma keying replacement, the system captures actual background footage from multiple camera angles simultaneously, ensuring that background elements remain consistent and in correct positions across all camera views.

Inventive Principle:

Principle #26Copying

2Reliability

If filming is done on location to ensure background consistency, then background reliability is improved, but production cost and complexity increase

Engineering Contradiction:

Improvebackground consistencyVSAvoidproduction complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The patent separates the filming process into two independent segments: foreground actors filmed in a controlled studio environment with green screens, and background plates captured separately (either on location or in studio). This segmentation allows each element to be optimized independently - actors can be filmed with consistent lighting and audio control, while backgrounds are captured with proper spatial relationships across multiple camera angles.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces background plates as an intermediary element that mediates between the studio-filmed foreground and the final composite scene. These plates serve as the reference background that maintains consistency across multiple camera angles, eliminating the need to transport entire crews to location while preserving background reliability.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Illumination intensity

If LED video screens are used for backgrounds, then visual quality is improved, but camera movement capability deteriorates due to tracking delay

Engineering Contradiction:

Improvevisual qualityVSAvoidcamera movement capability

Core Design Contradiction:

Illumination intensityVSEase of operation

Solution Approach 1:

Instead of relying on real-time LED screen rendering that suffers from tracking delays, the patent uses pre-captured background plates that are composited onto the foreground footage. This approach eliminates the need for real-time tracking and rendering, allowing free camera movement without the picture jumping issue inherent in LED wall systems.

Inventive Principle:

Principle #26Copying

4Adaptability or versatility

If LED video screens are used for backgrounds, then scene generation capability is improved, but post-production flexibility deteriorates

Engineering Contradiction:

Improvescene generation capabilityVSAvoidpost-production flexibility

Core Design Contradiction:

Adaptability or versatilityVSEase of manufacture

Solution Approach 1:

The patent separates foreground and background into distinct layers that can be independently manipulated in post-production. Background plates are captured separately and composited onto foreground footage, allowing independent color correction, adjustments to ambient lighting, fog, and other effects to be applied to either layer without affecting the other. This maintains full post-production flexibility that is lost when foreground and background are filmed together on LED screens.

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS12407788B2Video game engine assisted virtual studio production process

Publication Date: 2025.09.02 PIPHER TIM

AI summary

A production process involves predetermined number of cameras simultaneously filming a background at predetermined angles, and filming actors in a studio with the same number of cameras and the same angles, used in conjunction with a virtual studio system. In studio, the actors perform before a green screen and the virtual studio system composites the actors onto the background in real-time. Camera tracking allows the in-studio cameras to pan, tilt, focus, zoom, and make limited other movements as the virtual studio system adjusts display of the background in a corresponding manner, resulting in a realistic scene without transporting actors and crew to the background location.