LED Wall Image Augmentation for Camera Pose Mismatch
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge in creating digitally augmented camera images using LED walls is the mismatch between the rendered scenery image data and the actual camera pose due to the delay in rendering and display, leading to visible artifacts, especially when camera movements are fast.
Innovation Solution
The method involves deriving augmentation data by projecting scenery image data rendered for a first camera pose onto an extended 3D model of the LED wall, considering the change in camera perspective between the first and second poses, and using a pipeline process to continuously re-render for consecutive camera poses.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If scenery image data is rendered for a first camera pose and displayed on the LED wall, then the LED wall shows virtual content, but when the camera moves to a second pose, the perspective mismatch causes artifacts in the augmented image
Solution Approach 1:
The patent applies preliminary action by predicting future camera poses and pre-rendering the corresponding LED wall image data in advance. The rendering system receives predicted camera pose information and renders the appropriate virtual content before the camera actually reaches that position, eliminating the perspective mismatch that would otherwise occur due to rendering delays.
Solution Approach 2:
The patent implements dynamics by making the rendering system adaptive to changing camera conditions. Instead of static rendering for a fixed pose, the system continuously updates the rendered LED wall image data based on predicted camera motion, adjusting the virtual content perspective dynamically to match the camera's movement trajectory.
2Ease of manufacture
If the LED wall dimensions are limited due to budget constraints, then cost is reduced, but the LED wall cannot fill the entire background frame
Solution Approach 1:
The patent applies dimensionality change by extending the LED wall content into the virtual space beyond the physical LED wall boundaries. Through digital augmentation, the system projects additional virtual content into the background areas not covered by the physical LED wall, effectively expanding the visible virtual environment from two dimensions to three-dimensional space.
Solution Approach 2:
The patent uses digital augmentation as an intermediary between the physical LED wall and the final augmented image. The digital augmentation component fills the gap between the limited physical LED wall area and the full background frame requirement, mediating the transition from constrained hardware to comprehensive visual coverage.
3Adaptability or versatility
If chroma keying is used with green screens, then virtual content can be inserted, but color non-uniformity and lighting adjustments are difficult
Solution Approach 1:
The patent applies copying by replacing the physical green screen with a digital copy of the background environment displayed on the LED wall. Instead of using chroma keying to extract and replace green areas, the system renders a realistic digital background that is directly displayed, eliminating the need for color-based selection and transparent area creation.
Solution Approach 2:
The patent substitutes the mechanical lighting adjustment system with a digital rendering system. Instead of physically adjusting lights to match virtual content, the system uses computer-generated imagery with built-in lighting information that automatically matches the virtual environment, replacing complex mechanical lighting control with software-based illumination.
Data Source
Figure 1~2
Figure 3~4b
Figure 5~6
AI summary
The present invention proposes a method, preferably computer-implemented method, for creating digitally augmented camera images based on camera images captured by a camera 2, which contain image data of a LED wall 11 displaying LED wall display data, which is contained in computer-rendered scenery image data, the method comprising: rendering scenery image data for a first pose P1 of the camera 2; sending LED wall display data comprised in the scenery image data to the LED wall 11 for display; capturing, with the camera 2, a camera image containing image data of the LED wall 11 displaying the LED wall display data at a second pose P2 of the camera 2; and augmenting the camera image with image augmentation data to obtain an augmented camera image, wherein the image augmentation data is derived from the scenery image data, taking into consideration the second pose P2 of the camera 2.