CGR Environment Video Content Integration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current virtual reality and augmented reality systems lack the ability to seamlessly integrate video content from display devices within a physical environment into the computer-generated reality experience, limiting the enhancement and modification of virtual environments based on real-world video inputs.
Innovation Solution
A method and system that detect access to display devices within a physical environment and trigger the display of video content associated with the computer-generated reality environment, allowing for real-time modification and integration of this content into the virtual environment, thereby enhancing the user's experience by blending or framing the video content within the virtual setting.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If video content from display devices is integrated into the CGR environment, then user immersion and experience are enhanced, but rendering overhead and system complexity increase
Solution Approach 1:
The system creates a virtual representation (copy) of the physical display device and its video content within the CGR environment. Instead of directly integrating complex video processing and rendering pipelines, the patent uses semantic segmentation to identify display devices and generates virtual counterparts that replicate their visual output, thereby enhancing immersion while managing rendering complexity through abstraction.
Solution Approach 2:
The patent introduces an intermediary processing layer that acts as a mediator between the physical display devices and the virtual environment. This intermediary system uses machine learning models to detect and segment display devices, then generates virtual representations that bridge the physical and virtual domains, reducing the direct computational burden while maintaining integration quality.
2Ease of operation
If display devices are detected and video content is triggered in real-time, then user interaction and dynamic experience improve, but processing time and computational resources increase
Solution Approach 1:
The system performs preliminary detection and classification of display devices using pre-trained machine learning models. By having the semantic segmentation model ready and pre-configured, the system can quickly identify and process display devices as they enter the field of view, reducing real-time processing delays while maintaining responsive interaction.
Solution Approach 2:
The patent replaces traditional computer vision and image processing mechanisms with machine learning-based semantic segmentation. This substitution enables more efficient real-time detection and classification of display devices, reducing computational overhead and processing time while improving the speed and accuracy of real-time interactions.
3Reliability
If the CGR environment is modified based on video content, then virtual environment realism and user perception are enhanced, but system complexity and computational load increase
Solution Approach 1:
The system applies local quality enhancement by modifying only the portions of the CGR environment that correspond to detected display devices. Instead of globally restructuring the entire virtual environment, the patent uses semantic segmentation masks to locally generate and integrate video content representations, thereby enhancing realism in specific regions while maintaining overall system simplicity.
Solution Approach 2:
The patent segments the visual field to identify and separate display devices from other objects. By dividing the environment into distinct semantic regions, the system can selectively process and integrate video content only where needed, reducing the complexity of environment modification while improving targeted realism and user perception.
Data Source
AI summary
In some implementations, a method is performed at a device including non-transitory memory and one or more processors. The method includes: presenting, by the device, a computer-generated reality (CGR) environment; while presenting the CGR environment, detecting access to a display device separate from the device; in response to determining that there is access to the display device, triggering the display of video content, associated with the CGR environment, on the display device in order to supplement the CGR environment; and modifying at least a portion of the CGR environment based on the video content displayed by the display device. In some implementations, the display device is present within a physical environment, and the device enables video pass-through or optical see-through of at least a portion of the physical environment including the display device.