XR Synthetic Object Replacement for Real-Time Display Alignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing extended reality (XR) technologies struggle to accurately integrate physical objects, such as electronic devices, into virtual environments, lacking the ability to seamlessly replicate the dynamic information displayed on these objects within the XR experience.
Innovation Solution
A method for replacing a portion of a synthetic object within an XR environment with a corresponding portion of a physical object or its virtual representation, using image capture, orientation data, and transformation matrices to align and update the displayed information in real-time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If synthetic objects are used in XR environments, then the virtual world can be rendered, but the dynamic information of physical objects cannot be accurately reflected
Solution Approach 1:
The patent creates a virtual copy (synthetic object) of the physical object in the XR environment and continuously updates this copy with real-time data from the physical object via image capture and processing. This copying approach allows the virtual representation to reflect the dynamic information of the physical object, resolving the contradiction between rendering virtual content and maintaining information accuracy.
Solution Approach 2:
The system implements a feedback loop where the physical object's state is continuously captured through image processing, transformed into updated visual information, and fed back to refresh the synthetic object in the XR environment. This closed-loop feedback mechanism ensures the virtual representation remains synchronized with the physical object, improving both information accuracy and verisimilitude.
2Productivity
If real-time image processing is implemented, then dynamic information can be captured, but computational complexity increases
Solution Approach 1:
The patent segments the processing task by focusing specifically on capturing and processing the display panel region of the physical object, rather than processing the entire object or scene. This segmentation reduces computational complexity while still achieving real-time capture of dynamic information, as the system only needs to process relevant portions of the captured image.
Solution Approach 2:
The system extracts only the essential dynamic information (display panel content) from the captured image of the physical object, separating this critical data from the rest of the object's visual information. This extraction approach minimizes processing requirements while maintaining real-time updating capability, as only the extracted display information needs to be processed and transferred to the synthetic object.
3Reliability
If the display panel is continuously updated, then verisimilitude improves, but energy consumption increases
Solution Approach 1:
The patent implements periodic updates of the synthetic object's display panel based on captured images of the physical object, rather than continuous real-time streaming. The system captures images at specific intervals or when changes are detected, processes these periodic updates, and refreshes the virtual display accordingly. This periodic action maintains verisimilitude by keeping the virtual representation synchronized with the physical object while significantly reducing energy consumption compared to continuous updating.
Data Source
AI summary
A physical object is detected within one or multiple images captured by a camera of a head-mountable display (HMD). It is determined that the physical object is represented as a synthetic object within an extended reality (XR) environment of one or multiple images displayed by a display panel of the HMD. A portion of the synthetic object within the XR environment of the images displayed by the display panel of the HMD is replaced based on a corresponding portion of the physical object within the images captured by the camera.