Mixed Reality System UI Placement via Shared Coordinates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In mixed reality applications, the placement of system user interface elements such as keyboards often obstructs digital or physical objects in the environment, degrading the user experience by breaking immersion in virtual or augmented reality.
Innovation Solution
The system allows a mixed reality application to express a location in the mixed reality environment for the operating system to place system UI elements, using shared coordinate systems and APIs to control the placement, size, and orientation of these elements per frame, ensuring they do not obstruct other content and integrate well with the application.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If system UI elements are displayed in the mixed reality environment, then user input capability is enabled, but immersion in the digital environment is broken
Solution Approach 1:
The patent applies local quality by making the UI element's visual properties (transparency, opacity, color) location-dependent. The UI element dynamically adjusts its visual characteristics based on its position in the mixed reality environment, becoming more transparent when near digital content and more opaque when near physical surfaces, thus adapting to local context to minimize immersion disruption while maintaining usability
Solution Approach 2:
The patent implements dynamics by making the UI element's visual properties changeable in real-time. The system continuously monitors the UI element's position relative to digital and physical objects and dynamically adjusts transparency, opacity, and other visual characteristics during runtime, allowing the interface to adapt to changing environmental context and maintain optimal immersion quality
2Ease of operation
If virtual keyboard is displayed over digital environment content, then keyboard input is available, but content visibility is obstructed
Solution Approach 1:
The patent applies local quality by making the keyboard's visual properties location-dependent. When the keyboard is positioned over digital content, it automatically increases transparency and adjusts opacity to allow content visibility. When positioned over physical surfaces or empty space, it becomes more opaque for better visibility. This local adaptation resolves the conflict between keyboard availability and content visibility
Solution Approach 2:
The system implements feedback by continuously monitoring the keyboard's position relative to digital and physical objects in the mixed reality environment. Based on this feedback, the system automatically adjusts the keyboard's transparency and opacity settings in real-time, creating a closed-loop system that maintains optimal content visibility while preserving keyboard functionality
3Device complexity
If UI element position is fixed by operating system, then system control is simplified, but application-specific optimal placement is lost
Solution Approach 1:
The patent applies segmentation by separating the UI element placement control into two independent components: the operating system manages the UI element's functional properties (keyboard input handling, system-level behavior), while the application controls the UI element's spatial properties (position, orientation, visual appearance in 3D space). This segmentation allows both system simplicity and application adaptability to coexist
Solution Approach 2:
The patent transitions from traditional 2D UI placement to 3D spatial placement in the mixed reality environment. The application can specify the UI element's position, orientation, and visual properties in three-dimensional space, allowing optimal placement that considers depth, angle, and spatial relationships with physical and digital objects, thereby achieving high adaptability while the OS maintains simplified functional control
Data Source
AI summary
A mixed reality display system determines a shared coordinate system that is understood by a mixed reality application running on the mixed reality display system and an operating system of the mixed reality display system. The operating system can display a system user interface (UI) element in a mixed reality environment. The system UI element can be displayed at a location in a mixed reality environment. The location is specified by the mixed reality application according to the shared coordinate system. A size and orientation for displaying the system UI element may also be specified. Also, the location, size and orientation may be specified through application program interfaces (API) of the operating system. API calls may be made per frame to adjust the location, size or orientation per frame of the displayed mixed reality environment.