CGR Event System for Unified Input Modality Interpretation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems require input-modality-specific instructions to interpret user interactions, which can be burdensome for app developers and may not allow for quick and realistic responses, especially in complex CGR environments like VR and MR.
Innovation Solution
A separate event system that detects and interprets user interactions across various input modalities, abstracting them into higher-level, input-modality-independent events, allowing apps to respond without needing specific instructions for each input modality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If apps include input-modality-specific instructions to detect and interpret user interactions, then the app can respond to specific input modalities, but the device complexity and development burden increase significantly
Solution Approach 1:
The patent introduces an event system as an intermediary layer between the input modalities and the apps. This event system detects and interprets user interactions from various input modalities (mouse, keyboard, touch, voice, gaze, hand movement) and translates them into unified events that apps can consume. This mediator eliminates the need for apps to include input-modality-specific instructions, while still providing comprehensive input modality support.
Solution Approach 2:
The event system is designed to handle multiple input modalities through a single unified interface. Instead of requiring separate handling logic for each input modality, the event system provides a universal mechanism that can interpret and translate any input modality into standard events, reducing complexity while maintaining versatility.
2Adaptability or versatility
If apps include separate logic for each input modality, then the app can recognize and respond to different interaction types, but the processing time and response speed decrease
Solution Approach 1:
The event system performs preliminary action by pre-detecting and interpreting user interactions into standardized events before passing them to the apps. This upfront processing eliminates the need for apps to perform complex interpretation logic during runtime, thereby improving response speed while maintaining comprehensive interaction recognition capability.
3Measurement precision
If the system uses input-modality-specific instructions, then the interpretation can be precise for each modality, but the overall system complexity and maintenance burden increase
Solution Approach 1:
The event system serves as a specialized intermediary that handles the complexity of input modality-specific interpretation. It maintains precise interpretation capabilities for each modality while shielding the apps from this complexity, allowing apps to work with simplified, standardized events without needing to understand the nuances of different input modalities.
Data Source
AI summary
Various implementations disclosed herein include devices, systems, and methods that provide a CGR environment in which virtual objects from one or more apps are included. User interactions with the virtual objects are detected and interpreted by a system that is separate from the apps that provide the virtual objects. The system detects user interactions received via one or more input modalities and interprets those user interactions as events. These events provide a higher-level, input modality-independent, abstractions of the lower-level input-modality dependent user interactions that are detected. The system uses UI capability data provided by the apps to interpret user interactions with respect to the virtual object provided by the apps. For example, the UI capability data can identify whether a virtual object is moveable, actionable, hover-able, etc. and the system interprets user interactions at or near the virtual object accordingly.