XR Input Recognition via OS-Mediated Privacy Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing user interaction systems fail to effectively facilitate interactions in extended reality environments while preserving user privacy, particularly in handling gaze and hand gestures, and integrating with legacy 2D systems.
Innovation Solution
An operating system process interprets user activity data outside of the application process, selectively providing interaction event data to the application, excluding raw gaze data and associated with intentional user actions, and using a simulation process to position UI elements in a 3D coordinate system, thus preserving privacy by limiting access to sensitive information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If raw gaze data and complete user activity data are provided to applications, then input recognition accuracy is improved, but user privacy is compromised
Solution Approach 1:
The patent extracts only the necessary interaction event data from the complete user activity data stream, separating useful interaction information from sensitive continuous gaze data. The OS process extracts and provides only discrete interaction events (such as hand gestures indicating intent to interact) to applications, while retaining continuous gaze data locally to maintain privacy.
Solution Approach 2:
The operating system process acts as an intermediary between the sensor system and applications. It receives complete user activity data including raw gaze data, processes it to identify intentional interactions, and provides only filtered interaction event data to applications. This intermediary layer protects user privacy by preventing direct access to sensitive continuous gaze data while still enabling accurate input recognition.
2Adaptability or versatility
If continuous gaze data is shared with applications, then gaze-based interaction functionality is enabled, but user identification and tracking privacy is compromised
Solution Approach 1:
The patent extracts only discrete interaction events from continuous gaze data streams, removing the continuous tracking information that enables user identification. Applications receive only isolated interaction events (such as when a hand gesture indicates intent to interact with a UI element) rather than continuous gaze trajectories, preventing user profile building while maintaining interaction functionality.
Solution Approach 2:
The patent segments continuous gaze data into discrete interaction events based on detected user intent. Rather than providing continuous streams of gaze data, the system segments the data flow to provide only those portions corresponding to intentional interactions, thereby eliminating continuous user tracking while preserving functional gaze-based interactions.
3Measurement precision
If all user activity data is provided to applications, then complete input recognition is achieved, but system complexity and data management burden increase
Solution Approach 1:
The patent extracts only the necessary interaction event data from the complete user activity data stream, separating useful interaction information from sensitive continuous gaze data. The OS process extracts and provides only discrete interaction events (such as hand gestures indicating intent to interact) to applications, while retaining continuous gaze data locally to maintain privacy.
Solution Approach 2:
The patent segments continuous gaze data into discrete interaction events based on detected user intent. Rather than providing continuous streams of gaze data, the system segments the data flow to provide only those portions corresponding to intentional interactions, thereby eliminating continuous user tracking while preserving functional gaze-based interactions.
Data Source
Figure 1A~1B
Figure 2~3
Figure 4~6
AI summary
Devices, systems, and methods that interpret user activity as user interactions with virtual elements (e.g., UI elements) positioned within in a 3D space such as an XR environment. Some implementations use an architecture that receives application UI geometry in a system or shared simulation area and outputs interaction events data (e.g., less than all user activity data) for an application to use to recognize input to the application. An OS process may be configured to provide an input support process to support recognizing input intended for one or more separately-executing applications, for example, by providing some input recognition tasks to recognize user activity as input for the applications or by converting user activity data into a format that can be more easily, accurately, efficiently, or effectively interpreted by the applications and/or in a way that facilitates preservation of user privacy.