Physical Object Form Factor for Tactile XR Gesture Control
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In Extended Reality (XR), there is a disconnect between visual and tactile experiences, limiting user engagement and full product understanding due to the reliance on visual displays without sufficient gestural manipulation.
Innovation Solution
A system and method for gesture-based tactile interaction using the form factor of a physical object as a marker or controller, recording transform data of the object and user's hand to identify gestures, and mapping these gestures to actions for immersive interaction with virtual objects.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If visual displays are used in XR, then information can be displayed to users, but tactile interaction and user engagement are limited
Solution Approach 1:
A physical object serves as an intermediary between the user and the virtual object. The physical object's form factor and gestures mediate the interaction, allowing users to manipulate virtual objects through tangible physical actions, thereby bridging the gap between visual display and tactile interaction
Solution Approach 2:
The patent replaces traditional mechanical interaction mechanisms with gesture-based interaction. Instead of requiring physical contact with virtual objects, the system captures hand gestures and translates them into control actions for virtual objects, enabling tactile-like interaction through motion
2Ease of operation
If gestures are captured and mapped to actions, then user engagement is enhanced, but system complexity increases
Solution Approach 1:
The system creates a digital copy of the physical object's form factor and uses it as a reference for gesture recognition. By capturing gestures relative to the physical object's geometry and using pre-defined gesture libraries, the system simplifies complex motion recognition into manageable patterns
Solution Approach 2:
The system transforms continuous gesture data into discrete action parameters by mapping gesture characteristics (position, orientation, movement) to predefined action categories. This parameter transformation reduces the complexity of processing continuous gestures into actionable commands
3Ease of operation
If physical object form factor is used as marker, then tactile interaction is enabled, but device requirements increase
Solution Approach 1:
The physical object serves multiple functions simultaneously: it acts as a marker for spatial positioning, a controller for gesture input, and a reference for form factor matching. This multi-functionality reduces the need for separate devices and simplifies the overall system requirements
Data Source
AI summary
A product is traditionally experienced in XR mainly through sight that accounts for about 80% of sensory inputs, thereby impeding an immersive experience of a product. In the present disclosure, an object's form factor guides a user's gestures in carrying out actions on an augmentation. A suitable physical object marker mimics physical touch interaction with a virtual product. Leveraging the sense of touch, the user gains deeper understanding of the virtual product, since the product responds to the user's natural gestures. In accordance with the present disclosure, transform data in the form of a position, an orientation and a scale associated with the physical object and user's hands handling the physical object are recorded to identify an associated gesture. Utilizing gesture to action mapping, the user's action is translated to the virtual product. The augmentation is modified in real time to provide an immersive experience.