Virtual Object Binding Across Screens Using Gesture Recognition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current air-gesture technologies for computing input lack versatility in manipulating virtual objects across multiple devices and screens, especially when integrating touch and air gestures for seamless cross-device interactions.
Innovation Solution
A system utilizing 3D cameras and touch screen sensors to track user body movements, mapping input data to control inputs that allow virtual objects to be moved between screens or to a bezel region, with virtual bindings providing visual feedback and dynamic behavior, enabling gestures like pinching, lifting, and releasing to control object manipulation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If air-gesture technologies are used for computing input, then body motion can be detected as modal input, but the versatility in manipulating virtual objects across multiple devices and screens is limited
Solution Approach 1:
The patent merges touch-screen gesture detection with air-gesture recognition by combining touch sensor data with 3D camera tracking. The system integrates both modalities to enable seamless cross-device object manipulation, allowing users to pick up objects from one screen and place them on another using combined touch and air gestures.
Solution Approach 2:
The system implements universal gesture recognition that works across multiple device types and screen configurations. The gesture recognition software can identify the same gestures whether performed on a single touchscreen, across multiple screens, or using air gestures, providing consistent interaction behavior across different computing platforms.
2Ease of operation
If 3D camera technology is used to track body movements, then air gestures can be detected, but the precision in mapping input data to control inputs for cross-device interaction is reduced
Solution Approach 1:
The patent introduces an intermediary gesture recognition software layer that mediates between the raw 3D camera data and the final control inputs. This software layer processes the tracked body movements, identifies gestures, and maps them to appropriate control actions across different devices, ensuring precise translation of air gestures into accurate object manipulation commands.
Solution Approach 2:
The system implements feedback mechanisms where the gesture recognition software continuously refines its interpretation of body movements based on contextual information from multiple screens and devices. This iterative feedback process improves the precision of gesture-to-control mapping by adjusting interpretations based on real-time system state and user intentions.
3Adaptability or versatility
If virtual objects are moved beyond the display using gestures, then cross-device interaction is enabled, but the visual feedback and binding mechanisms become more complex
Solution Approach 1:
The patent segments the visual feedback into distinct components: virtual bindings that connect user gestures to objects, and display updates that show object movement across screens. This segmentation allows each component to be optimized independently, reducing overall system complexity while maintaining rich visual feedback for cross-device interactions.
Data Source
AI summary
A device includes a processor to receive input data from an image detector, where the input data includes data obtained from tracking air movements of a user's body part interacting with a virtual object of the electronic display, the processor to map the input data to a control input to move the virtual object beyond the display. The device could, for example, include a mobile device such as a smartphone or a laptop. The virtual object could for example move to another display or to a bezel of the device. A touch screen sensor may allow the virtual object to be pinched from the display, before being lifted beyond the display. The processor may map the input data to control input to create a virtual binding of the virtual object in order to create a visual rendering of a connection between the virtual object and the user's body part.