Cross-device gesture mapping without app code modification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional cross-device interaction methods are limited by the need to modify application source code and provide a fixed, pre-defined set of user interface actions, lacking flexibility and intuitiveness for users.
Innovation Solution
A cross-device interface module that allows users to define and demonstrate user interface actions on a primary device and corresponding gestures on an accessory device, enabling flexible and intuitive interaction by analyzing and mapping user-demonstrated actions and gestures to emulate UI actions without modifying existing applications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If conventional cross-device interaction methods are used with pre-defined mapping, then implementation is straightforward, but user flexibility and intuitiveness are limited
Solution Approach 1:
The system performs preliminary action by automatically analyzing and modeling UI actions before actual cross-device interaction occurs. The UI action model is created in advance through automated analysis of UI element hierarchies and interaction patterns, eliminating the need for manual pre-definition of mappings while maintaining system simplicity.
Solution Approach 2:
The system enables self-service by allowing UI actions to be automatically analyzed and modeled without human intervention. The automated analysis system examines UI element hierarchies, determines action types, and creates actionable models independently, freeing users from complex configuration tasks while preserving flexibility.
2Adaptability or versatility
If application source code is modified to enable cross-device interaction, then interaction capability is achieved, but development cost and complexity increase
Solution Approach 1:
The system introduces an intermediary layer consisting of the automated analysis system and UI action model that sits between the existing application and the cross-device interaction framework. This intermediary enables cross-device functionality without modifying application source code, as the analysis occurs at the UI element level rather than requiring internal application changes.
Solution Approach 2:
The system creates a copy or representation of the UI action through automated analysis and modeling. Instead of modifying the original application, the system generates an actionable model that replicates the essential structure and behavior of UI interactions, allowing cross-device control to work with this model rather than the original application code.
3Device complexity
If fixed pre-defined UI actions are provided, then system implementation is simple, but user intuitiveness and flexibility are reduced
Solution Approach 1:
The system transitions from static pre-defined UI actions to dynamic automatically analyzed actions. The automated analysis system adapts to different applications and UI structures by examining actual UI element hierarchies and interaction patterns in real-time, allowing the system to remain simple while providing customized actions tailored to each specific application context.
Solution Approach 2:
The system changes parameters by automatically determining action types, target UI elements, and relevant parameters based on analyzed UI structures. Instead of using fixed pre-defined parameters, the system dynamically adjusts action parameters according to the specific application being controlled, enabling versatility without increasing overall system complexity.
Data Source
AI summary
A method includes identifying a user interface (UI) action demonstrated by a user for an application (104) executed at a first electronic device (102) and identifying a gesture demonstrated by a user using a second electronic device (108, 109, 110) as a gesture intended by the user to trigger the UI action for the application at the first electronic device. In response to detecting a subsequent instance of the gesture at the second electronic device, the method includes triggering an instance of the UI action for the application at the first electronic device.