Multi-touch Gesture Translation for Keyboard UI Remoting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Users experience input inaccuracies and reduced productivity when navigating mouse-and-keyboard based UIs using a multi-touch display surface during remoting sessions, as the multi-touch input requires approximation of multiple (x, y) coordinates to a single point, leading to cumbersome and time-consuming interactions.
Innovation Solution
The system translates multi-touch events into mouse-and-keyboard commands by receiving touch events on a multi-touch display surface, processing them through an agent that communicates with the host machine to emulate mouse and keyboard inputs, allowing for precise interaction with mouse-and-keyboard based UIs, including gestures for left-click, right-click, zooming, and cursor control.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If a multi-touch display surface is used for input, then ease of operation is improved, but measurement precision deteriorates due to approximation of multiple coordinates to a single point
Solution Approach 1:
The patent introduces an intermediary translation layer that converts multi-touch gestures into precise mouse coordinates. The system detects touch gestures on the multi-touch display surface and translates them into accurate cursor positions, allowing the user to interact with mouse-and-keyboard based UIs without direct coordinate approximation errors.
Solution Approach 2:
The system changes the parameter representation from raw multi-touch coordinates to translated mouse coordinates. By processing touch events through gesture recognition and coordinate transformation algorithms, the system converts the approximate single-point touch input into precise multi-point coordinate data that matches the expected mouse input format.
2Measurement precision
If constant zooming is used to improve input accuracy, then measurement precision is improved, but loss of time increases due to time-consuming zooming operations
Solution Approach 1:
The system performs preliminary coordinate translation when the user first interacts with the UI. By pre-processing the coordinate system mapping and establishing accurate pointer positions in advance, the system eliminates the need for users to repeatedly zoom in and out during interaction, saving time while maintaining precision.
Data Source
AI summary
A mouse-and-keyboard based user interface is updated based on gestures made on a touch screen that is displaying the mouse-and-keyboard based user interface. The user interface update process includes the steps of receiving one or more touch events in response to a gesture made on the touch screen, translating the touch events to a mouse-and-keyboard based command, transmitting the mouse-and-keyboard based command to an operating system, and receiving an updated display in response thereto.