Context Menu Virtualization for Touch-Screen Remote Desktop Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Users of touch-screen devices face challenges when interfacing with traditional desktop-based applications through remote desktop protocols due to latency and the difficulty of using a 'point-and-click' interface with touch-screen gestures, as existing solutions only provide cosmetic changes or require users to learn new gestures without improving latency.
Innovation Solution
A method is implemented to generate a local graphical user interface on a touch screen client device connected to a server device, receiving user interface metadata and base images to create native context menu GUI elements, allowing touch input to manipulate remote desktop context menus, and transmitting input events to the server for programmatic manipulation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If traditional point-and-click interface is used on touch-screen device, then compatibility with desktop applications is maintained, but user experience deteriorates due to latency and difficulty of operation
Solution Approach 1:
The interface is segmented into two parts: a local touch-screen optimized context menu for easy interaction and a remote desktop application for execution. The context menu is generated locally from metadata received from the server, allowing touch-friendly interaction without network latency affecting the user interface responsiveness.
Solution Approach 2:
An intermediary layer is introduced between the touch-screen client and the remote desktop server. This intermediary generates native GUI elements locally based on server metadata, translating touch input into appropriate remote commands while maintaining local responsiveness and eliminating the need for direct point-and-click through the network.
2Ease of operation
If cosmetic changes like font size adjustment are applied, then visual accessibility improves, but interaction latency and operational difficulty remain unchanged
Solution Approach 1:
The mechanical point-and-click system is replaced with a local rendering system that generates native GUI elements. Instead of displaying remote bitmap images and requiring pointer input, the system renders context menus locally from metadata, enabling touch gestures to directly control the interface without network-mediated pointer translation.
3Adaptability or versatility
If gesture-to-mouse translation is implemented, then touch-screen compatibility improves, but device complexity increases and requires learning new gestures
Solution Approach 1:
The solution applies local quality by generating context menus with properties optimized for the local touch-screen environment. The GUI elements are rendered with appropriate size, positioning, and interaction characteristics for touch input, rather than simply scaling or translating the remote desktop's original elements.
4Adaptability or versatility
If remote desktop protocol is used, then access to server resources is enabled, but user interface interaction becomes inefficient for touch-screen devices
Solution Approach 1:
The server pre-generates UI metadata describing the graphical interface elements before they are needed. This metadata is transmitted to the client, which then generates the actual GUI elements locally in advance of user interaction, allowing immediate responsive touch input without waiting for remote rendering or image updates.
Data Source
AI summary
User interface virtualization describes a technique for providing a user with access to one computing device from another computing device, while translating the ergonomics of one computer's user interface style into the ergonomics of the other's. An agent running on a remote desktop may use an interface interaction API or software framework to programmatically open and manipulate a context menu on the remote desktop, and act as a proxy for a corresponding client running on a client device that accepts a “touch and swipe” style input. The client constructs and displays a native graphical user interface element at the client device that is a virtual representation of the context menu at the remote desktop.