Voice and Chat Control for GUI-Based Remote Desktops
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing virtualization systems face challenges in providing a seamless user experience for GUI-based desktops and applications on devices with small displays or no display, such as smartphones, or in hands-free environments, due to the limitations of traditional input methods.
Innovation Solution
Implementing a voice or chat user interface using large language and action models to enable high-level declarative requests, allowing interaction with virtualized applications and desktops without the need for a display, through a LAM agent that identifies and executes actions based on textual inputs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If traditional GUI-based virtualized applications are deployed on devices with small displays or no display, then the user experience deteriorates due to inability to interact effectively, but adding display-dependent interfaces would improve usability
Solution Approach 1:
The patent replaces the mechanical/visual interaction system (GUI requiring display and manual input) with an acoustic/language-based interaction system (voice commands and natural language processing). This allows users to interact with virtualized applications on devices without displays or with limited display capabilities through speech recognition and natural language interfaces, resolving the contradiction between usability and device compatibility.
2Ease of operation
If voice or chat interfaces are added to virtualized applications, then hands-free interaction is enabled improving ease of operation, but system complexity increases due to integration of LLM and LAM
Solution Approach 1:
The patent introduces an intermediary layer consisting of a Large Language Model (LLM) and a Large Action Model (LAM) that sits between the user's voice/chat input and the virtualized application. The LLM processes natural language input and translates it into structured commands, while the LAM executes these commands within the application context. This intermediary architecture enables hands-free interaction while managing system complexity through specialized AI components that handle language processing and action execution separately.
3Measurement precision
If LAM agent is deployed to identify and execute actions based on textual input, then task completion accuracy is improved, but processing time increases due to model inference
Solution Approach 1:
The patent implements preliminary action by pre-processing and structuring the action space before LAM inference. The system identifies available actions, their parameters, and valid combinations in advance, creating a structured action space that the LAM can efficiently query. This pre-organization of action information reduces the computational burden during inference, maintaining high task execution accuracy while reducing response time through optimized model querying.
Data Source
AI summary
Methods and systems for enhanced remote desktop interfaces are described. A computing system may train, using historical or live information, a LAM to execute, within a remote desktop application, textual actions with their parameters if any. A user declarative request (voice or chat) may be interpreted by a LLM to match a specific action (and potentially ask for the corresponding parameters in a conversational way). Subsequently, from the textual action and its parameters, the LAM may execute the action within a remote desktop application and report the result to the user via voice or chat.