Cross-Context Task Automation with Domain-Agnostic Action Embeddings
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing computer applications require users to repeatedly perform semantically-similar tasks across different contexts, which can be cumbersome, prone to errors, and consume unnecessary computing resources, and recorded sequences of actions are often narrowly tailored and complex.
Innovation Solution
Implement a method to capture sequences of actions and natural language inputs, abstract them into a domain-agnostic action embedding space, and use domain models to translate these embeddings across different contexts, enabling seamless automation without programming knowledge.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If recorded sequences of actions (macros) are used to automate tasks within a particular computer application, then task automation is achieved, but the sequences are constrained to operation within that particular application and are narrowly tailored to very specific contexts
Solution Approach 1:
The patent creates a universal action embedding space that can represent tasks across multiple computer applications and domains. Domain models translate between application-specific action spaces and this universal embedding space, enabling a single recorded task to be adapted and executed across different applications without requiring separate macros for each context.
Solution Approach 2:
The patent introduces action embeddings and domain models as intermediary layers between the recorded task and the target application. The action embedding serves as a domain-agnostic representation that mediates between the source task and various target applications, while domain models act as translators between the universal embedding space and application-specific action spaces.
2Extent of automation
If traditional scripting languages are used to record and automate sequences of actions, then task automation is achieved, but the scripts become too complex to be understood or manipulated by individuals unfamiliar with computer programming
Solution Approach 1:
The patent creates simplified copies or representations of complex task sequences in the form of action embeddings. Instead of requiring users to work with complex scripting languages, the system captures the essential semantic meaning of tasks as embeddings that can be stored, retrieved, and executed without requiring programming knowledge, while still achieving full task automation.
3Adaptability or versatility
If users repeatedly perform semantically-similar tasks across different contexts manually, then task flexibility is maintained, but the process becomes cumbersome, prone to error, and consumes unnecessary computing resources and user attention
Solution Approach 1:
The system records a task once in a domain-agnostic action embedding space, creating a universal representation that can be executed across multiple contexts and applications. This eliminates the need for users to manually repeat semantically-similar tasks while maintaining flexibility through the ability to adapt the same task to different domains using domain models.
Data Source
AI summary
Disclosed implementations relate to automating semantically-similar computing tasks across multiple contexts. In various implementations, an initial natural language input and a first plurality of actions performed using a first computer application may be used to generate a first task embedding and a first action embedding in action embedding space. An association between the first task embedding and first action embedding may be stored. Later, subsequent natural language input may be used to generate a second task embedding that is then matched to the first task embedding. Based on the stored association, the first action embedding may be identified and processed using a selected domain model to select actions to be performed using a second computer application. The selected domain model may be trained to translate between an action space of the second computer application and the action embedding space.