Crossmodal UI Automation Without API Integration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional automation solutions rely on documented APIs, SDKs, and macros, which are fragile under UI changes and impractical for legacy applications, and vision-assisted RPA systems fail to align audio and visual contexts, limiting their applicability.
Innovation Solution
A system that captures crossmodal information (visual, input, audio, and text) to form tokens, maps them to UI-level actions using a Large Action Model (LAM), and orchestrates processes with a Large Orchestration Model (LOM) to execute OS-native inputs, avoiding published APIs, thus being robust to UI drift.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional automation solutions use documented APIs, SDKs, and macros, then automation can be implemented with structured interfaces, but the solutions become fragile under UI changes and impractical for legacy applications
Solution Approach 1:
The patent replaces the mechanical/API-based automation approach with a vision-based system that uses computer vision to detect and interact with UI elements. Instead of relying on documented APIs that break when UI changes, the system captures images of the UI, detects elements visually, and sends simulated mouse/keyboard events to interact with them. This substitution enables robust automation that adapts to UI changes without requiring API documentation or updates.
Solution Approach 2:
The patent introduces an intermediary layer between the automation system and the target application UI. This intermediary consists of the vision system that captures UI images, detects elements, and translates high-level commands into simulated user input events. This intermediary layer decouples the automation logic from the specific UI implementation, allowing the system to work with any UI regardless of its underlying technology or documentation status.
2Reliability
If vision-assisted RPA systems detect onscreen elements, then robustness under UI changes is improved, but audio instructions and textual procedures cannot be aligned with visual context
Solution Approach 1:
The patent merges multiple information modalities (visual, audio, and textual) into a unified processing framework. The system captures UI images, transcribes audio instructions, and processes textual procedures simultaneously, then aligns them through temporal synchronization and contextual analysis. This merging enables the system to leverage complementary strengths of each modality: visual detection for robustness, audio for natural interaction, and text for structured procedures.
Solution Approach 2:
The patent creates a universal processing framework that handles multiple types of input information (visual elements, audio instructions, textual procedures) through a common architecture. The same system that detects UI elements visually also processes audio commands and textual workflows, aligning them in a unified task execution framework. This multi-functionality eliminates the need for separate processing pipelines for different input modalities.
3Ease of manufacture
If process discovery tools record screens and infer workflows, then workflow automation can be discovered, but discovery is separated from execution and connectors or recorded scripts are still required
Solution Approach 1:
The patent enables the system to perform both workflow discovery and execution using the same vision-based architecture. During discovery, the system records UI images and user actions, infers workflows, and stores them as executable task definitions. During execution, the same vision system captures current UI state, detects elements, and executes the inferred workflows by sending simulated input events. This self-service capability eliminates the need for separate discovery and execution systems, as well as complex connectors or recorded scripts.
Data Source
AI summary
A system automates tasks in software applications without invoking a published API of the target application at runtime. Synchronized visual display frames, input events, audio of a task request, and text from digital procedures are captured and aligned to form crossmodal tokens. A Large Action Model maps tokens to user-interface actions, and a Large Orchestration Model composes and orders the actions to satisfy a goal and policy constraints. An executor issues operating-system native input signals to the target application and verifies outcomes from subsequent display frames using optical character recognition and layout cues. A feedback loop updates the models. The approach provides no-integration automation across legacy and modern applications with semantic reanchoring for UI changes and privacy protections.