Cross-Platform Workflow Automation With Hybrid UI Element Location
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional robotic process automation (RPA) systems are brittle and require high maintenance due to reliance on brittle selectors, scripts, or coordinate replay, breaking down with minor UI changes, and existing AI applications are fragmented and do not provide a unified system that learns, executes, and adapts in a closed loop.
Innovation Solution
An AI-driven RPA platform that records user interactions, learns generalized workflows with sequence models, executes across heterogeneous UIs using hybrid element location, and adapts via a continuous learning loop, integrating computer vision, machine learning, and context-aware task segmentation for resilient automation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If conventional RPA uses brittle selectors and coordinate replay, then automation can be implemented with simple tools, but the system becomes highly maintenance-intensive and breaks with minor UI changes
Solution Approach 1:
The patent replaces conventional mechanical RPA approaches (coordinate replay, brittle selectors) with AI/ML-based computer vision and sequence modeling. The system uses visual recognition and learned workflows instead of rigid coordinate-based automation, enabling robust operation across UI changes while maintaining implementation simplicity through high-level workflow definition.
2Extent of automation
If AI is applied narrowly for element detection or process discovery, then specific tasks can be automated, but the system lacks unified learning and continuous adaptation capabilities
Solution Approach 1:
The patent merges multiple AI functionalities (element detection, process discovery, sequence modeling, continuous learning) into a unified RPA platform. The system integrates these components through a cohesive architecture that enables end-to-end automation with continuous adaptation, avoiding the fragmentation of separate point-solution approaches.
Solution Approach 2:
The patent creates a universal RPA system that performs multiple functions: recording user interactions, learning workflows, segmenting tasks, executing across heterogeneous applications, and continuous adaptation. This multi-functional platform replaces multiple separate tools and processes with a single integrated system.
3Adaptability or versatility
If RPA executes across heterogeneous UIs, then versatility is improved, but element location becomes more difficult and less reliable
Solution Approach 1:
The patent introduces a hybrid element location system that acts as an intermediary between the automation engine and heterogeneous UIs. This system fuses multiple location strategies (accessibility APIs, DOM metadata, computer vision) to reliably identify elements across different platforms, abstracting away the complexity of each UI system.
Solution Approach 2:
The patent employs a composite approach to element location by combining multiple methods (accessibility trees, DOM structure, visual recognition) into a unified location system. This composite strategy leverages the strengths of each method while compensating for their individual weaknesses across different UI frameworks.
Data Source
AI summary
An AI-driven robotic process automation system learns and replicates user workflows across web, desktop, and legacy applications using computer vision and machine learning. During training, the system observes user actions with visual and structural UI context, segments the sequence into reusable tasks, and synthesizes a generalized workflow model. At runtime, a hybrid locator fusing vision with DOM/accessibility metadata binds abstract actions to live controls, while a self-healing subsystem detects anomalies and applies recovery actions. A continuous learning loop updates models and task definitions from execution telemetry so automations remain effective as interfaces evolve. The result is resilient “learn-once, run-anywhere” automation that reduces brittle scripting and maintenance overhead.