Web Automation Vectorization for Structural Adaptability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing web task automation methods are inflexible and require extensive user interaction, as they often rely on rigid models that cannot adapt to changes in web page structures and programming.
Innovation Solution
The proposed solution involves a system with a modelling component, a recorder component, and a playback component, which operates based on recorded demonstrations of web tasks. This system allows for the automation of web tasks by adapting to similar tasks and using a single command, such as a text or voice snippet, to execute tasks without the need for mouse or keyboard interactions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional web browser automation methods are used, then tasks can be executed on web pages, but the system requires extensive user interaction and is inflexible to web page structure changes
Solution Approach 1:
The system creates a vector representation (copy) of the web page layout that captures the spatial and structural relationships between elements. This vector model serves as an adaptable representation that can be compared against the actual web page to identify corresponding elements even when the web page structure changes, reducing the need for rigid model matching and extensive user interaction.
Solution Approach 2:
The patent replaces traditional mechanical interaction methods (mouse clicks, keyboard inputs) with an automated system that uses vector-based element identification and programmatic control. The headless browser automation engine substitutes manual mechanical operations with automated commands driven by vector representation matching, significantly reducing user interaction requirements.
2Extent of automation
If rigid model-based automation is used, then automation can be achieved, but the system cannot adapt to changes in web page structures and programming
Solution Approach 1:
The system employs dynamic vector representations of web page elements that can adapt to structural changes. Instead of using static rigid models, the vector-based representation captures the essential spatial and relational properties of web elements, allowing the automation system to dynamically identify corresponding elements even when the web page structure undergoes changes, thereby maintaining high automation capability while improving adaptability.
Solution Approach 2:
The patent transforms the web page representation from traditional DOM-based parameters to vector-based parameters that capture spatial coordinates, element sizes, and positional relationships. This parameter transformation enables the automation system to identify web elements through their vector characteristics rather than rigid structural parameters, allowing the system to adapt to web page changes while maintaining extensive automation capability.
3Productivity
If traditional automation tools requiring mouse and keyboard interactions are used, then tasks can be performed, but extensive user interaction is required
Solution Approach 1:
The system implements self-service automation where the headless browser automation engine independently identifies web elements using vector representations and executes tasks without requiring user guidance. The automated system serves itself by programmatically navigating web pages, identifying elements through vector matching, and performing actions based on the vector model, eliminating the need for extensive user interaction and significantly improving task execution efficiency.
Solution Approach 2:
The vector representation serves as an intermediary between the automation engine and the web page elements. Instead of requiring direct user interaction with the web page, the system uses the vector model as a mediator to translate automation commands into appropriate web element interactions. This intermediary layer enables automated task execution while reducing user interaction requirements.
Data Source
AI summary
A system and method uses a vectorization model to determine similar elements within a web page to the known web page. The vectorization model takes a known web element and generates a first set of vectors, representative of the various properties of the web element. A vectorization model generates a second set of vectors for each element in a new web page. The first set of vectors is compared to each second set of vectors for each element in the new web page to select the most similar web element.