Web Task Automation via Object Model Skeletons
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing web task automation technologies struggle to autonomously perform complex tasks on web pages using natural language instructions, as they lack the ability to interpret and execute sequences of actions accurately across different web elements.
Innovation Solution
The method involves generating an instruction performance skeleton by comparing natural language instructions to an object model of a web page, and then using a playback engine to execute these instructions autonomously on the web page, allowing for the performance of tasks that mimic human interactions with the web interface.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If conventional web browser automation tools are used, then basic web tasks can be performed, but complex tasks requiring accurate interpretation of natural language instructions and adaptation to different web elements cannot be executed autonomously
Solution Approach 1:
The patent introduces an object model as an intermediary layer between the natural language instruction and the web page elements. The object model translates human-readable instructions into structured actions that can be executed on specific web elements, enabling autonomous execution while maintaining adaptability across different web interfaces
Solution Approach 2:
The patent segments the web page into discrete objects with defined properties and actions. By breaking down the web page structure into manageable objects that can be individually addressed and manipulated, the system can accurately interpret natural language instructions and execute complex tasks across different web elements
2Productivity
If manual interaction methods are used, then tasks can be performed on web pages, but the process is time-consuming and lacks efficiency
Solution Approach 1:
The system enables self-service automation where the playback engine automatically executes tasks based on recorded instructions without requiring manual intervention for each action. The object model allows the system to autonomously navigate and interact with web elements, significantly reducing execution time while maintaining accuracy
Solution Approach 2:
The patent employs preliminary action by recording and storing the sequence of actions required to perform a task in an object model format. This pre-prepared instruction set can be replayed automatically, eliminating the need for repeated manual execution and dramatically improving productivity
3Measurement precision
If simple automation scripts are used, then execution speed is fast, but accuracy in interpreting natural language instructions and adapting to web page changes is poor
Solution Approach 1:
The object model serves as a sophisticated intermediary that bridges natural language instructions and web page elements. It provides a structured framework for interpreting instructions with high precision while managing the complexity of automation through standardized object definitions and relationships
Solution Approach 2:
The system utilizes parameter changes in the object model to adapt to web page modifications. By monitoring and responding to changes in object properties and structure, the system maintains high interpretation accuracy even when web pages are updated, without requiring complete system redesign
Data Source
AI summary
A method of generating an instruction performance skeleton employs an instruction unit configured to receive a natural language instruction. From the natural language instruction, a sequence of clauses may be extracted. The instruction unit then determines a target website or websites on which to perform the task. The object models of the target website are generated. A comparison of the sequence of actions to the object model and its labelling hierarchical class structure is performed. Based on this comparison, an instruction performance skeleton is generated. In future, on the basis of a further natural language instruction that is similar to the previous natural language instruction, the instruction performance skeleton may be modified to generate a playback performance skeleton to arrange performance of a task.