Headless Browser Automation for Dynamic Web Portals
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing web portal automation methods face challenges with dynamic content, such as Captcha and two-factor authentication, which require human intervention and are not adaptable to changing data sets or service interfaces, limiting their ability to automate interactions seamlessly.
Innovation Solution
A system and method for controlling a headless browser that allows pausing and resuming automation processes, converting dynamic content into a presentation-agnostic form, and engaging human users through a man-machine interface for response collection, using REST APIs and dynamic web controls to handle various data formats and interfaces.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If forced human interaction features (Captcha, two-factor authentication) are implemented to ensure data privacy and prevent bot access, then website security and data privacy are improved, but automation capability and ease of operation deteriorate
Solution Approach 1:
The patent introduces an intermediary system that sits between the headless browser and the web portal, translating automation actions into human-like interactions. This mediator captures the headless browser's navigation actions, translates them into human user actions, and submits them through the portal's human interaction requirements, thereby maintaining both security and automation capability
Solution Approach 2:
The system segments the automation process into distinct phases: headless browser navigation phase, human interaction phase, and result collection phase. This segmentation allows different techniques to be applied at different stages, with the headless browser handling data collection and the human user handling authentication challenges
2Adaptability or versatility
If web portal interfaces and data sets are frequently updated to improve service quality, then service relevance and adaptability are improved, but automation stability and reliability deteriorate
Solution Approach 1:
The patent implements dynamic adaptation mechanisms where the automation script automatically adjusts to interface changes. The system monitors portal structure changes, updates its navigation logic accordingly, and maintains stable automation despite frequent interface updates. This dynamic approach allows the automation to adapt to new data sets and service interfaces while maintaining reliability
Solution Approach 2:
The system incorporates feedback loops that monitor the success of automation actions and automatically adjust behavior based on portal responses. When interface changes are detected, the system learns from the interactions and modifies its approach, ensuring continued stability and reliability across updates
3Quantity of substance
If comprehensive data collection is performed to improve service quality, then data completeness and service capability are improved, but processing time and complexity increase
Solution Approach 1:
The patent implements preliminary actions by pre-configuring the headless browser with navigation scripts and data collection parameters before actual automation begins. This preparation phase allows the system to efficiently collect comprehensive data without excessive processing time during execution, as the collection strategy is already optimized in advance
Data Source
AI summary
A system and method for controlling a headless browser automation (an automation that does not render web pages or involve direct user input), where the web portals accessed by the browser automation employ a rule system for generating web content and the rules can result in the dynamic presentation of new response elements that could not be a priori determined from existing content already encountered, or the history of interaction with the web portal. The control system has the capability to pause the headless browser automation in place, collect the new information being requested by the web portal, present the requested new information to a remote user, capture the information from the remote user, feed the information collected to the portal automation, and restart the automation from the point that it left off.