UI Automation Engine Detecting Invisible Input Fields via DOM Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional clipboard technology fails to detect and fill input fields that are not visible on the screen, limiting its ability to automate the completion of long forms.
Innovation Solution
An interface engine that uses computer vision and document object model analysis to detect and automatically fill user interface elements, including those not visible on the screen, by extracting relevant DOM text and determining the types of UI elements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional clipboard technology uses computer vision based detection for destination analysis, then visible input fields can be detected, but input fields not visible on the screen cannot be detected
Solution Approach 1:
The patent transitions from two-dimensional screen-based computer vision detection to three-dimensional spatial reasoning by analyzing the Document Object Model (DOM) structure. This allows the system to detect input fields beyond the visible screen area by understanding the hierarchical and spatial relationships between DOM elements, effectively adding a dimensional layer to the detection capability.
Solution Approach 2:
The patent introduces the Document Object Model (DOM) as an intermediary between the visible screen content and the complete form structure. The DOM serves as a mediator that bridges the gap between what is visible on the screen and what exists in the complete form, enabling detection of invisible input fields through DOM analysis rather than direct visual detection.
2Device complexity
If conventional clipboard technology focuses on visible input fields, then processing is simplified, but mapping source field data to invisible input fields fails
Solution Approach 1:
The patent performs preliminary analysis of the DOM structure to identify and map input fields before the actual data entry process begins. By pre-processing the DOM to understand the complete form structure and relationships between fields, the system prepares mapping relationships in advance, enabling efficient data transfer to both visible and invisible input fields without increasing operational complexity.
Solution Approach 2:
The patent creates a virtual copy or representation of the complete form structure through DOM analysis, even though only portions are visible on the screen. This virtual copy allows the system to work with the complete form structure and map data to all input fields, including invisible ones, without requiring the user to scroll or display the entire form.
3Ease of operation
If a user repeatedly scrolls partially to access invisible input fields, then all input fields can be accessed, but time consumption increases
Solution Approach 1:
The patent replaces the mechanical scrolling action with a computational DOM analysis approach. Instead of requiring physical scrolling through the form, the system uses computer-based DOM parsing and spatial reasoning to directly identify and access invisible input fields, eliminating the need for repeated scrolling operations and significantly reducing time consumption.
Data Source
AI summary
A method is provided. The method is for detecting and automatically filling user interface (UI) elements of a page that are not visible on a screen. The method is executed by an interface engine implemented as a computer program within a computing environment. The method includes analyzing document object model (DOM) of the page to extract DOM text of relevant UI elements of the UI elements. The method includes performing computer vision (CV) analysis to determine types the relevant UI elements and extrapolating target UI elements of an invisible region of the page from the relevant UI elements.