Website Input Detection Using ML-Based Visual Field Recognition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing identity management systems face inefficiencies and unreliability in detecting input fields on websites, particularly when metadata formats are unknown or inconsistent, leading to time-consuming and insecure information input processes.
Innovation Solution
Utilizing a machine learning model for image-based input detection, where an image capturing system obtains website images, generates location predictions for inputs, and automatically inputs content into identified fields based on these predictions, enhancing accuracy and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If metadata searching is used to locate input fields, then the system can identify input locations, but the process becomes inefficient and unreliable when metadata format is unknown
Solution Approach 1:
The patent replaces the mechanical metadata parsing approach with a vision-based system using machine learning models. Instead of searching through website metadata structure, the system captures screenshots and uses image recognition to locate input fields, buttons, and other interactive elements. This substitution resolves the contradiction by making the system both reliable (works regardless of metadata format) and efficient (direct visual identification without exhaustive searching).
Solution Approach 2:
The system changes the detection parameter from metadata text analysis to visual image recognition. By transforming the input from structured data search to image processing, the system achieves consistent performance across different website formats and languages, improving reliability while maintaining speed through optimized image analysis pipelines.
2Reliability
If manual selection of interactive elements is required, then users can accurately input information, but the process becomes time-consuming
Solution Approach 1:
The system implements self-service by automatically detecting and selecting interactive elements without user intervention. The machine learning model identifies input fields, populates them with appropriate data, and automatically clicks submit buttons, enabling the system to complete the entire form-filling process autonomously and rapidly.
Solution Approach 2:
The system performs preliminary actions by pre-training the machine learning model on extensive website data before deployment. This pre-training enables the model to immediately recognize and interact with various input field types and button patterns across different websites, eliminating the need for users to manually learn or select elements during actual use.
3Productivity
If automated input is implemented without image-based detection, then the system can fill forms, but it becomes insecure and unreliable across different languages and formats
Solution Approach 1:
The vision-based system achieves universality by processing website interfaces as images rather than structured data. This approach allows the same machine learning model to accurately identify input fields, buttons, and form elements across different languages, cultural formats, and website designs, making the automated input system both fast and reliably applicable globally.
Data Source
AI summary
An identity management system may be associated with a software plug-in for input detection of a website. In some examples, the plug-in may obtain, via an image capturing system, an image of the website that includes a set of inputs, where the set of inputs includes an interactive interface element. Using the obtained image, a set of location predictions for the set of inputs of the website may be generated via a machine learning (ML) model. Further, the plug-in may obtain a set of locations of the set of inputs based on generating the set of location predictions. Thus, the plug-in may automatically, and in response to obtaining the set of locations of the set of inputs of the website, input content into the set of inputs of the website, select an interactive interface element on the website, or both, on the behalf of the user.