UI Element Detection Using Multi-Threshold Bounding Boxes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing techniques for detecting elements in digital user interfaces, such as computer vision models, underperform, and integrating step-by-step guidance into software applications is challenging, especially when third parties develop the guidance due to limited control over software changes.
Innovation Solution
A method involving image processing to detect potential elements of interest in a digital user interface using grayscale conversion, thresholding, dilation, and bounding boxes, followed by digital guidance creation and display.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional computer vision models are used to detect UI elements, then element detection is attempted, but detection accuracy and reliability are insufficient
Solution Approach 1:
The patent segments the UI element detection process into multiple distinct stages: grayscale conversion, thresholding to create binary images, dilation to expand element regions, and bounding box generation. Each stage processes the image data independently and contributes to the final detection result, improving overall accuracy and reliability through systematic breakdown of the complex detection task.
Solution Approach 2:
The patent creates binary image copies from the grayscale image at different threshold levels. These binary copies represent different interpretations of the same UI elements, and by processing multiple copies through dilation and bounding box extraction, the system cross-validates detection results across different threshold conditions, thereby improving reliability.
2Ease of operation
If step-by-step guidance is integrated into software applications, then user adoption is improved, but implementation complexity increases especially for third parties
Solution Approach 1:
The patent enables UI elements to self-identify through automated image processing. The system automatically converts UI screenshots to grayscale, applies thresholding, performs dilation, and generates bounding boxes without requiring manual annotation or complex configuration. This self-service approach simplifies guidance implementation for third parties while maintaining ease of operation for users.
Solution Approach 2:
The patent performs preliminary processing of UI elements by pre-generating binary images and dilation results that define element boundaries. These pre-computed bounding boxes are stored and reused for guidance creation, eliminating the need for complex real-time analysis during guidance implementation and reducing overall system complexity.
3Adaptability or versatility
If HTML attributes are used to identify UI elements, then element identification is attempted, but reliability decreases when attributes change
Solution Approach 1:
The patent replaces the mechanical attribute-matching approach with an image processing-based identification system. Instead of relying on HTML attributes that may change, the system uses grayscale conversion, thresholding, and dilation to visually identify and bound UI elements. This substitution makes identification reliable regardless of attribute changes while maintaining adaptability to different UI designs.
Data Source
AI summary
A method for detecting potential elements of interest present in digital user interfaces comprises receiving a grayscale image of a user interface comprising multiple elements. The grayscale image may be formed of pixels. Multiple threshold values corresponding to the pixels may be identified and selected by processing the grayscale image. Multiple binary images may be thereafter generated. Each of the binary images may correspond to a respective one of the threshold values. Thereafter, multiple sets of one or more bounding boxes may be obtained. Each of the sets may correspond to a respective one of the binary images. Each of the bounding boxes may encapsulate a respective one or more of the elements. Subsequently, one or more bounding boxes may be obtained, each one encapsulating a respective one of the potential elements of interest among the elements present in the user interface, by processing the multiple sets of bounding boxes.