Neural Network UI Element Hierarchy for Automation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies are unable to consistently and accurately determine and invoke the various elements and functionality of image-based application/website interfaces using automated scripts, due to the subjective nature of human pattern recognition and the sensitivity of conventional approaches to pixelation and compression algorithms.
Innovation Solution
A method that determines logical relationships between textual and non-textual elements of images of a user interface, builds a hierarchy of these elements to form a data structure representing the interface's functionality, and outputs this data structure to memory, enabling automated interaction with the interface.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If conventional template matching approaches are used to identify interface elements, then the process can be automated to some extent, but the approach becomes sensitive to pixelation and compression algorithms, leading to inaccurate element identification
Solution Approach 1:
The patent replaces conventional template matching (mechanical pattern recognition) with neural network-based deep learning. The neural network learns semantic features and logical relationships between interface elements, making the system invariant to pixelation and compression artifacts while maintaining high automation capability.
Solution Approach 2:
The patent transforms the approach from pixel-level template matching to semantic-level understanding by changing the parameter space. Instead of comparing raw pixel values, the system uses neural networks to extract and compare semantic features, logical relationships, and hierarchical structures of interface elements.
2Reliability
If image-based interfaces are used to reduce network bandwidth and improve security, then host system functionality is protected, but automated scripts cannot consistently determine and invoke interface elements
Solution Approach 1:
The patent introduces neural networks as an intermediary layer between the image-based interface and automated scripts. The neural network translates visual interface elements into structured semantic representations that automated scripts can process, enabling automation without compromising the security and bandwidth benefits of image-based interfaces.
Solution Approach 2:
The patent replaces traditional automated recognition methods with neural network-based semantic understanding. This substitution enables automated scripts to accurately identify and interact with interface elements while maintaining the protective image-based interface architecture.
3Ease of operation
If human pattern recognition methods are used to interpret interface images, then subjective criteria can be applied, but the process cannot be reduced to program instructions for automation
Solution Approach 1:
The patent replaces human subjective pattern recognition with neural network-based semantic understanding. The neural network learns to interpret interface elements by analyzing logical relationships and hierarchical structures, producing results that are both accurate (like human interpretation) and programmable (unlike human cognition).
Solution Approach 2:
The patent transforms subjective human pattern recognition into objective computational processes by changing the parameters from pixel values to semantic features. The neural network extracts meaningful attributes such as element relationships, hierarchy, and function, making the process both interpretable and automatable.
4Productivity
If I/O events are tracked by position and timing to determine user interactions, then interaction data can be captured, but slight changes in pixelation prevent proper invocation of interface functions
Solution Approach 1:
The patent replaces position-and-timing-based interaction tracking with neural network-based semantic recognition. Instead of relying on precise pixel coordinates that are sensitive to compression, the system identifies interface elements through their semantic characteristics and logical relationships, making interaction mapping robust to pixelation variations.
Data Source
AI summary
The presently disclosed inventive concepts are directed to systems, computer program products, and methods for intelligent screen automation. According to one embodiment, a method includes: determining one or more logical relationships between textual elements and non-textual elements of one or more images of a user interface; building a hierarchy comprising some or all of the non-textual elements and some or all of the textual elements in order to form a data structure representing functionality of the user interface; and outputting the data structure to a memory.