Touch Screen Intent Prediction via Neural Network Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Mobile devices with limited touch screen sizes and dense graphical user interfaces often lead to unintentional selection of incorrect icons, resulting in unnecessary network traffic and resource wastage due to the inability to accurately predict user intentions without application-specific metadata or APIs.
Innovation Solution
The use of machine learning models, specifically convolutional neural networks and recurrent neural networks, to predict the intended user selection by analyzing screenshots and user input coordinates, identifying selectable and non-selectable areas within the GUI, and determining the nearest intended pixel based on Euclidean distance calculations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the number of activation icons per square inch is increased to provide more functions, then the functionality of the device is improved, but the accuracy of user selection deteriorates
Solution Approach 1:
The system performs preliminary analysis of the touch input by examining the touch duration, pressure, and movement patterns before finalizing the selection. This preliminary action allows the system to predict whether the user intends to select a specific icon or perform a different gesture, thereby maintaining high functionality while improving selection accuracy through predictive modeling.
2Length of moving object
If the touch screen size is reduced to make the device more portable, then the portability is improved, but the ease of accurate selection deteriorates
Solution Approach 1:
The system introduces an intermediary processing layer that analyzes touch gestures and predicts user intent before executing the selection. This intermediary layer acts as a mediator between the user's physical touch and the final selection outcome, allowing small touchscreen devices to maintain accurate selection by interpreting gesture patterns rather than relying solely on precise finger placement.
3Device complexity
If conventional touch selection methods are used without prediction, then the system complexity is reduced, but the loss of time due to erroneous selections increases
Solution Approach 1:
The system implements feedback mechanisms where the predicted selection is presented to the user for confirmation before execution. This feedback loop allows the system to use minimal computational resources while still providing correction opportunities, reducing time loss from erroneous selections without requiring complex full-scale prediction models.
Data Source
AI summary
The subject matter of this specification describes a method to identify a point intended to be selected by a user on a touch screen of a user device. The method includes receiving user input including location coordinates for the user input, generating a screenshot image pertaining to a graphical user interface (GUI), determining intended location coordinates for a point intended to be selected in the GUI based on a detected layout of a GUI area corresponding to the user input, and providing, to the application, the determined intended location coordinates for the point intended to be selected in the GUI.