Active Element Detection via Screenshot Comparison and OCR
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current computing systems lack an effective method to determine the active element on the screen during user interactions, such as typing or pasting text, which hinders accurate monitoring and automation of user interactions.
Innovation Solution
A feedback loop process that compares current and previous screenshots to identify visual changes, performs optical character recognition (OCR) on the changed areas, and matches the results with a keyboard queue to determine the active element, using a combination of image processing and keyboard hooking techniques.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If key hooking is used to determine key presses, then key input detection is achieved, but the active element on the screen cannot be identified
Solution Approach 1:
The patent introduces screenshot comparison as an intermediary mechanism between key hooking and active element identification. By capturing screen images at different times and comparing them, the system can determine which element was active when the key was pressed, thus bridging the gap between key detection and element identification without requiring direct element tracking.
Solution Approach 2:
The system uses feedback from screenshot comparisons to identify active elements. By continuously capturing and comparing screenshots, the system receives feedback about visual changes on the screen, which indicates which element was active during the key press event, enabling accurate active element detection.
2Measurement precision
If screenshot comparison is performed to identify visual changes, then active element detection is improved, but processing time and computational resources increase
Solution Approach 1:
The patent segments the screenshot comparison process by focusing only on specific regions or elements that changed between frames, rather than analyzing the entire screen. This segmentation approach maintains detection accuracy while reducing the computational burden and processing time required for screenshot comparison.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Detection of typed text and active element detection for a computing system are disclosed. The location on the screen associated with a computing system where the user has been typing text, potentially including hot keys or other keys that do not cause visible characters to appear, can be identified and the physical position on the screen where typing occurred can be provided based on the current resolution of where one or more characters appeared. This can be done by identifying locations on the screen where changes occurred and performing text recognition on these locations. The physical position of the typing activity allows determination of an active or focused element in an application displayed on the screen.