Active Element Detection via Screenshot OCR and Caret Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current detection technologies for computing systems fail to accurately determine the active element on the screen during user interactions, such as typed or pasted text, caret tracking, and active element detection, which is crucial for robotic process automation and process discovery.
Innovation Solution
A computer program that generates a keyboard character queue and performs OCR on screenshot frames to identify and match characters or sequences, determining the active element by comparing recognized characters to the queue and tracking caret positions through pixel changes and binary matrices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If key hooking is used to detect key presses, then key press detection is achieved, but the active element on the screen cannot be determined
Solution Approach 1:
The patent introduces screenshot comparison as an intermediary mechanism between key press detection and active element identification. By capturing the screen state before and after key presses and comparing the differences, the system can identify which element received the input without requiring direct interaction with the input device, thus resolving the information loss problem
Solution Approach 2:
The patent replaces direct mechanical key tracking with a visual comparison system. Instead of relying solely on keyboard hooks to determine input location, the system uses image processing to compare screenshots and infer the active element based on visual changes on the screen
2Measurement precision
If OCR is performed on the entire screen, then all text can be recognized, but processing time increases significantly
Solution Approach 1:
The patent segments the screen into multiple regions and only applies OCR to regions that have changed between screenshots. This selective approach maintains text recognition accuracy for relevant areas while dramatically reducing overall processing time by skipping unchanged regions
Solution Approach 2:
The patent applies different processing quality to different regions of the screen. High-quality OCR is applied only to regions with changes (where text input is likely occurring), while unchanged regions are processed minimally or not at all, optimizing the balance between accuracy and speed
3Measurement precision
If multiple screenshot frames are compared to detect changes, then text detection accuracy improves, but system complexity increases
Solution Approach 1:
The patent performs preliminary screenshot capture and storage before text input occurs. By having the baseline screenshot ready in advance, the system can quickly compare it with subsequent frames to detect changes, reducing the complexity of real-time processing while maintaining high detection accuracy
Data Source
AI summary
Detection of typed and/or pasted text, caret tracking, and active element detection for a computing system are disclosed. The location on the screen associated with a computing system where the user has been typing or pasting text, potentially including hot keys or other keys that do not cause visible characters to appear, can be identified and the physical position on the screen where typing or pasting occurred can be provided based on the current resolution of where one or more characters appeared, where the cursor was blinking, or both. This can be done by identifying locations on the screen where changes occurred and performing text recognition and/or caret detection on these locations. The physical position of the typing or pasting activity allows determination of an active or focused element in an application displayed on the screen.