Screen Sharing Pointer Tracking via Visual Pattern Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In web collaboration and communication sessions, delays in communication connections can cause the shared screen to scroll, resulting in the pointing position no longer referencing the intended item, leading to confusion during discussions.
Innovation Solution
A method where the coordinates of the intended item on the shared screen are received, along with a screenshot of the surrounding area, which is then matched with the current screen content to maintain the pointing position's accuracy even after scrolling, using a control unit that processes this data to ensure the pointer remains fixed on the intended item.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If screen sharing is implemented in real-time communication sessions, then participants can view relevant content displayed at the shared screen, but delays in communication connections cause the shared screen to scroll, resulting in the pointing position no longer referencing the intended item
Solution Approach 1:
The system captures a screenshot of the intended item and its surrounding area at the moment the user points at it, before any scrolling occurs. This preliminary capture of the visual state allows the system to later match this screenshot against updated screen content to track the item's position even when the screen scrolls during communication delays
Solution Approach 2:
The system creates a copy of the visual content (screenshot) at the pointing moment and uses this copy as a reference to search through subsequent screen updates. By matching the captured screenshot against new screen content, the system can locate the intended item regardless of scrolling, maintaining pointing position accuracy without requiring real-time synchronization
2Productivity
If the shared screen is updated in real-time during communication, then the latest content is displayed to all participants, but scrolling occurs during updates, causing coordinate-based pointing to reference incorrect positions
Solution Approach 1:
Instead of relying on coordinate systems that become invalid after scrolling, the system copies the visual appearance of the intended item (via screenshot) and matches this visual copy against updated screen content. This allows rapid screen updates to proceed while the pointing system independently tracks the intended item through visual pattern matching rather than unstable coordinates
Solution Approach 2:
The system performs a preliminary screenshot capture of the item of interest before screen updates occur. This preliminary visual record serves as a template that can be matched against subsequent screen states, allowing the system to maintain accurate pointing references even as the screen dynamically updates and scrolls during high-productivity communication sessions
Data Source
AI summary
A computer-implemented method of sharing a screen amongst at least two users participating in a real-time communication and collaboration session via a communication network can be configured so that a screen displaying a first content can be shared by a user of a first client so as to be displayed at at least a second client used by a second user. Embodiments of the method can include receiving, at a control unit for controlling the communication and collaboration session, coordinates specifying the position of an intended item of the first content of the shared screen and receiving, at the control unit, a screenshot of the intended item and a predetermined surrounding area as a sample picture or as sample pixels.