Input Latency Mitigation via Partial Display Mirroring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems for using direct input devices with spatial input software face compatibility issues and suffer from latency and network-related problems, such as variable network latency, jitter, and packet loss, which degrade the user experience, especially in tasks like drawing.
Innovation Solution
A system that allows a direct input device to interact with spatial input software on a secondary device over a network by mirroring a selected portion of the secondary device's display on the input device, using interim and unacknowledged input rendering to mitigate latency and network issues, and allowing users to select the display portion for mirroring, enabling compatibility with incompatible software.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If input data is transmitted over a network to a secondary device for processing, then compatibility with spatial input software is improved, but network latency and jitter degrade the user experience
Solution Approach 1:
The system performs preliminary actions by sending input data to the secondary device before receiving confirmation of processing completion. The client device continues to render based on the sent input data without waiting for acknowledgments, effectively performing the rendering action in advance while the network transmission and server processing occur in the background.
Solution Approach 2:
The client device creates a local copy of the input data for rendering purposes. Instead of waiting for the server to process and return the input data, the client makes a copy of the sent input data and uses it immediately for rendering, thereby eliminating the perceived latency while the actual server processing occurs asynchronously.
2Loss of information
If the entire server display is mirrored to the client device, then complete visual feedback is provided, but network bandwidth consumption increases
Solution Approach 1:
The system extracts only the necessary portion of the server display that corresponds to the active input area and mirrors only that specific region to the client device. Instead of transmitting the entire display, the system takes out and transmits only the relevant visual feedback portion, reducing network bandwidth consumption while maintaining complete visual feedback for the user's current task.
3Measurement precision
If input rendering waits for server acknowledgment, then input accuracy is improved, but responsiveness and user experience deteriorate
Solution Approach 1:
The client device performs self-service by rendering input data locally based on its own sent data without requiring server confirmation. The system serves itself by using the already-transmitted input data for immediate rendering, eliminating the need to wait for server acknowledgments while maintaining input accuracy through the use of the originally sent coordinate data.
Data Source
AI summary
In an embodiment, the system includes a direct input device (the “client”) executing a client application (“app”), a secondary device (the “server”) executing a server app, and a network link between the server and the client. The server app transmits to the client app, via the network link, a portion of the content displayed on the server's computer screen. The client app may display this portion. Whenever the user interacts with the client device, the client app transmits those interactions to the server app over the network link. The server app sends an acknowledgement to the client app and provides the received interactions to the spatial input software executing on the server, and the spatial input software consumes the interactions as if the user performed the interactions on the server device itself.