Virtual Frame Synchronization for Remote Streaming Input
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Synchronization of user input events with rendered content in a streaming application environment over a packet-switched network is challenging due to packet delays, losses, and jitter, leading to poor user experience, especially in mobile and vehicle applications where network connectivity is compromised.
Innovation Solution
A method that involves generating virtual frames corresponding to frame refresh events from a remote device, controlling the injection of input events into a virtual streaming application to synchronize them with the correct frames, and using techniques like scheduling and queuing to manage delays and losses, ensuring that input events are injected within the correct virtual frames, even in the presence of network jitter and packet loss.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If input events are transmitted over a packet-switched network to enable remote device interaction with virtual streaming applications, then user interaction capability is improved, but synchronization accuracy between input events and rendered content deteriorates due to packet delays, losses, and jitter
Solution Approach 1:
The system generates virtual frames in advance corresponding to expected frame refresh events, and prepares to inject input events into the appropriate virtual frames before actual rendering occurs. This preliminary preparation allows the system to accommodate network delays and jitter by having the correct frame structure ready when input events arrive, maintaining synchronization accuracy while enabling remote interaction.
Solution Approach 2:
The patent introduces virtual frames as an intermediary layer between the remote device input events and the actual rendered content. These virtual frames act as a buffer and synchronization mechanism, allowing input events to be injected at the correct logical point in time even when network transmission causes delays or reordering. The virtual frames mediate between the asynchronous network input and the synchronous rendering requirements.
2Reliability
If packet retransmission protocols are used to ensure reliable delivery of input events and rendered content, then data reliability is improved, but interaction responsiveness deteriorates due to unacceptable delays
Solution Approach 1:
The system sends input events and frame data without waiting for confirmation of successful receipt or acknowledgment, performing the transmission action partially (without full retransmission protocol handshaking). This approach sacrifices some reliability guarantees but maintains responsiveness by avoiding the time-consuming retransmission delays, which is acceptable for interactive applications where timeliness is more critical than absolute data reliability.
3Measurement precision
If multiple touch events are generated during a single frame interval to track finger movement, then gesture tracking precision is improved, but frame synchronization difficulty increases leading to jerky movements
Solution Approach 1:
The system segments the injection of multiple input events into different virtual frames based on their temporal association with frame refresh events. Instead of attempting to inject all events into a single frame or handling them as a bulk operation, the patent divides the events and distributes them to appropriate virtual frames, simplifying the synchronization logic while maintaining gesture tracking precision for multi-event gestures like finger swipes.
Data Source
AI summary
In a streaming application environment, input generated in a remote device may be synchronized with rendered content generated by a virtual streaming application in the streaming application environment in part by using frame refresh events passed between the remote device and the streaming application environment to enable the streaming application environment to effectively track a frame refresh rate of the remote device such that input events received from the remote device may be injected into the virtual streaming application at appropriate frame intervals.