Streaming Input Event Recovery for Packet-Loss Interaction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Synchronization of user inputs with rendered content in a streaming application environment over packet-switched networks is challenging due to packet loss, delay, and out-of-order delivery, leading to a poor user experience, especially in mobile and vehicle applications.
Innovation Solution
Implement modules within remote devices and streaming application environments to synchronize input events with frame rendering, using control and frame refresh handlers, jitter buffers, and event resending mechanisms to ensure timely and accurate injection of user inputs into virtual streaming applications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If packet retransmission protocols (TCP) are used to ensure reliable delivery of input events and rendered content, then packet loss is reduced, but transmission delays increase significantly
Solution Approach 1:
The system segments data transmission into two distinct channels: a reliable channel for critical input events using TCP, and an unreliable but fast channel for rendered content frames using UDP. This segmentation allows each type of data to use the most appropriate protocol, avoiding the delay penalty of TCP retransmissions for content frames while ensuring input events are delivered reliably.
Solution Approach 2:
The system sends input events multiple times (excessive action) with different sequence numbers to increase the probability of at least one copy arriving successfully. Rather than waiting for confirmation of successful delivery, the system proactively transmits redundant copies, accepting some bandwidth overhead to minimize delay and ensure reliability.
2Reliability
If all packets are waited for before processing to ensure complete data reception, then data integrity is improved, but processing delays increase
Solution Approach 1:
The system performs preliminary actions by pre-sending input events with sequence numbers before the corresponding rendered content frames arrive. The remote device prepares to process input events as soon as they are received, without waiting for confirmation that all related content packets have been delivered, enabling parallel processing and reduced latency.
Solution Approach 2:
The system implements feedback mechanisms where the remote device sends acknowledgments for received input events and the host device uses this feedback to determine when to send subsequent content frames. This feedback loop allows the system to proceed with processing based on actual reception status rather than waiting for predetermined packet sequences.
3Manufacturing precision
If input events are sent frequently to match frame refresh rates (60 Hz or more), then synchronization with rendered content is improved, but network bandwidth consumption increases
Solution Approach 1:
The system uses periodic action by sending input events at regular intervals synchronized to the frame refresh rate (e.g., 60 Hz). This periodic transmission ensures consistent synchronization between input events and rendered frames while maintaining predictable bandwidth consumption patterns that can be efficiently managed by the network stack.
Solution Approach 2:
The system discards duplicate input events that arrive with the same sequence number and recovers from potential losses by using sequence number tracking. The remote device maintains a buffer of received events and can request retransmission or use alternative copies, allowing the system to tolerate some redundant transmissions while maintaining synchronization precision.
Data Source
Figure 1
Figure 2~4
Figure 5~6
AI summary
In a streaming application environment coupled to a remote device over a packet-switching network, correction events may be synthesized from input events received from the remote device and injected into a virtual streaming application to account for lost or delayed input events. In addition, trailing events may be reissued by a remote device in frames during which no new input events are generated.