Multipoint Screen Sharing Server Frame Forwarding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In multipoint content sharing systems, the large data size of IDR frames worsens video experience due to data bursts, especially when new receivers join, and controlling IDR frames or data size compromises user experience or video quality.
Innovation Solution
A server in the multipoint communication system stores key reference frames and determines network conditions and decoding capacity to classify receivers, forwarding all key reference frames to new receivers and selectively forwarding key and non-key reference frames to existing receivers based on their capacity and conditions, reducing the need for IDR frames and optimizing bandwidth usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If an IDR frame is sent to all receivers when a new receiver joins, then the new receiver can quickly decode and display the content, but the data burst deteriorates the uplink channel and worsens video experience for all receivers
Solution Approach 1:
The server pre-stores multiple key reference frames (including IDR frames and intra-refresh frames) in its buffer before they are needed. When a new receiver joins, the server can immediately retrieve and forward these pre-stored frames without requiring the sender to generate new IDR frames, thus avoiding uplink channel deterioration while ensuring fast decoding for new receivers
Solution Approach 2:
The server acts as an intermediary between the sender and receivers. Instead of having the sender directly send IDR frames to all receivers (which causes uplink channel deterioration), the server receives frames from the sender, stores key reference frames, and then forwards appropriate frames to receivers. This intermediary role decouples the IDR frame generation from the distribution process, eliminating the harmful data burst effect on the uplink channel
2Object-affected harmful factors
If the data size of each IDR frame is reduced to alleviate uplink channel deterioration, then the channel condition improves, but the video quality at all receivers decreases
Solution Approach 1:
The server pre-stores multiple key reference frames with full quality (including high-quality IDR frames and intra-refresh frames) in its buffer. When forwarding frames to receivers, the server can select from these pre-stored high-quality frames without needing to reduce their size, thus maintaining video quality while avoiding the need to send new large-size IDR frames that would deteriorate the uplink channel
Solution Approach 2:
The server recycles and reuses previously received key reference frames (IDR frames and intra-refresh frames) by storing them in its buffer. Instead of discarding these frames after initial use, the server recovers them for future use when new receivers join or when existing receivers need frame synchronization, thereby avoiding the need to generate new large-size IDR frames that would burden the uplink channel
3Object-affected harmful factors
If the number of IDR frames is controlled to reduce uplink channel burden, then the channel condition improves, but some receivers have to wait longer
Solution Approach 1:
The server pre-stores multiple key reference frames (IDR frames and intra-refresh frames) in its buffer before they are needed. This preliminary storage ensures that when a new receiver joins, the server can immediately retrieve and forward appropriate frames without waiting for the sender to generate new IDR frames, thus reducing receiver waiting time while controlling the number of IDR frames sent over the uplink channel
Solution Approach 2:
The server dynamically selects which frames to forward to which receivers based on real-time conditions. The server can adjust its frame forwarding strategy dynamically - selecting from pre-stored key reference frames based on receiver capabilities, network conditions, and session state - thereby optimizing the balance between uplink channel burden and receiver waiting time without fixed constraints
Data Source
AI summary
Techniques are provided to improve user experience at endpoint devices that are participating in a multipoint screen content sharing session. A server in a multipoint communication system receives video frames from a sending endpoint device in the communication session that involves one or more other endpoint devices that are to receive video frames from the sending endpoint device. The video frames include key reference frames that use either intra-frame prediction or inter-frame prediction based only on other key reference frames. The server stores the key reference frames. When a new receiving endpoint device joins the communication session, the server sends all the stored key reference frames to the new receiving endpoint device. All video frames received from the sending endpoint device are forwarded to existing ordinary receiving endpoint devices. Key reference frames and some, but not all, non-key reference video frames are forwarded to existing low-capacity receiving endpoint devices.