Video Streaming Latency Reduction via Viewport Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current technologies face challenges in providing low latency image rendering for augmented reality (AR), virtual reality (VR), and mixed reality (MR) applications due to constraints in wireless data transmission and tracking delays, leading to mismatches between the viewer's actual position and the rendered images, resulting in reduced immersive quality.
Innovation Solution
The approach involves partitioning video images into target view portions with high spatiotemporal resolution and non-target view portions with lower resolution, allowing the client device to render images with zero latency by interpolating the target view portion to match the viewer's new position and direction, even before receiving updated viewpoint data from the server.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If high speed connections are provisioned to transmit large amount of image data within frame interval, then low latency image rendering is achieved, but client device becomes bulky with too much external wiring
Solution Approach 1:
The patent segments the video image into multiple frames and further divides each frame into multiple tiles. By processing and transmitting smaller tile portions instead of complete frames, the system reduces the amount of data that needs to be transmitted over high-speed connections, thereby reducing wiring requirements while maintaining low latency rendering.
Solution Approach 2:
The system performs preliminary rendering of video images before they are needed for display. Frames are pre-rendered and stored in a buffer, allowing the client device to retrieve and display them without requiring high-speed real-time transmission, thus reducing the need for extensive external wiring while maintaining low latency.
2Device complexity
If wireless networks are used to stream image data, then device complexity is reduced, but transmission latency increases significantly
Solution Approach 1:
The server pre-renders video frames and stores them in a buffer before they are requested by the client device. This preliminary action allows wireless transmission to occur without significant latency, as the data is already prepared and waiting in the buffer, eliminating the need for real-time rendering during transmission.
Solution Approach 2:
By segmenting frames into smaller tiles, the system reduces the total data size that needs to be transmitted wirelessly. This segmentation allows faster wireless transmission of divided data portions, reducing overall transmission latency while maintaining reduced device complexity.
3Manufacturing precision
If complete frames are transmitted with high resolution, then image quality is maintained, but data transmission time exceeds frame interval
Solution Approach 1:
The patent divides complete frames into multiple smaller tiles and transmits only the necessary tile portions to the client device. This segmentation reduces the total data volume that needs to be transmitted while maintaining high resolution quality for the displayed portions, allowing transmission to complete within the frame interval.
Solution Approach 2:
Instead of transmitting complete high-resolution frames, the system transmits only the partial tile portions that are currently needed for display. This partial action approach reduces transmission time to fit within frame intervals while maintaining sufficient image quality for the visible portions.
4Measurement precision
If tracking and informing server of viewer's field and view position is performed frequently, then positional accuracy is improved, but round trip time delay increases
Solution Approach 1:
The client device continuously tracks the viewer's field and view position locally and maintains this information in a buffer. This preliminary tracking action allows the device to use the most recent positional data immediately without waiting for server confirmation, maintaining high positional accuracy while minimizing round trip time delays.
Data Source
AI summary
Based on viewing tracking data, a viewer's view direction to a three-dimensional (3D) scene depicted by a first video image is determined. The first video image has been streamed in a video stream to the streaming client device before the first time point and rendered with the streaming client device to the viewer at the first time point. Based on the viewer's view direction, a target view portion is identified in a second video image to be streamed in the video stream to the streaming client device to be rendered at a second time point subsequent to the first time point. The target view portion is encoded into the video stream with a higher target spatiotemporal resolution than that used to encode remaining non-target view portions in the second video image.