Spatial-Element Frame Streaming for VR Latency Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for distributing high-resolution immersive video, such as VR360, face challenges in reducing motion-to-high-res latency due to inefficiencies in HTTP Adaptive Streaming, leading to increased latency and potential motion sickness in VR headsets.
Innovation Solution
The method involves a distribution node that processes requests for high-resolution spatial-element frames by storing and transmitting temporal segments of spatial-element frames in close proximity within a Content Delivery Network chunk size, allowing for efficient caching and reduced latency by enabling byte-range requests, which can include both inter-coded and intra-coded frames, thereby optimizing data retrieval and playback.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If HTTP Adaptive Streaming is used to distribute high-resolution immersive video, then bandwidth efficiency is improved, but motion-to-high-res latency increases
Solution Approach 1:
The video stream is divided into temporal segments that are further segmented into spatial elements (tiles). Each spatial element can be independently requested and decoded. This segmentation allows the system to serve only the portions of the video that are currently in the user's field of view, reducing both bandwidth requirements and latency by avoiding the need to download entire temporal segments before accessing specific spatial regions.
Solution Approach 2:
The system pre-encodes multiple temporal segments with different resolutions and qualities, and organizes them in a way that allows random access. By preparing these segments in advance and structuring them for efficient retrieval, the system can quickly deliver high-resolution spatial elements when needed without waiting for sequential download, thereby reducing motion-to-high-res latency while maintaining bandwidth efficiency.
2Ease of manufacture
If full spherical video is delivered to the client as traditional video, then existing technology is re-used, but bandwidth consumption increases significantly
Solution Approach 1:
The full spherical video is segmented into multiple spatial elements (tiles) that can be independently requested. Instead of delivering the complete spherical video at high resolution, the system only transmits the specific spatial elements that are currently visible in the user's field of view. This dramatically reduces bandwidth consumption while maintaining the ability to display high-resolution video when needed.
Solution Approach 2:
The system implements viewport-adaptive streaming, where only the portion of the video corresponding to the user's current field of view is transmitted and decoded. This partial action approach ensures that bandwidth is not wasted on regions that are not visible, while still providing high-resolution experience for the active viewport. The system can request additional spatial elements when the user moves their head to expand the visible area.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method of transmitting a request for video data comprises a client device transmitting a request for high-resolution spatial-element frames of a spatial element of a video to a distribution node for each spatial element of the video in a user's field of view for which a client device does not possess a current high-resolution spatial-element frame. The video comprises a plurality of spatial elements and the plurality of spatial-element frames comprises both non-inter-coded spatial-element frames (152) and inter-coded spatial-element frames (151). The request identifies the spatial element and specifies a starting point (179) corresponding substantially to a current time. The request is for data comprising a temporal segment of high-resolution spatial-element frames starting substantially at the starting point (179) and of which the first high-resolution spatial-element frame (173) is not inter coded. The method further comprises the client device receiving the requested data.