Volumetric Video Streaming via Predicted Viewpoint Cell Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Smartphone devices from 2018 lack the computation power to decompress volumetric videos encoded in point cloud representation at the required frame rate of 30 frames per second, leading to unacceptably long stalls during streaming, even when multiple CPU cores are utilized.
Innovation Solution
A system that predicts a viewer's viewpoint within a volumetric video, segments the point cloud into cells, generates cell occupancy bitmaps, and transmits only visible cells for playback, reducing computational complexity and bandwidth requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If volumetric videos are streamed in uncompressed point cloud representation, then video quality is improved, but computational load and bandwidth requirements increase significantly
Solution Approach 1:
The point cloud is divided into multiple cells organized in a three-dimensional grid structure. Each cell contains a subset of points, allowing the system to process and transmit only relevant portions rather than the entire point cloud, thereby reducing computational complexity while maintaining video quality.
Solution Approach 2:
The system extracts and transmits only the visible cells that contain points visible from the predicted viewpoint, removing invisible cells from the data stream. This extraction process reduces bandwidth requirements and computational load on the client device while preserving the quality of visible content.
2Loss of information
If all cells in the point cloud are transmitted, then complete video content is provided, but bandwidth consumption and transmission time increase
Solution Approach 1:
The system performs preliminary prediction of the viewer's future viewpoint based on current motion trends before transmission. This allows the server to pre-identify and prepare only the visible cells for upcoming frames, reducing bandwidth consumption by excluding invisible cells while ensuring complete video content is available when needed.
3Manufacturing precision
If the entire point cloud is decompressed on the client device, then full video resolution is achieved, but playback stalls occur due to insufficient processing power
Solution Approach 1:
By segmenting the point cloud into cells and transmitting only visible cells, the client device receives a reduced subset of data that requires less computational resources to decompress. This enables the device to maintain the required 30 frames per second frame rate while still achieving full video resolution for the visible portion.
4Productivity
If visible cells are predicted and transmitted in advance, then playback smoothness is improved, but prediction accuracy requirements increase
Solution Approach 1:
The system transmits visible cells for future frames based on predicted viewpoints, performing the action partially in advance. This preliminary transmission of visible cells improves playback smoothness by preparing data before it is needed, while the prediction accuracy requirement is managed through incremental updates as the viewer's actual viewpoint becomes known.
Data Source
AI summary
Aspects of the subject disclosure may include, for example, a device having a processing system including a processor; and a memory that stores executable instructions that, when executed by the processing system, facilitate performance of operations, the operations comprising predicting a viewpoint within a volumetric video based on a movement input provided by a viewer, resulting in a predicted viewpoint at a future time, retrieving a cell occupancy bitmap for a point cloud of the volumetric video for the future time, determining visible cells based on the cell occupancy bitmap and the predicted viewpoint, where the visible cells are not obscured by points in other cells, and retrieving points of the point cloud that are within the visible cells prior to the future time.