Virtual Skeleton Video Compression Bandwidth Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Digital media content transmission over communications networks is limited by latency and bandwidth constraints, particularly in real-time streaming applications, where existing compression algorithms do not effectively manage data transmission efficiently.
Innovation Solution
The method involves modeling a human subject with a virtual skeleton using optical sensor information, transmitting the virtual skeleton at a higher frame rate than surface information, and using additional virtual skeleton frames to estimate and render surface information for frames not transmitted, thereby conserving bandwidth and reducing latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If compression algorithms are applied to reduce data transmission, then bandwidth usage is reduced, but transmission latency increases
Solution Approach 1:
The patent segments the human subject representation into two distinct components: virtual skeleton data and surface information. The virtual skeleton is transmitted at a higher frame rate (e.g., 60 fps) while surface information is transmitted at a lower frame rate (e.g., 30 fps). This segmentation allows the system to reduce overall bandwidth usage while maintaining the temporal fidelity needed for real-time interaction through the high-frame-rate skeleton data.
Solution Approach 2:
The patent implements dynamic frame rate adjustment for different data types. Instead of using a uniform frame rate for all video data, the system dynamically transmits virtual skeleton at a higher frame rate than surface information. The receiver dynamically estimates surface information for frames where it was not transmitted, adapting the reconstruction process based on the availability of skeleton data to maintain real-time performance.
2Reliability
If full surface information is transmitted at high frame rate, then video quality is maintained, but bandwidth consumption increases
Solution Approach 1:
The patent extracts and transmits only the essential structural information (virtual skeleton) at high frame rate, separating it from the full surface information. The surface information is transmitted at a reduced frame rate. The receiver extracts and uses the transmitted skeleton data to estimate and reconstruct the missing surface information, thereby maintaining video quality while significantly reducing bandwidth consumption compared to transmitting complete high-frame-rate surface data.
Solution Approach 2:
The virtual skeleton acts as an intermediary between the transmitted low-frame-rate surface information and the final high-quality video output. The skeleton data serves as a temporal bridge, allowing the receiver to estimate surface information for frames that were not directly transmitted, thereby reconstructing high-quality video at the playback frame rate without requiring proportional bandwidth.
3Loss of energy
If frame rate is reduced to conserve bandwidth, then bandwidth usage decreases, but motion estimation accuracy deteriorates
Solution Approach 1:
The patent applies partial action by transmitting skeleton data at a higher frame rate than the surface information. This excessive transmission of structural motion data (skeleton at 60 fps vs. surface at 30 fps) provides sufficient motion information for accurate estimation, while avoiding the need to transmit excessive surface data. The partial transmission strategy focuses resources on the most critical motion-capturing component.
Solution Approach 2:
The patent changes the frame rate parameter differently for different data types. Instead of uniformly reducing the frame rate for all video data, the system applies different frame rate parameters to virtual skeleton (higher) and surface information (lower). This parameter differentiation allows the receiver to use the high-frame-rate skeleton data to accurately estimate motion for the lower-frame-rate surface information, maintaining motion estimation accuracy while reducing overall bandwidth usage.
Data Source
AI summary
Optical sensor information captured via one or more optical sensors imaging a scene that includes a human subject is received by a computing device. The optical sensor information is processed by the computing device to model the human subject with a virtual skeleton, and to obtain surface information representing the human subject. The virtual skeleton is transmitted by the computing device to a remote computing device at a higher frame rate than the surface information. Virtual skeleton frames are used by the remote computing device to estimate surface information for frames that have not been transmitted by the computing device.