Mixed Reality Coding with HEVC Overlay Auxiliary Pictures
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current video coding methods lack flexibility in displaying mixed reality content, as video content and rendered content are composited and encoded together at the front end, limiting post-encoding modifications and user interaction.
Innovation Solution
The implementation of HEVC overlay auxiliary pictures allows for encoding mixed reality content using separate layers for real-world content, rendered content, map data, and alpha data, enabling interleaving and decoding these components separately for enhanced flexibility in client-side display decisions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If video content and rendered content are composited and encoded together at the front end, then encoding simplicity is maintained, but flexibility in displaying mixed reality content is lost
Solution Approach 1:
The patent segments the mixed reality content into separate layers: base layer (real-world video content) and overlay auxiliary pictures (rendered content, map data, alpha data). Each layer is encoded independently, allowing the client to selectively decode and display specific layers based on interaction needs, thus achieving both encoding simplicity and display flexibility.
2Productivity
If video content and rendered content are composited and encoded together, then processing steps are reduced, but user interactivity is limited
Solution Approach 1:
By segmenting content into base layer and overlay auxiliary pictures with independent encoding, the system maintains processing efficiency through standardized encoding pipelines while enabling user interactivity at the client side through selective layer decoding and compositing control.
Solution Approach 2:
The patent enables dynamic control of overlay auxiliary pictures at the client side, allowing users to interactively select which rendered content layers to display, hide, or composite with the base video content, thereby achieving ease of operation and user interactivity without compromising encoding productivity.
3Loss of energy
If separate layers are used for encoding mixed reality content, then compression efficiency is improved, but encoding complexity increases
Solution Approach 1:
The patent segments mixed reality content into base layer and overlay auxiliary pictures that can be encoded separately, improving compression efficiency by allowing independent optimization for each layer type while managing encoding complexity through standardized layer structures and interoperable encoding interfaces.
Data Source
AI summary
A system includes a camera to capture real world content and a semiconductor package apparatus. The semiconductor package apparatus includes a substrate and logic. The logic includes a graphics pipeline to generate rendered content, a base layer encoder to encode real world content into a base layer and a first layer encoder to encode rendered content into a first non-base layer, a multiplexer to interleave the base layer with the first non-base layer to obtain a single output signal having mixed reality content, and a transmitter to transmit the single output signal. The system further includes a second layer encoder to encode map data into a second non-base layer. The multiplexer to interleave the second non-base layer with the first non-base layer and the base layer. The first and second layer encoders encode the rendered content and the map data into overlay auxiliary pictures.