Multi-Decoder Interface for Synchronized Immersive Media Rendering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current application programming interfaces (APIs) for video decoding, such as OpenMAX IL and Android's MediaCodec, lack the capability to discover aggregate processing capabilities of multiple video decoders and provide fine-grained control over scheduling, leading to inefficiencies in handling immersive media that requires simultaneous decoding of multiple video streams.
Innovation Solution
Abstract extensions to the decoder APIs are introduced to support immersive media, allowing for the discovery of instantaneous capabilities of multiple video decoders and enabling synchronized rendering of decoded data from multiple decoder instances, which can be mapped to existing APIs like OpenMAX IL and MediaCodec.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple video decoder instances are used for immersive media, then decoding capability and processing throughput are improved, but API capability to discover aggregate processing capabilities and control scheduling is insufficient
Solution Approach 1:
The patent segments the decoder functionality into multiple independent decoder instances that can be individually controlled and scheduled. Each decoder instance can process different video streams simultaneously, enabling parallel decoding while maintaining independent control over each instance's operation, resource allocation, and synchronization parameters.
Solution Approach 2:
The patent creates a universal decoder interface that can handle multiple video streams across different formats and codecs through a single unified API framework. This multi-functional interface provides aggregate capability discovery and centralized scheduling control that works across diverse decoder instances, eliminating the need for separate API implementations for each decoder.
2Reliability
If synchronized rendering of multiple video streams is required, then immersive media quality is improved, but control over decoder scheduling and resource allocation becomes more complex
Solution Approach 1:
The patent introduces a mediator component that sits between the multiple decoder instances and the rendering system. This mediator handles synchronization coordination, timestamp management, and resource allocation centrally, allowing decoders to operate independently while ensuring synchronized output. The mediator abstracts the complexity of coordination from individual decoder control.
Solution Approach 2:
The patent implements feedback mechanisms where decoder instances report their processing status, buffer states, and timing information to a central controller. The controller uses this feedback to dynamically adjust scheduling decisions, resource allocation, and synchronization parameters in real-time, ensuring reliable synchronized rendering while adapting to changing system conditions.
Data Source
AI summary
An example device for decoding media data includes a memory configured to store the media data and one or more processors implemented in circuitry and communicatively coupled to the memory. The one or more processors are configured to determine whether two or more decoder instances are intended to be synchronized. The one or more processors are configured to, based on the two or more decoder instances being intended to be synchronized, control the two or more decoder instances so as to enable rendering of decoded data from each of the two or more decoder instances at a same presentation time.