Multi-Camera Scene Encoding With Priority-Based CPU-GPU Allocation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Mixed-reality telepresence systems face high computational demands due to multiple cameras capturing 3D models, unpredictable user pose changes, and GPU contention, making existing cloud and edge solutions inefficient for large-scale MR applications.
Innovation Solution
Habbiton, an edge-compute framework, selectively activates a subset of cameras based on user pose and FoV, employs an AIMD resource scaling policy, and uses context-aware GPU sharing to manage compute demands and reduce latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multiple cameras are used to capture 3D models with sufficient fields-of-view, then the immersive quality and 3D degrees of freedom are improved, but the computational complexity and latency synchronization requirements increase significantly
Solution Approach 1:
The patent segments the multi-camera system into independent encoding subsystems, each handling a subset of cameras. The scene manager divides camera streams into multiple encoded streams, allowing parallel processing and reducing the computational burden on any single encoder. This segmentation enables the system to manage multiple cameras while maintaining synchronized latency through independent processing pipelines.
Solution Approach 2:
The patent introduces a temporal dimension to the encoding process by implementing frame-rate conversion and time-synchronization mechanisms. The system converts between different frame rates and interpolates frames to achieve synchronized output timing. This temporal dimension allows the system to handle cameras with different capture rates while maintaining coherent 3D model construction with consistent latency across all camera streams.
2Adaptability or versatility
If cloud scaling approaches such as Function-as-a-Service are used to handle compute demands, then resource flexibility is improved, but the response time to user pose changes increases due to operating on the order of hundreds of milliseconds to seconds
Solution Approach 1:
The patent implements preliminary action by pre-configuring encoding parameters, buffer sizes, and processing pipelines before user interactions occur. The scene manager pre-establishes encoding subsystems with allocated resources, so when user pose changes occur, the system can immediately activate pre-prepared encoding configurations rather than dynamically allocating resources. This reduces the response time from hundreds of milliseconds to much faster intervals while maintaining resource flexibility through pre-configured scalability.
3Measurement precision
If GPU resources are allocated to process ML tasks for all cameras, then the scene understanding quality is improved, but GPU contention increases with multiple cameras generating 3D frames at 30 FPS or higher
Solution Approach 1:
The patent applies local quality by allocating GPU resources selectively to specific cameras or camera subsets based on their priority and the current scene context. The scene manager identifies which cameras provide the most valuable scene understanding information and directs GPU ML processing to those specific streams. This allows the system to maintain high scene understanding quality for critical cameras while reducing overall GPU contention by not processing all cameras at maximum quality levels simultaneously.
4Reliability
If a single 3D model is constructed using data from multiple cameras, then the completeness of the virtual world is improved, but the computational latency between cameras must be kept similar which increases synchronization complexity
Solution Approach 1:
The patent introduces an intermediary scene manager that acts as a mediator between multiple camera encoding subsystems. The scene manager receives encoded streams from various cameras, performs frame-rate conversion, temporal interpolation, and time-stamping to synchronize the streams. This intermediary layer handles the synchronization complexity centrally, allowing individual camera encoders to operate independently while the scene manager ensures all streams are temporally aligned before constructing the unified 3D model, thus maintaining virtual world completeness without requiring tight synchronization at the camera level.
Data Source
AI summary
A scene encoding subsystem for multi-user mixed-reality telepresence has at least one encoder that processes signals from a plurality of cameras. A scene manager determines relative priorities of the cameras and allocates CPU resources and GPU resources in the at least one encoder based on the relative priorities of the cameras, where (i) greater CPU resources are allocated to higher priority cameras than to lower priority cameras and (ii) greater GPU resources are allocated to lower priority cameras than to higher priority cameras. The scene manager instructs a CPU to process a skipped frame using a bounding-box expansion algorithm based on object motion determined using GPU object detection and segmentation processing of previous non-skipped frames. The scene manager multiplicatively decreases and additively increases CPU and GPU resources for a set of multiple cameras based on performance of any one camera in the set.