Video Tile AI Hardware Pipeline for Real-Time Playback
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video playback systems on smartphones face delays and inefficiencies due to software-scheduled tile processing, which fail to meet hard real-time requirements and are prone to software lag, impacting smooth video playback in applications like video conferences and phone calls.
Innovation Solution
A hardware-based pipeline mechanism using synchronization signals to synchronize video decoding, AI model execution, and display operations, with a pipeline manager controlling these components directly within the hardware portion, reducing reliance on software stacks and enabling efficient, real-time processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If software-scheduled tile processing is used, then development flexibility is maintained, but processing delay and software lag increase
Solution Approach 1:
The video frame is divided into multiple tiles that can be processed independently through the hardware pipeline, allowing parallel decoding and AI processing while maintaining software control over the overall flow. This segmentation enables real-time processing without sacrificing software flexibility.
Solution Approach 2:
A hardware pipeline manager acts as an intermediary between the software and hardware components, coordinating the decoding and AI processing operations. This mediator ensures timely data transfer and synchronization without requiring direct software control of hardware operations, reducing processing delay while maintaining software flexibility.
2Productivity
If hardware-based pipeline with synchronization signals is implemented, then processing speed and real-time performance improve, but hardware complexity increases
Solution Approach 1:
Synchronization signals are used to provide feedback between different hardware components (decoder and AI accelerator), ensuring proper timing and data availability. This feedback mechanism enables high-speed processing while keeping the hardware design manageable through clear timing protocols.
Solution Approach 2:
The hardware pipeline manager performs multiple functions including coordinating data transfer, managing synchronization signals, and orchestrating both decoding and AI processing operations. This multi-functionality reduces the need for separate dedicated hardware components, thereby limiting the increase in hardware complexity despite the added pipeline functionality.
3Loss of time
If software stacks are reduced for real-time processing, then processing latency decreases, but system reliability and security may be compromised
Solution Approach 1:
Time-critical pipeline coordination functions are extracted from the software stack and implemented as dedicated hardware circuitry. This extraction eliminates software processing delays while maintaining security through hardware-level isolation and dedicated data paths that reduce exposure to software-based vulnerabilities.
Data Source
AI summary
Aspects of the present disclosure provide a video playback system. For example, the video playback system can include a video controller configured to receive one or more video frames and partition each of the video frames into a plurality of video tiles, a video decoder configured to decode one of the video tiles partitioned from each of the video frames, an artificial intelligence (AI) accelerator configured to execute an executable AI model on the decoded video tile, a display configured to display the processed video tile, and a pipeline manager installed in the video controller. The pipeline manager can be configured to control the video decoder to decode the video tile, control the AI accelerator to execute the executable AI model on the decoded video tile, and control the display to display the processed video tile.