Sub-Image Streaming via DPU-GPU Pipeline for Low-Latency Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image processing systems face bottlenecks due to the need for central processing unit (CPU) intervention, leading to latency and inefficiencies in handling high-speed, high-quality media streams, particularly in applications like virtual reality and telemedicine, where high bandwidth and low latency are crucial.
Innovation Solution
A system that separates physical layer processing (PLP) from content layer processing (CLP) by delegating payload to a graphics processing unit (GPU) directly from a field programmable gate array (FPGA) via a data processing unit (DPU), bypassing CPU intervention, and using sequence numbers and direct packet placement to ensure efficient handling and reliability of image sections.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If CPU intervention is used for image processing, then system reliability is improved, but processing latency increases and productivity decreases
Solution Approach 1:
A data processing unit (DPU) is introduced as an intermediary component between the FPGA and GPU, handling packet processing, sequence number management, and data routing. This intermediary architecture allows the CPU to maintain system control and reliability while the DPU handles time-critical data processing tasks, eliminating CPU intervention bottlenecks and reducing processing latency in the imaging pipeline.
Solution Approach 2:
The system divides image processing into distinct functional segments: physical layer processing (FPGA) handles signal acquisition and initial processing, the DPU handles data packet management and sequencing, and the GPU handles content layer processing. This segmentation allows each component to specialize in specific tasks, improving overall processing speed while maintaining system reliability through distributed functionality.
2Reliability
If CPU intervention is used for image processing, then data integrity is improved, but system overhead increases and productivity decreases
Solution Approach 1:
The DPU serves as an intermediary that manages data packet flow, sequence numbering, and routing between the FPGA and GPU. This intermediary handles data integrity tasks that would otherwise require CPU intervention, reducing system overhead while maintaining data integrity through dedicated hardware-based packet processing and sequencing mechanisms.
Solution Approach 2:
The system implements self-service mechanisms where the DPU autonomously manages data packet processing, sequence number assignment, and routing decisions without requiring CPU intervention. The GPU also performs self-service by directly processing image sections from the DPU, reducing overall system overhead and complexity while maintaining data integrity through hardware-based protocols.
3Productivity
If all images are processed, then completeness of processing is improved, but processing time increases and productivity decreases
Solution Approach 1:
The system extracts and processes only the necessary image sections (payloads) from the media stream using the DPU, which identifies and separates relevant data from complete images. This extraction approach allows the GPU to process only the essential image sections rather than entire images, improving processing throughput while reducing the time required for complete image processing through selective data extraction.
Solution Approach 2:
The system implements partial processing by focusing computational resources on processing image sections that are most critical for the application. The DPU enables selective extraction and routing of specific image sections to the GPU, allowing partial processing of image data that achieves sufficient productivity for the application while avoiding the time cost of processing complete images.
Data Source
AI summary
Systems and methods herein are for distributed image processing by at least a data processing unit (DPU) and using at least a graphics processing unit (GPU) in possible association with a field programmable gate array (FPGA). For example, the FPGA may be used to perform physical layer processing for images captured by the image sensor or from a simulation and can provide a media stream for the DPU and the DPU can provide payload of only image sections from the images in a media stream for the GPU to perform content layer processing for only the image sections of the images.