Edge Video Tensor Batching for Multi-GPU Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video data processing systems in edge computing require significant computing and storage resources due to the need for separate service processing modules and full-scale deep learning models for each video frame, making it difficult to deploy such systems at the edge.
Innovation Solution
A method involving obtaining first tensor data from video frame data within a preset time window, splicing sub-tensor data to form second tensor data, and allocating these to graphics processing units based on deployed deep learning models for efficient data inference.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If separate service processing modules and full-scale deep learning models are used for each video frame, then processing accuracy is maintained, but computing resource usage increases significantly
Solution Approach 1:
The patent segments video data into fixed-size batches and divides deep learning models into sub-tensor data units. This segmentation allows the graphics processor to handle multiple video frames simultaneously in batch processing mode, reducing the per-frame computing overhead while maintaining processing accuracy through consistent batch inference.
Solution Approach 2:
The patent merges multiple video frames into a single batch for processing. By combining multiple first tensor data into second tensor data that represents a batch of video frames, the system leverages the graphics processor's parallel processing capabilities to reduce overall computing resource consumption compared to processing each frame separately.
2Productivity
If separate service processing modules are deployed for each video frame, then processing capability is ensured, but device complexity increases
Solution Approach 1:
The patent implements a universal batch processing framework that handles multiple video frames and multiple deep learning models through a single integrated system. The graphics processor executes multiple deep learning models on batched video data, eliminating the need for separate dedicated processing modules for each frame and reducing system complexity.
3Measurement precision
If full-scale deep learning models are used for each video frame, then inference accuracy is maintained, but graphics processor utilization is low
Solution Approach 1:
The patent performs preliminary batching of video frames into fixed-size batches before inference. By pre-organizing video data into batches and preparing the corresponding tensor data structures in advance, the system maximizes graphics processor utilization during the actual inference phase, ensuring that the processor operates at full capacity while maintaining inference accuracy.
Solution Approach 2:
The patent implements continuous batch processing where the graphics processor continuously receives and processes batches of video frames without idle periods. This continuous operation ensures high graphics processor utilization while maintaining inference accuracy through consistent batch processing of video data through the deployed deep learning models.
Data Source
AI summary
A video data processing method, a device, a storage medium and an edge device are provided, which relate to artificial intelligence, in particular to computer vision and edge computing. The method includes: obtaining a plurality of first tensor data based on a plurality of video frame data, where the first tensor data includes a plurality of sub-tensor data each corresponding to a respective one of a plurality of deep learning models; splicing the plurality of sub-tensor data in the first tensor data to obtain a plurality of second tensor data each corresponding to a respective one of the plurality of deep learning models; and allocating the plurality of second tensor data to a plurality of graphics processing units in a graphics processor, based on the deep learning models respectively deployed by the graphics processing units, such that the graphics processor performs data inference on the plurality of second tensor data.