Dynamic CPU-Accelerator Task Allocation for AI Preprocessing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing AI training frameworks are inflexible in terms of computation resource utilization during data preprocessing, leading to suboptimal CPU and accelerating device resource utilization, and resource contention between training and data pipelines.
Innovation Solution
A method and system for intelligent heterogeneous computation that dynamically generates resource computation job lists based on monitoring data, such as occupancy rates and utilization factors, to distribute tasks between CPUs and accelerating devices like GPUs, NPUs, and TPUs, allowing for parallel processing and adaptive resource allocation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data preprocessing tasks are assigned to accelerating devices, then processing speed is improved, but resource contention between training and data pipelines worsens
Solution Approach 1:
The system dynamically determines whether to use CPU or accelerating devices for data preprocessing tasks based on real-time resource availability and task characteristics. This dynamic allocation allows the system to adapt to changing conditions, improving preprocessing speed when accelerators are available while avoiding resource contention when they are not, thus resolving the contradiction between speed improvement and resource allocation flexibility.
2Productivity
If heterogeneous computing is used in AI training pipelines, then computational efficiency is improved, but system complexity worsens
Solution Approach 1:
The system employs self-service mechanisms where the data preprocessing pipeline automatically monitors accelerator utilization and independently makes decisions about task allocation. The system self-regulates by adjusting the degree of parallelism and selecting appropriate computation resources without requiring complex external management, thus improving AI training efficiency while keeping system complexity manageable.
Solution Approach 2:
The system implements feedback mechanisms that continuously monitor accelerator utilization metrics and use this information to dynamically adjust task allocation decisions. This feedback loop allows the system to optimize computational efficiency by directing preprocessing tasks to accelerators when they are underutilized, while automatically falling back to CPU when accelerators are heavily loaded, thereby resolving the contradiction between productivity improvement and device complexity.
3Adaptability or versatility
If CPU operations are used for data preprocessing, then resource contention is reduced, but processing speed worsens
Solution Approach 1:
The system changes the operational parameters of data preprocessing by dynamically switching between CPU-only mode and accelerator-utilizing mode based on real-time conditions. When accelerators are available, the system changes parameters to utilize them for faster processing; when they are not, it reverts to CPU operations. This parameter change strategy allows the system to maintain resource utilization flexibility while improving preprocessing speed when opportunities arise.
Data Source
AI summary
The disclosed systems and methods for intelligent heterogeneous computation directed to receiving monitoring data and a set of training data, wherein the monitoring data includes an occupancy rate of a preprocessed data queue and a utilization factor of accelerating devices, generating a resource computation job list in accordance with the monitoring data, forwarding jobs, in the resource computation job list to be executed on a central processing unit (CPU), to a CPU worker queue, forwarding control messages to the CPU worker queue, wherein the control messages are associated with jobs in the resource computation job list to be executed on the accelerating devices, and executing, by the accelerating devices, jobs in the resource computation job list to be executed on the accelerating devices.