Node Cache Scheduling for AI Training Storage Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In AI resource management platforms, scheduling of training tasks is hindered by insufficient storage resources on nodes, where large data sets are often cached, leading to idle CPU and memory, and a lack of suitable nodes for caching, due to limited storage and inefficient resource allocation.
Innovation Solution
A data-set-and-node-cache-based scheduling strategy that selects nodes based on available storage and task requirements, involving scoring nodes based on storage space, data set cache usage, and cache deletion to optimize resource utilization and reduce download times.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If more tasks using large data sets are scheduled to the same node, then the utilization of CPU and memory is improved, but the storage resources of the node become insufficient
Solution Approach 1:
The patent transitions from single-node data set caching to multi-node distributed caching architecture. Instead of concentrating all data sets on one node, the system distributes data sets across multiple nodes in the cluster, adding the dimension of spatial distribution to resolve the storage capacity bottleneck while maintaining high CPU and memory utilization.
Solution Approach 2:
The patent segments the data set storage function across multiple nodes rather than concentrating it on a single node. Each node caches specific data sets locally, dividing the overall storage burden into manageable segments distributed throughout the cluster, thereby increasing total available storage capacity.
2Reliability
If all computing nodes simultaneously cache large numbers of data sets, then data set availability is improved, but storage resources of nodes become insufficient and suitable nodes for caching are unavailable
Solution Approach 1:
The patent creates a universal caching system where any node in the cluster can serve as a cache for any data set required by training tasks. Instead of dedicating specific nodes to specific data sets, the system enables multi-functionality where nodes can dynamically cache and serve different data sets based on task requirements, improving overall data set availability without proportionally increasing storage resources.
Solution Approach 2:
The patent merges the caching capabilities of multiple nodes into a unified distributed cache system. By combining the storage resources of all nodes in the cluster, the system achieves greater total capacity and flexibility, allowing any node to contribute its storage to the collective caching pool, thereby improving data set availability while efficiently utilizing available storage resources.
3Ease of operation
If manual data set download is optimized to automatic download, then operational efficiency is improved, but download time and node waiting time increase when storage is limited
Solution Approach 1:
The patent implements preliminary action by pre-caching data sets on multiple nodes before they are actually needed for training tasks. The system proactively identifies data sets that are likely to be needed and distributes them across the cluster in advance, so when training tasks are initiated, the data sets are already available locally, eliminating download delays and reducing node waiting time.
Solution Approach 2:
The patent introduces a distributed cache system as an intermediary between the central storage and the training tasks. Instead of directly downloading data sets from central storage to individual nodes when needed, the system uses the distributed cache across multiple nodes as an intermediate layer, allowing tasks to access data sets from nearby cached locations, thereby reducing download time and improving operational efficiency.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Disclosed is a data set and node cache-based scheduling method, which includes: obtaining storage resource information of each host node; in response to receiving a training task, obtaining operation information of the training task, and according to the operation information and the storage resource information, screening host nodes that satisfy the space required by the training task; in response to no host nodes satisfying the space required by the training task, scoring each host node according to the storage resource information; according to the scoring results, selecting from among all of the host nodes a host node to perform execution that is used to execute the training task; and obtaining and deleting an obsolete data set cache in the host node to perform execution, and executing the training task in the host node to perform execution. Further disclosed is a corresponding device. In the present disclosure, a training task may be effectively operated in a node the storage space of which is temporarily insufficient.