Resource Management Platform Task Allocation for AI Data Sets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing resource management platforms waste time and resources due to the large amount of data sets needed for artificial intelligence model training and testing, as they allocate tasks based solely on node performance without considering data set availability and size, leading to inefficient task processing.
Innovation Solution
A task allocation method and system that acquires data set distribution information, scores nodes based on data set size, and selects nodes with the required data sets meeting preset requirements for task allocation, reducing unnecessary data downloads and network pressure.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If tasks are allocated based on node performance without considering data set availability, then task allocation speed is improved, but data download time and network pressure increase significantly
Solution Approach 1:
The system performs preliminary actions by pre-distributing data sets to nodes before task allocation occurs. The data set management module proactively copies required data sets from storage to multiple nodes in advance, so when tasks are allocated, nodes already have the necessary data locally, eliminating download delays during task execution
Solution Approach 2:
A data set management module acts as an intermediary between storage and nodes. This module handles data set distribution, caching, and management, mediating between the need for data availability and the desire for fast task allocation by maintaining local copies of data sets on nodes
2Reliability
If data sets are downloaded to each node before task processing, then task processing reliability is improved, but resource consumption and time cost increase
Solution Approach 1:
The system applies local quality by distributing data sets selectively to specific nodes based on their assigned tasks rather than universally to all nodes. Each node receives and stores only the data sets required for its specific task, creating local data availability without redundant duplication across the entire system
Solution Approach 2:
The system changes the parameter of data location from centralized storage to distributed local storage on nodes. By transforming where data resides (from central storage to node-local cache), the system ensures reliable access for task processing while reducing the total amount of data that needs to be actively managed and transferred
3Productivity
If data sets are pre-distributed to multiple nodes, then task processing efficiency is improved, but system complexity increases
Solution Approach 1:
The data set management module performs multiple functions: it manages data set storage, handles distribution to nodes, maintains local caches, and coordinates with the task allocation module. This multi-functional approach consolidates complexity into a single management entity rather than requiring separate mechanisms for each function
Solution Approach 2:
The system implements feedback mechanisms where nodes report their data set status and task completion status back to the data set management module. This feedback allows the system to dynamically adjust data distribution, track which nodes have which data sets, and optimize future allocations based on actual system state
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The present application discloses a task allocation method and system based on a resource management platform. The method comprises: receiving an artificial intelligence model training and/or testing task and a name of data set required for processing the task; acquiring data set distribution information of a plurality of nodes; judging if the node has the required data sets according to names of the data sets in the node; and selecting a node with the size of the required data set meeting preset requirements for task allocation according to the size of the required data set in the node if the node has the required data set. It may be seen that, in the present application, the situation of data sets in a node is taken into account during task allocation, and the node with the size of the required data set meeting preset conditions is selected for task allocation, such that the node does not need to download the required data set or reduce the frequency of downloading data sets by a node, thereby improving the efficiency in processing tasks.