Distributed Model Training with Global Index Data Exchange
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed model training faces challenges with ultra-large datasets that exceed storage space, leading to increased storage overheads and reduced read efficiency when datasets are stored remotely.
Innovation Solution
A model training system that distributes the global dataset among training nodes, using a global index list to manage and exchange data, ensuring each node obtains complete training data through distributed storage and data exchange without additional storage costs, improving read efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If the dataset is stored remotely in a data center or database, then storage overhead is reduced, but read efficiency decreases
Solution Approach 1:
The patent divides the global dataset into separate local datasets stored on different training devices, with each device holding a portion of the data. This segmentation allows devices to access data locally without relying on remote storage, thereby improving read efficiency while maintaining distributed storage architecture.
Solution Approach 2:
The patent introduces a global index list as an intermediary that maps data indices across all training devices. This intermediary enables efficient data location and retrieval by coordinating the distributed local datasets, allowing nodes to quickly find and access required data without direct remote storage dependencies.
2Speed
If each training node stores a complete copy of the dataset, then read efficiency is improved, but storage overhead increases
Solution Approach 1:
Instead of storing complete datasets on all nodes, the patent segments the dataset into distributed portions where each training device stores only its local dataset. This reduces storage overhead while maintaining efficient local access through the global index list for coordination.
3Ease of operation
If distributed training uses sequential data access based on data index, then data processing is simplified, but training efficiency decreases when dataset is extremely large
Solution Approach 1:
The patent segments the data access process into local dataset queries guided by the global index list. Each training device processes its local data independently while coordinated through the index list, enabling parallel processing that maintains simplicity while significantly improving training efficiency for large datasets.
Data Source
AI summary
The present invention provides a model training system, a model training method, a training device, and a training node. The system includes: n training devices, in which a total of M training nodes are deployed. Each training device stores a global index list and a dataset corresponding to a training node deployed in the training device, and the global index list indicates indexes of data in datasets respectively corresponding to the M training nodes. Each training device is configured to: shuffle the global index list by using a random number seed, and query, based on a shuffled list, the dataset stored in the training device, to obtain exchange data of the training node deployed in the training device, where the exchange data is hit data obtained through data query.