Dataloader Worker Sharding for Disjoint Downloads and Deterministic Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training large-scale machine-learned models is challenging due to the complexity and scale of training data and parameters, leading to issues such as hardware underutilization, high egress fees, and non-deterministic data ordering, which affects model convergence and reproducibility.
Innovation Solution
A method using streaming dataset objects to perform sample downloads with disjoint shard files, ensuring deterministic sample ordering and elastic resumption, thereby reducing startup times and costs by allowing different training machines to download disjoint subsets of the dataset asynchronously.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple computing nodes download the entire dataset for distributed training, then each node can independently process data, but egress fees increase and download time increases
Solution Approach 1:
The dataset is divided into multiple shards, and each computing node is assigned to download and process only specific shards rather than the entire dataset. This segmentation reduces the download volume per node, decreasing egress fees and startup time while maintaining distributed training capability.
Solution Approach 2:
The patent introduces a new dimension of data distribution by organizing shards across multiple storage locations and assigning them to different computing nodes based on their training requirements. This dimensional organization allows nodes to access only necessary data portions, reducing redundant downloads and costs.
2Productivity
If computing nodes wait for large dataset downloads before training, then data can be processed, but hardware accelerators are under-utilized
Solution Approach 1:
Shard files are pre-prepared and staged in cloud storage before training begins. Computing nodes can immediately download and process their assigned shards without waiting for the entire dataset to be prepared, reducing startup time and hardware idle time.
Solution Approach 2:
Each computing node downloads only the partial dataset (specific shards) needed for its training tasks rather than waiting for or processing the entire dataset. This partial action approach allows nodes to start training immediately with available shards.
3Productivity
If data is partitioned across nodes for distributed training, then parallel processing is enabled, but data ordering becomes non-deterministic affecting model convergence
Solution Approach 1:
The patent introduces deterministic parameters into the data sharding and distribution process, such as fixed shard assignment rules and ordered sample selection within shards. These parameter changes ensure that despite parallel processing across multiple nodes, the data ordering remains deterministic and reproducible.
4Device complexity
If the same data is used to train on each machine in distributed training, then simpler data distribution is achieved, but model convergence is impacted due to duplicate samples
Solution Approach 1:
The dataset is segmented into distinct shards that are distributed to different computing nodes. Each node processes only its assigned shards, ensuring that no duplicate samples are processed across nodes. This segmentation maintains simple data distribution mechanics while preventing convergence issues from duplicate data.
Data Source
AI summary
A data processing service accesses data files from data streams, each data file including samples to be processed for training a machine-learning model. The service converts the data files to discrete shard files, each shard file comprising a subset of samples. Each sample ID is mapped to a shard index of a respective shard file that includes the sample. The service may generate partition tensors that partition sample index spaces into a number of physical nodes, devices, workers, and batches. The service may shuffle the shard files and divide the sample IDs into a number of logical nodes and shuffle the sample IDs. The service generates shuffled sample ID arrays that map the sample indices to the sample IDs. During training, workers download disjoint shard files and map the assigned sample indices to corresponding batches of sample IDs based on the shuffled sample ID arrays.