FSDP Shard Replication for Node-Failure-Resilient ML Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional fully sharded data parallel (FSDP) techniques lack resilience to node failures in distributed machine learning, leading to significant disruptions and the need to re-initialize training processes when a compute node becomes unavailable.
Innovation Solution
Implement a failure resiliency approach in the FSDP framework by sharing and replicating model parameter shards among compute nodes, using all-gather operations during forward and backward propagation phases to maintain model states and minimize communication overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If model parameter shards are distributed across compute nodes in conventional FSDP, then memory usage is optimized and parallel training efficiency is improved, but resilience to node failures deteriorates
Solution Approach 1:
The patent implements copying by creating redundant copies of model parameter shards across multiple compute nodes. Each compute node stores not only its own shard but also copies of shards from other nodes, enabling failover capability. This copying mechanism allows the system to tolerate node failures while maintaining parallel training efficiency, as remaining nodes can continue training using their local copies and received copies of failed nodes' shards.
2Reliability
If redundant shard copies are stored across compute nodes to improve failure resilience, then reliability is improved, but communication overhead and memory usage increase
Solution Approach 1:
The patent applies local quality by implementing a selective copying strategy where each compute node stores copies of other nodes' shards based on specific criteria such as failure probability, shard importance, or load balancing considerations. This localized approach to redundancy means not all nodes copy all shards, thereby reducing overall communication overhead and memory usage while still achieving acceptable failure resilience. The system optimizes the distribution of redundant copies to balance reliability improvements against resource consumption.
Data Source
AI summary
Systems and methods are provided for failure resiliency in distributed training of machine learning (ML) models. Examples include a plurality of compute nodes storing shards of a plurality of shards of model states of an ML model, and a first compute node storing a first shard of model states of the ML model. The first compute node can store a plurality of shard portions. Each shard portion can be received from a respective compute node of the plurality of compute nodes and can be a replica of a portion of a respective shard, of the plurality of shards, stored at the respective compute node. Responsive to a failure of a compute node of the plurality of compute nodes, the first compute node can update the first shard with a shard portion corresponding to the failed compute node and the ML model can be trained based on the updated first shard.