Optimizer State Sharding for Node-Failure Recovery in FSDP
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional fully sharded data parallel (FSDP) techniques in distributed machine learning lack resilience to node failures, leading to significant disruptions and the need to re-initialize training processes when a compute node becomes unavailable.
Innovation Solution
Implement a failure resiliency approach in the FSDP framework by sharing optimizer shards among compute nodes, replicating and partitioning these shards into portions, and distributing them across the network to maintain optimizer states, ensuring uninterrupted training even in the event of node failures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If optimizer states are stored only at individual compute nodes in FSDP, then device complexity is reduced, but reliability deteriorates due to node failures causing training interruptions
Solution Approach 1:
The optimizer states are segmented into multiple shards, with each compute node storing only a portion of the total optimizer states. This segmentation allows the system to distribute the storage burden while enabling fault tolerance through replication across nodes.
Solution Approach 2:
Optimizer state shards are copied and distributed across multiple compute nodes. When a node fails, its optimizer state shard can be recovered from another node that holds a copy, ensuring training continuity without requiring complex backup systems.
2Reliability
If optimizer states are replicated across all compute nodes, then reliability improves for fault tolerance, but device complexity increases due to synchronization overhead
Solution Approach 1:
Instead of replicating complete optimizer states across all nodes, the system segments optimizer states into shards and distributes them. Each node stores only specific shards, reducing the synchronization overhead while maintaining fault tolerance through strategic replication of individual shards.
3Ease of operation
If full optimizer states are stored at each compute node, then ease of operation improves for local updates, but device complexity increases and memory usage grows
Solution Approach 1:
Optimizer states are segmented into manageable shards that fit within the memory constraints of individual compute nodes. Each node performs local updates on its assigned shards, maintaining ease of operation while avoiding the memory burden of storing complete optimizer states.
Solution Approach 2:
The system transitions from a single-node complete storage model to a multi-node distributed shard storage model. This dimensional change from centralized to distributed architecture allows local updates to proceed efficiently while managing memory usage through partitioning.
Data Source
AI summary
Systems and methods are provided for failure resiliency in distributed training of machine learning (ML) models. Examples include a plurality of compute nodes storing optimizer shards of a plurality of optimizer shards and a first compute node storing a first optimizer shard of optimizer states. The first compute node can store optimizer shard portions, each of which can be received from a respective compute node of the plurality of compute nodes and can be a replica of a portion of a respective optimizer shard of the plurality of optimizer shards, stored at the respective compute node. Responsive to a failure of a compute node of the plurality of compute nodes, the first compute node can update the first optimizer shard with an optimizer shard portion corresponding to the failed compute node and the ML model can be trained based on the updated first optimizer shard.