Distributed Model Training With Hierarchical Communication

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing distributed training techniques for large machine learning models do not account for network heterogeneity, leading to high networking costs and poor scalability in computing environments with varying bandwidth and latency, which results in communication bottlenecks and inefficient training processes.

Innovation Solution

Implementing scale-aware model partitioning, hierarchical communication strategies, and multi-level gradient synchronization to reduce communication participants and optimize data distribution across heterogeneous computing devices, utilizing techniques such as partition groups, replication groups, and coordinated communication channels.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Quantity of substance

If distributed training is implemented across multiple computing devices, then the capacity to store and process large model state information is improved, but communication overhead and networking costs increase

Engineering Contradiction:

Improvemodel state information capacityVSAvoidcommunication overhead

Core Design Contradiction:

Quantity of substanceVSLoss of energy

Solution Approach 1:

The model state information is segmented into multiple partitions distributed across different computing devices. Each device stores and processes a specific partition, enabling parallel training while reducing the communication burden on any single device. The segmentation allows the system to scale to larger models by adding more devices rather than increasing communication overhead on existing devices.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

A parameter server acts as an intermediary between worker devices in the distributed training system. The parameter server coordinates gradient aggregation and model parameter distribution, reducing direct communication requirements between worker devices. This intermediary structure optimizes network traffic patterns and reduces overall communication overhead in the distributed system.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Productivity

If network heterogeneity is not accounted for in distributed training, then implementation simplicity is maintained, but training efficiency and scalability deteriorate

Engineering Contradiction:

Improvetraining efficiencyVSAvoidcommunication strategy complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The communication strategy is customized for different network conditions and device pairs. Rather than using a uniform communication approach, the system adapts communication parameters such as gradient accumulation frequency and synchronization timing based on local network characteristics. This allows efficient utilization of heterogeneous network resources while maintaining overall system productivity.

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The distributed training system dynamically adjusts communication strategies based on observed network conditions and training progress. Communication frequency, batch size, and synchronization points are adapted in real-time to optimize training efficiency. This dynamic approach allows the system to handle network heterogeneity without requiring complex pre-configured communication strategies.

Inventive Principle:

Principle #15Dynamics

3Loss of time

If all computing devices participate in every communication round, then model convergence is maintained, but communication time and scalability worsen

Engineering Contradiction:

Improvecommunication timeVSAvoidmodel convergence

Core Design Contradiction:

Loss of timeVSReliability

Solution Approach 1:

Not all computing devices participate in every communication round. The system selectively involves subsets of devices based on their current workload, network conditions, and the specific training phase. This partial participation reduces communication time significantly while maintaining model convergence through careful selection of which devices synchronize at which points in the training process.

Inventive Principle:

Principle #16Partial or excessive action

Solution Approach 2:

Communication between devices occurs periodically rather than continuously. The system uses gradient accumulation techniques where devices perform multiple local training steps before synchronizing with other devices. This periodic communication pattern reduces overall communication time while maintaining convergence properties by ensuring sufficient gradient updates are accumulated between synchronization points.

Inventive Principle:

Principle #19Periodic action

Data Source

PatentUS12423578B1Distributed training of machine learning models

Publication Date: 2025.09.23 AMAZON TECH INC

AI summary

A resource set which includes multiple servers with a respective plurality of training computing devices is identified for training a machine learning model. The resource set is subdivided into partition groups, such that each partition group can store a respective replica of state information of the model. The model is trained using the partition groups. The training comprises a multi-stage gathering of a portion of the state information at training computing devices of a particular partition group. Different types of communication channels between training computing devices are used in respective stages of the gathering, including inter-server communication channels in one stage and an intra-server communication channel during another stage. A trained version of the model is stored.