Large Model Federated Learning with Incremental Parameter Aggregation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Federated learning for large models is inefficient due to the lengthy and complex exchange of model parameters, which results in significant costs.

Innovation Solution

A method where client devices train a target large model with a frozen original parameter and an incremental parameter, aggregating only the incremental parameters across devices to update the model, and using aggregation weights based on training sample ratios and similarities to enhance convergence.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If all model parameters are exchanged between client devices and server in federated learning, then collaborative training can be achieved, but the transmission cost and time increase significantly due to the large number of parameters in large models

Engineering Contradiction:

Improvecollaborative training effectivenessVSAvoidparameter exchange time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The patent extracts only the incremental parameters (differences between original and updated parameters) from the complete model parameters for transmission. Instead of exchanging all billion+ parameters, only the changed portions are transmitted between client devices and server, dramatically reducing communication overhead while preserving collaborative training effectiveness.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The model parameters are segmented into original parameters and incremental parameters. The patent separates the large parameter set into a base model (original parameters) and updates (incremental parameters), transmitting only the incremental segment which constitutes a small fraction of the total parameter size.

Inventive Principle:

Principle #1Segmentation

2Reliability

If all model parameters are exchanged between client devices and server in federated learning, then model updates can be performed, but the transmission cost and complexity increase significantly due to the large number of parameters in large models

Engineering Contradiction:

Improvemodel update accuracyVSAvoidparameter exchange complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

Inventive Principle:

Principle #2Taking out (Extraction)

3Quantity of substance

If incremental parameters are used instead of full parameters in federated learning, then transmission volume is reduced, but the aggregation process becomes more complex

Engineering Contradiction:

Improvedata transmission volumeVSAvoidaggregation process complexity

Core Design Contradiction:

Quantity of substanceVSDevice complexity

Solution Approach 1:

The patent changes the parameter representation from absolute model parameters to incremental parameter differences. By transforming the transmission format from full parameters to parameter differences (increments), the data volume is reduced while the aggregation process uses simple addition operations to combine incremental updates with the base model.

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS20250245571A1Large model federated learning methods and apparatuses, storage media, and electronic devices

Publication Date: 2025.07.31 ALIPAY (HANGZHOU) INFORMATION TECH CO LTD

AI summary

Described is large model federated learning applied to a server. For each participating client device, an incremental parameter is sent by the client device after the client device trains a target large model of the client device, where a model parameter of the client device includes an original parameter and an incremental parameter, a magnitude of the incremental parameter is less than a magnitude of the original parameter, the original parameter remains unchanged, and the incremental parameter changes. The incremental parameter of the client device is aggregated by using incremental parameters of all client devices to obtain an aggregation parameter returned to the client device and used to update the incremental parameter of the client device. Based on the original parameter and an updated incremental parameter, redetermining a model parameter, used until target large model convergence in retraining the target large model.