Federated Learning Layer Freezing for Faster Training and Accuracy

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing layer freezing techniques in Federated Learning (FL) fail to balance high accuracy and acceleration, with early-stage freezing leading to accuracy loss and accuracy-guaranteed freezing providing marginal training time improvement.

Innovation Solution

A Parallel Device/Server Freeze Framework that combines aggressive regularization-based layer freezing on the device-side with conservative convergence-based layer freezing on the server-side, using a dual-step strategy to identify layers to freeze based on local and global gradients.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Loss of time

If early-stage layer freezing is applied to accelerate training, then training time is reduced, but final accuracy deteriorates

Engineering Contradiction:

Improvetraining timeVSAvoidfinal accuracy

Core Design Contradiction:

Loss of timeVSManufacturing precision

Solution Approach 1:

The patent segments the freezing decision process into two independent components: device-side aggressive freezing for acceleration and server-side accuracy guarantee for precision. Each side operates with its own freezing criteria and state list, allowing them to serve different objectives without interfering with each other.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent applies different freezing strategies to different locations in the system. Device-side uses aggressive regularization-based freezing with a local state list for speed, while server-side uses conservative convergence-based freezing with a global state list for accuracy. This local differentiation resolves the contradiction by allowing each side to optimize for its specific goal.

Inventive Principle:

Principle #3Local quality

2Manufacturing precision

If accuracy-guaranteed layer freezing is applied to maintain high accuracy, then training time improvement is marginal

Engineering Contradiction:

Improvefinal accuracyVSAvoidtraining speed

Core Design Contradiction:

Manufacturing precisionVSProductivity

Solution Approach 1:

The patent segments the freezing function between device-side aggressive freezing and server-side conservative freezing. The device-side component handles acceleration requirements while the server-side component handles accuracy requirements, allowing both objectives to be met simultaneously through division of labor.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The device-side aggressive freezing performs preliminary freezing actions early in training to achieve acceleration benefits. By freezing layers aggressively on devices before final convergence, the system gains time benefits early while the server-side later ensures accuracy is maintained through conservative freezing decisions.

Inventive Principle:

Principle #10Preliminary action

3Productivity

If a large number of layers are frozen early to reduce computational burden, then acceleration is achieved, but accuracy loss increases substantially

Engineering Contradiction:

Improvecomputational efficiencyVSAvoidaccuracy

Core Design Contradiction:

ProductivityVSManufacturing precision

Solution Approach 1:

The patent applies different freezing criteria to device-side and server-side operations. Device-side uses aggressive regularization-based criteria that freeze more layers for computational efficiency, while server-side uses conservative convergence-based criteria that freeze fewer layers to preserve accuracy. This local quality differentiation allows the system to achieve both efficiency and accuracy.

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The server acts as an intermediary that receives local state lists from devices and produces a global state list that guarantees accuracy. The server-side conservative freezing serves as a mediator that corrects or adjusts the aggressive device-side freezing decisions to ensure final accuracy is maintained.

Inventive Principle:

Principle #24Intermediary (Mediator)

4Manufacturing precision

If pre-trained weight initialization is used for early-stage freezing, then accuracy loss is reduced, but domain shift between pre-training dataset and target dataset causes significant accuracy loss

Engineering Contradiction:

ImproveaccuracyVSAvoiddomain adaptability

Core Design Contradiction:

Manufacturing precisionVSAdaptability or versatility

Solution Approach 1:

The patent extracts the accuracy guarantee function from the device-side aggressive freezing and places it on the server-side. By separating the acceleration function (device-side) from the accuracy guarantee function (server-side), the system eliminates the need for pre-trained weights and achieves both speed and accuracy without domain shift limitations.

Inventive Principle:

Principle #2Taking out (Extraction)

Data Source

PatentUS20260080262A1Accelerating local training and achieving a highly accurate global model for federated learning (FL)

Publication Date: 2026.03.19 RAKUTEN MOBILE INC

AI summary

Global gradients of a global model from a server are received at a plurality of device. Aggressive regularization-based layer freezing is applied at the plurality of devices to the global gradients to identify local layers to freeze in a local model. Based on the local layers identified to freeze, a local state list of the local model is produced. Local gradients produced by the plurality of devices are received at the server. Global gradients are created at the server based on the local gradients. Conservative convergence-based layer freezing is applied at the server to produce a list of frozen layers of the global model based on the global gradients. The list of frozen layers of the global model are provided to the plurality of devices for producing the local state list.