Asynchronous Model Fusion Buffering During Large-Model Pre-Training

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current online fusion methods during the pre-training of large models reduce the efficiency of the pre-training process by interleaving model parameter fusion with training, leading to inefficiencies.

Innovation Solution

A model fusion method involving a main process that caches intermediate model parameters into a main buffer and a sub-process that asynchronously reads and fuses these parameters, utilizing high-speed memory and inter-process communication to improve efficiency and accuracy.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If online fusion is used to fuse intermediate model parameters during pre-training, then model performance is improved, but pre-training efficiency deteriorates

Engineering Contradiction:

Improvemodel performanceVSAvoidpre-training efficiency

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The system divides the fusion process into two independent segments: a main process that continues pre-training without interruption, and a sub-process that handles parameter fusion operations. This segmentation allows both processes to execute independently, resolving the contradiction between maintaining model performance through fusion and preserving pre-training efficiency.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The main process performs preliminary actions by caching intermediate model parameters into a main buffer during pre-training. This preliminary caching enables the sub-process to perform fusion operations on cached parameters without blocking the main pre-training workflow, thus maintaining both model performance improvement and training efficiency.

Inventive Principle:

Principle #10Preliminary action

2Measurement precision

If pre-training and parameter fusion are interleaved, then model accuracy is improved, but processing speed deteriorates

Engineering Contradiction:

Improvemodel accuracyVSAvoidprocessing speed

Core Design Contradiction:

Measurement precisionVSSpeed

Solution Approach 1:

A main buffer serves as an intermediary data structure between the main pre-training process and the sub-process fusion operations. The buffer stores intermediate parameters, allowing the sub-process to read and fuse parameters without the main process needing to pause or slow down, thus maintaining both accuracy and processing speed.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The main process continues its pre-training actions continuously without interruption, while the sub-process performs fusion operations on cached parameters. This continuity ensures that the useful action of pre-training maintains its speed, while the fusion operations still improve model accuracy through continuous parameter refinement.

Inventive Principle:

Principle #20Continuity of useful action

Data Source

PatentUS20260010833A1Model fusion method, electronic device and storage medium

Publication Date: 2026.01.08 BEIJING BAIDU NETCOM SCI & TECH CO LTD

AI summary

A model fusion method includes: calling a main process during a pre-training process of a large model, to cache intermediate model parameters obtained during the pre-training process into a main buffer; and calling a sub-process via the main process to read the intermediate model parameters from the main buffer and perform a parameter fusion process based on the intermediate model parameters.