Model Parameter Fusion via Node Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current model parameter fusion methods in machine learning require high performance from parameter servers, leading to potential system breakdowns, and involve significant data transmission and storage, which can be inefficient.
Innovation Solution
A method where model parameters are divided into blocks and distributed among nodes for fusion, allowing for simultaneous data transmission and reception using full-duplex networks, with a fusion controller determining participating nodes based on preset conditions, enabling dynamic resource adjustment and improved network utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a parameter server consolidates and fuses model parameters from all nodes after multiple iterative computations, then model parameter fusion can be completed, but the parameter server requires high performance and system breakdown is very likely to occur
Solution Approach 1:
The patent segments the model parameter fusion process by dividing model parameters into multiple parameter blocks and distributing them to different nodes. Each node independently fuses specific parameter blocks from multiple data subsets, eliminating the need for a single high-performance parameter server. This segmentation resolves the contradiction by distributing the fusion burden across multiple nodes, thereby improving system reliability while reducing the performance requirements of individual components.
2Reliability
If model parameters are fused by transmitting data between nodes, then parameter fusion can be achieved, but a relatively large amount of data needs to be stored and transmitted
Solution Approach 1:
The patent applies local quality by having each node fuse only specific parameter blocks (e.g., first parameter blocks, second parameter blocks, etc.) rather than all parameter blocks. Each node processes a subset of parameters locally, reducing the amount of data that needs to be transmitted and stored. This localized approach maintains parameter fusion capability while significantly reducing data transmission volume.
3Measurement precision
If all nodes perform iterative computations on multiple data subsets, then comprehensive model training can be achieved, but the computation time and resources increase
Solution Approach 1:
The patent segments both the model parameters and the computation tasks. Different nodes perform iterative computations on different data subsets and fuse different parameter blocks. This parallel segmentation allows comprehensive model training to be achieved more efficiently by distributing computations across multiple nodes simultaneously, thereby reducing the overall computation time while maintaining training comprehensiveness.
Data Source
AI summary
Embodiments of the present invention provide a model parameter fusion method and apparatus, which relate to the field of machine learning and intend to reduce a data transmission amount and implement dynamical adjustment of computing resources during model parameter fusion. The method includes: dividing, by an ith node, a model parameter of the ith node into N blocks, where the ith node is any node of N nodes that participate in a fusion, and 1≤i≤N≤M; receiving, by the ith node, ith model parameter blocks respectively sent by other nodes of the N nodes than the ith node; fusing, by the ith node, an ith model parameter block of the ith node and the ith model parameter blocks respectively sent by the other nodes, so as to obtain the ith general model parameter block; and distributing, by the ith node, the ith general model parameter block to the other nodes of the N nodes.