Model Parameter Combination Method for Machine Learning Systems

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current model parameter combination methods in machine learning require high performance from parameter servers and result in large data transmission volumes, leading to inefficiencies and potential system shutdowns.

Innovation Solution

The proposed method involves a machine learning system with parameter collection and delivery groups, where model parameters are combined within groups when certain conditions are met, and then sent to corresponding delivery groups, reducing the burden on parameter servers and data transmission.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Quantity of substance

If model parameters are combined by collecting from all nodes and combining at a parameter server, then comprehensive parameter aggregation is achieved, but parameter server performance requirements increase and system shutdown risk increases

Engineering Contradiction:

Improveparameter aggregation completenessVSAvoidparameter server stability

Core Design Contradiction:

Quantity of substanceVSReliability

Solution Approach 1:

The patent divides nodes into parameter collection groups and parameter delivery groups, segmenting the parameter combination process into local group combinations and inter-group aggregations. This reduces the burden on any single parameter server by distributing combination operations across multiple groups, thereby lowering performance requirements and improving stability.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces intermediate parameter combination nodes within each parameter collection group that perform local parameter combination before delivering to the parameter delivery group. This intermediary layer reduces the direct burden on the main parameter server by pre-aggregating parameters at the group level, thus improving overall system reliability.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Quantity of substance

If model parameters are combined by transmitting from each node to another specified node, then parameter combination is achieved, but data transmission volume increases

Engineering Contradiction:

Improveparameter combination completenessVSAvoiddata transmission volume

Core Design Contradiction:

Quantity of substanceVSLoss of energy

Solution Approach 1:

The patent merges parameter transmission and combination operations within parameter collection groups, where nodes combine parameters locally before delivering to the parameter delivery group. This merging reduces the total data transmission volume by avoiding redundant transmissions of intermediate parameter sets across the entire network.

Inventive Principle:

Principle #5Merging (Combining)

Solution Approach 2:

The patent performs preliminary parameter combination within each parameter collection group before the final aggregation at the parameter delivery group. This preliminary action reduces the amount of data that needs to be transmitted in subsequent steps, as parameters are already combined at the group level.

Inventive Principle:

Principle #10Preliminary action

3Productivity

If multiple data subsets are allocated to different nodes for iteration calculation, then data processing parallelism is improved, but parameter combination complexity increases

Engineering Contradiction:

Improvedata processing efficiencyVSAvoidparameter combination process complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent segments nodes into parameter collection groups and parameter delivery groups with distinct functions. Parameter collection groups focus on data processing and local parameter combination, while parameter delivery groups handle parameter aggregation and distribution. This segmentation simplifies the overall parameter combination process by dividing complex operations into manageable, specialized tasks.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent designs parameter collection groups to serve multiple functions: data processing, local parameter combination, and participation in inter-group aggregation. This multi-functionality reduces the need for separate dedicated components, thereby simplifying the overall system architecture while maintaining processing efficiency.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS11386350B2Model parameter combination method and apparatus

Publication Date: 2022.07.12 HUAWEI TECH CO LTD

AI summary

The method and apparatus that are applied to a machine learning system which includes at least one parameter collection group and at least one parameter delivery group. Each parameter collection group is corresponding to at least one parameter delivery group. The method includes: when any parameter collection group meets an intra-group combination condition, combining model parameters of M nodes in the parameter collection group to obtain a first model parameter of the parameter collection group, where a smallest quantity s of combination nodes in the parameter collection group≤M≤a total quantity of nodes included in the parameter collection group; and sending the first model parameter of the parameter collection group to N nodes in a parameter delivery group corresponding to the parameter collection group, where 1≤N≤a total quantity of nodes included in the parameter delivery group corresponding to the parameter collection group.