Coordinated Version Control for Asynchronous Machine Learning Parameter Servers

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional asynchronous execution methods in machine learning lead to undesirable mismatches between parameter versions, causing slow convergence or failure to converge due to the combined design of server-side and client-side aggregation techniques.

Innovation Solution

A coordinated version control system is introduced, featuring a leader parameter server that collects and generates new aggregated parameter sets, broadcasts events, and ensures follower parameter servers match the leader version, while also enabling learners to continue training without waiting for the latest aggregated parameters.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If asynchronous execution is implemented on server side with relaxed aggregation constraint, then training speed is improved, but parameter version consistency deteriorates

Engineering Contradiction:

Improvetraining speedVSAvoidparameter version consistency

Core Design Contradiction:

ProductivityVSStability of the object's composition

Solution Approach 1:

The system performs preliminary actions by having the leader parameter server generate the new aggregated parameter version first, then broadcast this version information to follower parameter servers before they perform their aggregation. This ensures that all followers have the correct target version information in advance, maintaining version consistency while allowing asynchronous execution to proceed.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system implements feedback mechanisms where follower parameter servers detect the broadcast event containing the new leader version, check their own parameter set versions against this new version, and generate matching versions if needed. This feedback loop ensures that all parameter servers maintain consistent versions while allowing asynchronous training to continue.

Inventive Principle:

Principle #23Feedback

2Device complexity

If follower parameter servers operate independently without version checking, then system complexity is reduced, but convergence reliability deteriorates

Engineering Contradiction:

Improvesystem complexityVSAvoidconvergence reliability

Core Design Contradiction:

Device complexityVSReliability

Solution Approach 1:

Follower parameter servers perform self-service by autonomously detecting broadcast events, checking their own parameter versions against the leader's new version, and generating matching versions when needed. This self-service mechanism maintains convergence reliability without requiring complex centralized coordination, as each follower independently ensures its version consistency.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The broadcast event acts as an intermediary mechanism that carries the leader's new version information to follower parameter servers. This intermediary allows followers to obtain version information without direct communication with the leader, simplifying the system architecture while maintaining version consistency through the broadcast detection and version checking mechanism.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS11669502B2Coordinated version control system, method, and recording medium for parameter sensitive applications

Publication Date: 2023.06.06 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

Version vector-based rules are used to facilitate asynchronous execution of machine learning algorithms. The method uses version vector based rule to generate aggregated parameters and determine when to return the parameters. The method also includes coordinating the versions of aggregated parameter sets among all the parameter servers. This allows to broadcast to enforce the version consistency; generate parameter sets in an on-demand manner to facilitate version control. Furthermore the method includes enhancing the version consistency at the learner's side and resolving the inconsistent version when mismatching versions are detected.