Parameter Version Vectors for Deterministic Distributed Replay

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

In distributed machine learning computations, existing technologies face challenges in achieving deterministic replay, especially in asynchronous execution environments, where stragglers can lead to unrepeatable execution and make debugging and testing difficult due to non-deterministic results.

Innovation Solution

The implementation of parameter version vectors and a parameter server system that stores and aggregates parameter versions across epochs, allowing for deterministic replay by generating aggregated parameter sets and providing them to learner instances for reproducible computations.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If distributed execution is performed in asynchronous environment, then productivity is improved, but reliability of deterministic replay deteriorates

Engineering Contradiction:

Improveexecution speedVSAvoiddeterministic replay capability

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The system performs preliminary actions by storing parameter versions and epoch execution histories during the initial distributed execution. This allows the execution environment to be captured and preserved for later deterministic replay without affecting the original asynchronous execution performance.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system creates a copy of the execution state by storing parameter versions and epoch histories in a parameter server. This copy enables deterministic replay while the original system continues to operate asynchronously without interruption.

Inventive Principle:

Principle #26Copying

2Reliability

If parameter versions are stored and aggregated for deterministic replay, then reliability of replay is improved, but device complexity increases

Engineering Contradiction:

Improvedeterministic replay capabilityVSAvoidparameter server structure

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The parameter server performs multiple functions: it acts as a storage mechanism for parameter versions, maintains epoch execution histories, and provides parameter aggregation during replay. This multi-functionality reduces the need for separate dedicated components for each function.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

The parameter server serves as an intermediary between learner instances during asynchronous execution and during deterministic replay. It mediates parameter storage and retrieval operations, simplifying the interface while managing the complexity of version control and aggregation internally.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Measurement precision

If parameter version vectors are created to track parameter usage, then measurement precision of parameter tracking is improved, but loss of time in aggregation increases

Engineering Contradiction:

Improveparameter version trackingVSAvoidaggregation time

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

Parameter versions are recorded and tracked in real-time during the original execution using version vectors. This preliminary capture of parameter states eliminates the need for time-consuming tracking during replay, as all version information is already captured and organized.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system uses lightweight parameter version representations that can be quickly serialized and stored. These compact version objects minimize storage overhead and enable rapid aggregation during replay, reducing the time penalty associated with tracking parameter versions.

Inventive Principle:

Principle #27Cheap short-living objects (Disposable)

Data Source

PatentUS10643150B2Parameter version vectors used for deterministic replay of distributed execution of workload computations

Publication Date: 2020.05.05 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

A method includes storing parameter versions utilized by learner instances in each of two or more epochs in a parameter receiving buffer of a parameter server, the learner instances performing distributed execution of workload computations of a machine learning algorithm. The method also includes creating a parameter roster in the parameter server comprising parameter version vectors specifying the parameter versions used by each of the learner instances during each of the two or more epochs. The method further includes generating one or more aggregated parameter sets for storage in an aggregated parameters buffer by aggregating parameter versions from the parameter receiving buffer based on the parameter version vectors in the parameter roster and providing aggregated parameter sets from the aggregated parameters buffer to the learner instances for deterministic replay of the distributed execution of the workload computations of the machine learning algorithm.