Fine-Grained Synchronization for Distributed Machine Learning

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing synchronization methods in data-parallel jobs for distributed machine learning, such as barrier primitives, are slow, inefficient, and can lead to mixed-version issues and network resource spikes, particularly in large-scale surveillance applications.

Innovation Solution

Implementing a fine-grained synchronization mechanism using per-receiver notifications and acknowledgments, which allows data parallel workers to continue iterations only after data consumption is confirmed, reducing the need for global barriers and improving consistency and performance.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If barrier synchronization is used to synchronize data parallel workers, then all workers can be synchronized, but the synchronization speed is slow and network resource spikes occur

Engineering Contradiction:

Improvesynchronization correctnessVSAvoidsynchronization speed

Core Design Contradiction:

ReliabilityVSSpeed

Solution Approach 1:

The patent segments the global barrier synchronization into fine-grained, per-receiver acknowledgments. Instead of requiring all workers to wait at a single barrier point, each worker receives individual acknowledgments from its specific receivers, allowing independent progression through synchronization points and eliminating the need for slow global barriers

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces dynamic synchronization where workers can proceed asynchronously based on their individual completion status. The system transitions from static barrier-based synchronization to dynamic acknowledgment-based synchronization, where each worker's progression is independently controlled by its specific communication needs

Inventive Principle:

Principle #15Dynamics

2Reliability

If global barriers are used for synchronization, then all workers are coordinated, but mixed-version issues and torn-reads occur

Engineering Contradiction:

Improvesynchronization coordinationVSAvoiddata version consistency

Core Design Contradiction:

ReliabilityVSManufacturing precision

Solution Approach 1:

The patent implements feedback mechanisms where receivers send acknowledgments back to senders confirming data consumption. This feedback loop ensures that workers only proceed when their specific data dependencies are satisfied, preventing mixed-version issues and torn-reads by providing explicit confirmation of data state transitions

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The patent replaces the mechanical barrier synchronization system with an acknowledgment-based notification system. Instead of forcing all workers to stop at barrier points, the system uses software-based acknowledgments to signal individual completion, eliminating the rigid coordination mechanism that causes version consistency problems

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

3Reliability

If barrier synchronization is used, then workers are synchronized, but network bandwidth is overloaded due to simultaneous transmissions

Engineering Contradiction:

Improvesynchronization guaranteeVSAvoidnetwork bandwidth utilization

Core Design Contradiction:

ReliabilityVSQuantity of substance

Solution Approach 1:

The patent segments network communications into individual acknowledgment messages between specific worker pairs rather than simultaneous bulk transmissions at barrier points. This segmentation distributes network traffic over time and across different communication channels, preventing bandwidth overload while maintaining synchronization guarantees

Inventive Principle:

Principle #1Segmentation

4Speed

If asynchronous operation is implemented without barriers, then synchronization speed improves, but correctness semantics break down

Engineering Contradiction:

Improveiteration progression speedVSAvoidcorrectness semantics

Core Design Contradiction:

SpeedVSReliability

Solution Approach 1:

The patent maintains correctness semantics through feedback acknowledgments that confirm data consumption before allowing asynchronous progression. Each worker receives explicit feedback from its receivers that the data has been processed, ensuring correctness while enabling asynchronous operation without global barriers

Inventive Principle:

Principle #23Feedback

Data Source

PatentUS10402235B2Fine-grain synchronization in data-parallel jobs for distributed machine learning

Publication Date: 2019.09.03 NEC CORP

AI summary

A computer-implemented method and computer processing system are provided. The method includes synchronizing, by a processor, respective ones of a plurality of data parallel workers with respect to an iterative distributed machine learning process. The synchronizing step includes individually continuing, by the respective ones of the plurality of data parallel workers, from a current iteration to a subsequent iteration of the iterative distributed machine learning process, responsive to a satisfaction of a predetermined condition thereby. The predetermined condition includes individually sending a per-receiver notification from each sending one of the plurality of data parallel workers to each receiving one of the plurality of data parallel workers, responsive to a sending of data there between. The predetermined condition further includes individually sending a per-receiver acknowledgement from the receiving one to the sending one, responsive to a consumption of the data thereby.