OAM Packet-Based Node State Synchronization in Distributed Deep Learning

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional distributed deep learning systems face a slowdown in cooperative processing speed as the number of computation nodes connected to a communication network increases, due to high communication overheads.

Innovation Solution

A distributed deep learning system where each computation node includes an arithmetic operation device, storage, and network processing device with transmission and reception circuits, utilizing Operation Administration and Maintenance (OAM) packets to manage and synchronize data transmission among nodes, reducing the need for redundant calculations and memory access.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If the number of computation nodes is increased to increase parallel processing, then learning speed is improved, but communication overhead increases sharply

Engineering Contradiction:

Improvelearning speedVSAvoidcommunication overhead time

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The patent segments the arithmetic operation results into multiple packets and transmits them through different paths (data transmission path and OAM packet path) simultaneously. By dividing the communication task and using parallel transmission channels, the system reduces the time loss associated with communication overhead while maintaining high learning speed with increased computation nodes.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent uses OAM (Operation Administration and Maintenance) packets as an intermediary mechanism to carry state information between computation nodes. These packets enable efficient synchronization and state tracking without requiring separate dedicated communication channels, thereby reducing communication overhead time while supporting scalability.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Adaptability or versatility

If software is used for arithmetic operation processing and data transmission, then system flexibility is maintained, but processing speed decreases due to large overheads

Engineering Contradiction:

Improvesystem flexibilityVSAvoidprocessing speed

Core Design Contradiction:

Adaptability or versatilityVSSpeed

Solution Approach 1:

The patent makes the OAM processing circuit capable of multiple functions: it processes both data transmission and state information transmission, and can operate in different modes (normal data path and alternative data transmission path). This multi-functionality maintains system flexibility while achieving hardware-level processing speed through the dedicated circuit implementation.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

The patent replaces software-based arithmetic operation processing and data transmission with hardware-based OAM processing circuits. This substitution of mechanical/software systems with hardware circuits dramatically increases processing speed while the circuits are designed to maintain system flexibility through configurable operation modes.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

3Productivity

If constituent parameters are transmitted among computation nodes for mean value calculation, then distributed learning is achieved, but communication processing time increases

Engineering Contradiction:

Improvedistributed learning capabilityVSAvoidparameter transmission time

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The patent performs preliminary transmission of state information through OAM packets before the actual data transmission is completed. By提前 notifying other nodes about the state and preparing for data reception, the system reduces the overall communication processing time while maintaining distributed learning capability.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent enables continuous useful action by using the OAM packet transmission path as an alternative data transmission path when the normal path is congested or slow. This continuity ensures that parameter transmission for distributed learning continues without interruption, reducing total communication time while maintaining learning capability.

Inventive Principle:

Principle #20Continuity of useful action

Data Source

PatentUS20220398457A1Distributed Deep Learning System and Distributed Deep Learning Method

Publication Date: 2022.12.15 NIPPON TELEGRAPH & TELEPHONE CORP

AI summary

A distributed deep learning system includes a plurality of computation nodes mutually connected through a communication network, wherein each of the plurality of computation nodes includes a network processing unit including: a reception section that receives an OAM packet indicating states of the plurality of computation nodes; an OAM processing section that makes a record, in the OAM packet received by the reception section, of whether or not a partial arithmetic operation result is outputted from an arithmetic operation unit of the own node; and a transmission section that transmits the OAM packet including the record made by the OAM processing section to another computation node, wherein the OAM processing section, depending on the state of the other computation node indicated by the OAM packet, causes the transmission section to transmit the partial arithmetic operation result stored in a storage unit to the other computation node.