Distributed Processing Ring with Dual Ports for Parallel Aggregation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

In distributed deep learning systems, the increase in the number of distributed processing nodes leads to reduced processing speed due to increased aggregation communication time and dispatch communication time, limiting the effectiveness of deep learning performance.

Innovation Solution

A distributed processing system with N nodes arranged in a ring shape, utilizing two communication ports for simultaneous bidirectional communication, allowing parallel aggregation and dispatch communication processes, and issuing warnings for consolidation delays to manage communication anomalies.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If the number of distributed processing nodes is increased to improve deep learning speed, then the processing capacity increases, but the aggregation communication time and dispatch communication time increase, reducing the overall processing speed

Engineering Contradiction:

Improvedeep learning processing speedVSAvoidaggregation communication time and dispatch communication time

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The patent segments the communication process into two independent parallel channels: aggregation communication (collecting distributed data from all nodes to a consolidation node) and dispatch communication (distributing consolidated data from the consolidation node back to all nodes). By using separate communication paths for these two functions, the system can perform both operations simultaneously, preventing the sequential execution bottleneck that would otherwise occur with a single communication channel.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces a dimensional change by adding a second communication port to each distributed processing node, transforming the single-channel communication system into a dual-channel system. This allows aggregation and dispatch communications to occur in parallel along different communication dimensions, effectively resolving the time loss issue by eliminating the sequential dependency between these two communication processes.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

2Device complexity

If a single consolidation processing node is used to aggregate data from all distributed processing nodes, then the system structure is simple, but the communication port speed becomes a bottleneck, limiting the aggregation communication speed

Engineering Contradiction:

Improvesystem structureVSAvoidaggregation communication speed

Core Design Contradiction:

Device complexityVSSpeed

Solution Approach 1:

The patent makes each distributed processing node multi-functional by equipping it with both a first communication port for aggregation communication and a second communication port for dispatch communication. The consolidation processing node also possesses dual communication capabilities, allowing it to simultaneously receive data from all nodes via the first communication port and send consolidated data back to all nodes via the second communication port. This universal dual-function design eliminates the single-point bottleneck while maintaining relatively simple system structure.

Inventive Principle:

Principle #6Universality (Multi-functionality)

3Productivity

If distributed processing is implemented to improve gradient calculation speed, then more sample data can be processed per unit time, but communication overhead between nodes increases, reducing the net processing speed

Engineering Contradiction:

Improvegradient calculation speedVSAvoidcommunication overhead time

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The patent ensures continuous useful action by enabling aggregation communication and dispatch communication to proceed simultaneously without interruption. While the consolidation node is aggregating distributed data from all processing nodes through the first communication port, it is concurrently dispatching the consolidated data back to all nodes through the second communication port. This continuous parallel operation minimizes communication overhead time and ensures that the distributed processing system maintains high gradient calculation speed throughout the entire communication cycle.

Inventive Principle:

Principle #20Continuity of useful action

Data Source

PatentUS11240296B2Distributed processing system and distributed processing method

Publication Date: 2022.02.01 NIPPON TELEGRAPH & TELEPHONE CORP

AI summary

A first distributed processing node transmits distributed data to a second distributed processing node as intermediate consolidated data. A third distributed processing node generates intermediate consolidated data after update from received intermediate consolidated data and distributed data, and transmits the intermediate consolidated data to a fourth distributed processing node. The first distributed processing node transmits the received intermediate consolidated data to fifth distributed processing node as consolidated data. The third distributed processing node transmits the received consolidated data to a sixth distributed processing node. When an aggregation communication time period required by each distributed processing node to consolidate the distributed data or an aggregation dispatch communication time period being a total time period of the aggregation communication time period and a time period required by each distributed processing node to dispatch the consolidated data exceeds a predetermined time period, the first distributed processing node issues a warning.