Multi-stage Pipelining for Distributed Graph Processing Latency

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing distributed graph processing frameworks, such as Giraph on Map-Reduce and GraphFrame, rely on the bulk-synchronous execution model, which results in significant latency for workloads with small computation and communication requirements, particularly in interactive graph exploration and vertex/edge property access tasks.

Innovation Solution

A multi-stage pipelining communication mechanism is introduced, enabling asynchronous communication between graph processing machines through user-defined lambda functions and communication agents, allowing for low-latency processing of fixed-depth graph exploration tasks and vertex property access without the need for global synchronization barriers.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If bulk-synchronous execution model is used, then parallel computation is maximized and batched communication is enabled, but latency increases significantly for tasks with small computation and communication requirements

Engineering Contradiction:

Improveparallel computation efficiencyVSAvoidoperation latency

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The execution model is segmented into two distinct modes: bulk-synchronous mode for parallel computation workloads and asynchronous pipelining mode for low-latency workloads. This segmentation allows each mode to be optimized independently, resolving the contradiction between maximizing parallel computation efficiency and minimizing operation latency.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system dynamically switches between bulk-synchronous execution and asynchronous pipelining based on the specific workload characteristics. The execution engine adapts its behavior in real-time, selecting the appropriate mode to optimize performance for each task, thereby resolving the trade-off between parallel efficiency and low latency.

Inventive Principle:

Principle #15Dynamics

2Reliability

If bulk-synchronous execution model is used, then global synchronization is ensured between steps, but computational efficiency decreases for tasks requiring quick vertex/edge property access and fixed-depth graph exploration

Engineering Contradiction:

Improveglobal synchronizationVSAvoidcomputational efficiency

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system changes the execution parameters dynamically based on workload type. For low-latency tasks, the system switches to asynchronous pipelining mode with different synchronization parameters, allowing faster execution without global synchronization barriers, thereby improving computational efficiency while maintaining reliability through targeted synchronization only where needed.

Inventive Principle:

Principle #35Parameter changes

3Loss of time

If asynchronous pipelining is introduced, then latency is reduced for low-latency tasks, but system complexity increases due to user-defined lambda functions and communication agents

Engineering Contradiction:

Improveoperation latencyVSAvoidcommunication mechanism complexity

Core Design Contradiction:

Loss of timeVSDevice complexity

Solution Approach 1:

Communication agents are introduced as intermediary components that manage the asynchronous communication between pipeline stages. These agents encapsulate the complexity of asynchronous message passing, allowing users to define simple lambda functions without directly managing communication details, thus reducing the effective complexity while enabling low-latency operation.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The system provides self-service through automatically generated communication agents and pipeline management. The execution engine automatically handles message routing, buffering, and synchronization between pipeline stages, freeing users from manual communication management and reducing the perceived system complexity while maintaining low-latency performance.

Inventive Principle:

Principle #25Self-service

Data Source

PatentUS11363093B2Multi-stage pipelining for distributed graph processing

Publication Date: 2022.06.14 ORACLE INT CORP

AI summary

Techniques are described herein for evaluating graph processing tasks using a multi-stage pipelining communication mechanism. In a multi-node system comprising a plurality of nodes, each node of said plurality of nodes executes a respective communication agent object. The respective communication agent object comprises: a sender lambda function is configured to perform sending operations and generate source messages based on the sender operations. An intermediate lambda function is configured to read source messages marked for a node, perform intermediate operations based on the source messages and generate intermediate messages based on the intermediate operations. A final receiver lambda function configured to: read intermediate messages marked for said each node, perform final operations based on the intermediate messages and generate a final result based on the final operations.