Distributed Graph Database Edge Insertion via Asynchronous Communication

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Distributed graph databases face challenges in efficiently managing unique edge identifiers across disparate machines, leading to increased complexity and reduced data throughput and integrity due to the need for sequential updating.

Innovation Solution

The implementation of asynchronous communication and a firehose mechanism for edge addition, along with a query manager for parallel processing of graph queries, reduces the number of messages required for edge addition and enhances query throughput and latency by employing asynchronous communication and batched edge addition, and parallel query execution.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If synchronous communication is used for edge addition in distributed graph databases, then data consistency is maintained, but the number of messages increases and data throughput decreases

Engineering Contradiction:

Improvedata consistencyVSAvoiddata throughput

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The system pre-generates unique edge identifiers (edge IDs) and stores them in a dedicated edge ID generator component before they are needed for edge addition. This preliminary action eliminates the need for multiple message exchanges during edge addition, as the edge ID is already available locally rather than requiring sequential requests and responses across distributed machines.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system separates the edge ID generation function from the edge addition operation by introducing a dedicated edge ID generator component. This segmentation allows edge IDs to be pre-generated and cached independently, enabling the edge addition process to proceed with local operations rather than requiring coordinated communication between distributed nodes.

Inventive Principle:

Principle #1Segmentation

2Reliability

If sequential updating is used for edge addition, then data integrity is maintained, but the number of messages required increases

Engineering Contradiction:

Improvedata integrityVSAvoidnumber of messages

Core Design Contradiction:

ReliabilityVSQuantity of substance

Solution Approach 1:

The system pre-generates and caches unique edge identifiers before they are needed for edge addition operations. This preliminary generation of edge IDs eliminates the need for sequential message exchanges to obtain unique identifiers, reducing the total number of messages while maintaining data integrity through the use of pre-validated unique IDs.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

Each distributed machine is equipped with local access to the edge ID generator or local cache of edge IDs, enabling it to self-serve unique identifier requirements without requiring messages to other machines. This self-service capability reduces inter-machine communication while maintaining the uniqueness and integrity of edge identifiers.

Inventive Principle:

Principle #25Self-service

3Productivity

If parallel processing is used for query execution, then query throughput increases, but system complexity increases

Engineering Contradiction:

Improvequery throughputVSAvoidsystem complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The system divides the graph database into multiple shards distributed across different machines, with each shard independently handling a portion of the data. This segmentation enables parallel query execution across shards while maintaining manageable complexity at each individual node, as each shard processes queries independently without requiring complex coordination.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system combines results from multiple parallel shard queries into a unified result set. This merging approach allows the system to maintain simple, independent processing at each shard while achieving high overall throughput through parallel execution, with the query coordinator handling result aggregation rather than complex distributed transactions.

Inventive Principle:

Principle #5Merging (Combining)

Data Source

PatentUS11321393B2Distributed graph databases that facilitate streaming data insertion and queries by reducing number of messages required to add a new edge by employing asynchronous communication

Publication Date: 2022.05.03 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

A novel distributed graph database is provided that is designed for efficient graph data storage and processing on modern computing architectures. In particular a single node graph database and a runtime & communication layer allows for composing a distributed graph database from multiple single node instances.