Distributed Graph Database Edge Insertion Throughput

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Distributed graph databases face challenges in efficiently managing unique edge identifiers across disparate machines, leading to increased complexity and reduced data throughput and integrity due to the need for sequential updating.

Innovation Solution

The implementation of asynchronous communication and a firehose mechanism for edge addition, along with a query manager for parallel processing of graph queries, reduces the number of messages required for edge addition and enhances query throughput and latency by employing asynchronous communication, batched edge addition, and concurrent query execution.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If sequential updating is used to manage unique edge identifiers across distributed machines, then data integrity is maintained, but data throughput is reduced and system complexity increases

Engineering Contradiction:

Improvedata integrityVSAvoiddata throughput

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The system segments the distributed graph database into multiple independent machines or nodes, each capable of autonomously generating and managing its own unique edge identifiers. This segmentation eliminates the need for centralized sequential updating, allowing parallel identifier generation across all nodes while maintaining global uniqueness through coordinated algorithms, thus improving data throughput without sacrificing integrity

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system performs preliminary action by pre-generating batches of unique edge identifiers and caching them locally at each machine before they are needed. This allows machines to insert edges without real-time network communication for identifier allocation, significantly increasing data throughput while ensuring uniqueness through the pre-established identifier pools

Inventive Principle:

Principle #10Preliminary action

2Reliability

If synchronous communication is used for edge addition across distributed machines, then data consistency is ensured, but latency increases and throughput decreases

Engineering Contradiction:

Improvedata consistencyVSAvoidedge addition throughput

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The system dynamically switches between synchronous and asynchronous communication modes based on the specific operation. For edge additions, it uses asynchronous communication where machines can independently proceed without waiting for confirmation from other nodes, improving throughput. For operations requiring consistency, it employs synchronous communication, thus optimizing performance without sacrificing data consistency when needed

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The system introduces an intermediary mechanism in the form of a distributed coordinate system or version vector that indirectly coordinates identifier generation across machines without requiring direct synchronous communication. Each machine uses this intermediary to generate unique identifiers independently, eliminating communication latency while ensuring consistency through the intermediary's coordination protocol

Inventive Principle:

Principle #24Intermediary (Mediator)

3Reliability

If centralized identifier management is used in distributed graph databases, then uniqueness is guaranteed, but system complexity and communication overhead increase

Engineering Contradiction:

Improveidentifier uniquenessVSAvoidsystem complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The system segments the centralized identifier management function into distributed components at each node. Each machine runs its own identifier generation algorithm locally, eliminating the need for a centralized management system. This reduces system complexity and communication overhead while maintaining uniqueness through distributed coordination mechanisms

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Each machine in the distributed system provides self-service by autonomously generating its own unique edge identifiers using local algorithms and cached pools. This eliminates dependency on centralized identifier management, reducing system complexity and communication overhead while ensuring uniqueness through locally-executed coordination protocols

Inventive Principle:

Principle #25Self-service

4Productivity

If asynchronous communication is used for edge addition, then throughput is improved and latency is reduced, but message coordination complexity increases

Engineering Contradiction:

Improveedge addition throughputVSAvoidmessage coordination complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The system performs preliminary action by pre-establishing communication channels and protocols for asynchronous message passing before edge addition operations begin. This includes pre-defining message formats, routing rules, and confirmation mechanisms, which simplifies the coordination complexity during actual asynchronous operations while maintaining high throughput and low latency

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS11372919B2Distributed graph databases that facilitate streaming data insertion and queries by efficient throughput edge addition

Publication Date: 2022.06.28 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

A novel distributed graph database is provided that is designed for efficient graph data storage and processing on modern computing architectures. In particular a single node graph database and a runtime & communication layer allows for composing a distributed graph database from multiple single node instances.