Throughput-Based Fan-Out Control in Distributed Graph Databases
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Complex querying of social and online professional networks is time- and resource-intensive due to exponential link scanning, leading to issues like tail latency, network congestion, and CPU-bounding, which hampers performance and scalability.
Innovation Solution
Implementing a graph database with mechanisms for partitioning, replicating, and providing throughput-based fan-out control to efficiently process queries by distributing data across multiple storage nodes and clusters, allowing for optimal data retrieval and storage without requiring knowledge of a relational model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If complex queries are executed on social network data, then comprehensive search results can be obtained, but query execution time and resource consumption increase exponentially
Solution Approach 1:
The patent segments the social network graph into multiple partitions distributed across different storage nodes. Each partition contains a subset of the graph data, allowing queries to be processed in parallel across multiple nodes rather than scanning the entire graph sequentially. This segmentation reduces query execution time while maintaining complete search results.
Solution Approach 2:
The patent introduces a new dimension of data organization by creating inverted indexes and adjacency lists that provide alternative access paths to the graph data. These auxiliary data structures enable efficient traversal and filtering operations without requiring full graph scans, thus reducing query execution time while preserving query completeness.
2Productivity
If graph data is distributed across multiple storage nodes, then query processing scalability is improved, but system complexity increases
Solution Approach 1:
The patent implements a universal query processing framework that handles multiple types of graph queries (traversal, filtering, aggregation) through a unified architecture. The same partitioning and indexing mechanisms serve multiple query types, reducing the need for separate specialized systems and thereby managing complexity while maintaining scalability.
Solution Approach 2:
The patent introduces intermediary components such as query routers and coordination services that manage the complexity of distributed query processing. These intermediaries handle task distribution, result aggregation, and coordination between storage nodes, shielding application logic from the underlying system complexity while enabling scalable query processing.
3Measurement precision
If fan-out is increased to improve query coverage, then more comprehensive results are achieved, but network congestion and tail latency worsen
Solution Approach 1:
The patent applies partial fan-out by selectively expanding only the necessary portions of the graph based on query filters and constraints. Instead of blindly traversing all outgoing edges from each node, the system evaluates filter conditions at each step and prunes branches that cannot contribute to the final results. This partial action maintains query coverage for relevant data while reducing network traffic and avoiding congestion.
4Speed
If data is replicated across clusters, then query performance and availability are improved, but storage overhead increases
Solution Approach 1:
The patent implements selective replication where different partitions are replicated to different extents based on their access patterns and importance. Frequently accessed partitions or those containing critical data are replicated across more clusters, while less frequently accessed data has lower replication factors. This local quality approach optimizes query retrieval speed for hot data while minimizing overall storage overhead.
Data Source
AI summary
The disclosed embodiments provide a system for processing data. During operation, the system determines a current incoming queries per second (QPS) to one or more components for processing queries of a graph database, wherein the graph database is replicated across multiple clusters and distributed among a set of storage nodes in each of the clusters. Next, the system uses the current incoming QPS to estimate, for the one or more components, an expected QPS associated with fanning out of the queries to the clusters. The system then selects a number of clusters in the multiple clusters for fanning out of a query based on the expected QPS and one or more throughput limits for the one or more components. Finally, the system transmits the query to one or more of the storage nodes in the selected number of clusters.