Distributed Graph Vertex Replication for Latency Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed graph processing systems face inefficiencies due to uneven data distribution and network bottlenecks caused by imbalanced access to graph elements, leading to increased latency and decreased throughput.
Innovation Solution
The technique involves intelligent replication of frequently accessed vertices and tables based on degree information, database schema, and workload statistics, using a degree threshold and capacity limits to determine which data to replicate, and dynamically adjusting replication strategies to optimize storage and access performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Volume of stationary object
If graph data is partitioned and distributed across multiple machines, then storage capacity is increased, but network I/O latency increases due to data transfer between machines
Solution Approach 1:
The patent applies local quality by making frequently accessed vertices (hot vertices) available locally on multiple machines through replication. Instead of uniformly distributing all data, the system identifies vertices with high access frequency based on degree information and replicates them to multiple machine partitions. This ensures that when queries access these hot vertices, the data is already present locally on the requesting machine, eliminating network I/O latency for these critical data elements while maintaining the distributed storage architecture.
2Volume of stationary object
If graph data is evenly partitioned across machines, then storage distribution is balanced, but access performance deteriorates due to imbalanced access patterns
Solution Approach 1:
The patent applies asymmetry by implementing non-uniform data distribution through selective replication of hot vertices. While the base partitioning maintains symmetry for storage balance, the system introduces asymmetry by identifying vertices with degree above a threshold and replicating them to multiple partitions. This creates an asymmetric distribution where frequently accessed vertices have higher availability (present on multiple machines) compared to less frequently accessed vertices. The asymmetric replication strategy aligns data distribution with actual access patterns, ensuring that hot data is optimally positioned for fast access while cold data remains in its original partition.
3Loss of time
If data is replicated to multiple machines, then access latency is reduced, but storage overhead increases
Solution Approach 1:
The patent applies partial action by selectively replicating only a subset of vertices rather than replicating all graph data. The system identifies hot vertices based on degree information (vertices with degree exceeding a threshold) and replicates only these partial elements to multiple machines. This partial replication approach focuses storage resources on the most frequently accessed vertices, accepting that some data (cold vertices) will still require network access. The degree threshold mechanism automatically adjusts the replication scope based on the distribution of vertex degrees, ensuring that replication overhead is applied only where it provides performance benefit.
4Productivity
If hot vertices are replicated to all machines, then access performance is maximized, but storage capacity requirements exceed available resources
Solution Approach 1:
The patent applies parameter changes by dynamically adjusting the replication factor and replication scope based on vertex degree parameters. The system uses a degree threshold parameter to identify hot vertices and replicates only those vertices to multiple machines. The replication factor (how many copies of each hot vertex are created) and the set of vertices subject to replication are both determined by the degree parameter distribution. This parameter-driven approach allows the system to adapt replication intensity to the actual data characteristics, maximizing performance for hot vertices while controlling overall storage overhead by not replicating low-degree vertices.
Data Source
AI summary
In an embodiment, multiple computers cooperate to retrieve content from tables in a relational database. Each table contains respective rows. Each row contains a vertex of a graph. Many high-degree vertices are identified. Each high-degree vertex is connected to respective edges in the graph. A count of the edges of each high-degree vertex exceeds a degree threshold. A central computer detects that all vertices in a high-degree subset of tables are high-degree vertices. Based on detecting the high-degree subset of tables, multiple vertices of the graph that are not in the high-degree subset of tables are replicated. Within local storage capacity limits of the computers, this degree-based replication may be supplemented with other vertex replication strategies that are schema based, content based, or workload based. This intelligent selective replication maximizes system throughput by minimizing graph data access latency based on data locality.