Graph Relationship Queries Using Parallel Random Walk Precomputation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Determining relationships between nodes in large social graphs, such as those with hundreds of millions or billions of users, is computationally expensive and requires frequent updates due to user interactions, making it difficult to provide accurate recommendations in real-time.
Innovation Solution
A system that pre-determines relationship information using a random walk with restart (RWR) algorithm, distributing database queries across multiple computing devices to parallelize the workload and reduce calculation time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If relationship information is determined in real-time for every node in large social graphs, then recommendation accuracy is improved, but computational cost and processing time become prohibitively expensive
Solution Approach 1:
The system pre-determines relationship information for all nodes in the social graph before actual recommendation requests are made. This preliminary computation stores relationship data in advance, so when a recommendation request arrives, the system can quickly retrieve pre-computed relationship information instead of calculating it in real-time, thus resolving the contradiction between accuracy and processing time
Solution Approach 2:
The computation of relationship information is divided into independent segments that can be processed in parallel. By distributing the calculation across multiple computing devices and processing different portions of the social graph simultaneously, the system reduces overall processing time while maintaining comprehensive relationship analysis accuracy
2Speed
If relationship information is pre-determined for all nodes, then recommendation speed is improved, but the information becomes outdated due to frequent user interactions
Solution Approach 1:
The system periodically updates relationship information at scheduled intervals rather than continuously recalculating after every user interaction. This periodic refresh approach maintains information freshness at acceptable levels while avoiding the computational overhead of real-time updates, thus balancing recommendation speed with information reliability
Solution Approach 2:
The system dynamically adjusts the update frequency and triggers based on system state, user activity levels, and resource availability. When user interactions are particularly active or when resources are available, updates occur more frequently; during quieter periods, update frequency decreases, optimizing the balance between information freshness and computational efficiency
3Device complexity
If the RWR algorithm is executed on a single computing device, then implementation simplicity is maintained, but the processing time for large social graphs becomes unacceptably long
Solution Approach 1:
The social graph is partitioned into multiple segments or shards that can be processed independently in parallel across different computing devices. Each device handles a specific portion of the graph, performing RWR calculations on its assigned segment, which dramatically increases overall processing throughput while keeping individual device implementations relatively simple
Solution Approach 2:
A coordinator component acts as an intermediary that manages the distributed computation across multiple devices. It distributes work segments to available computing devices, aggregates results from all devices, and maintains the overall system state, enabling parallel processing without requiring complex peer-to-peer communication between all computing devices
Data Source
AI summary
One example method includes generating, from a relevancy graph, a first table and a second table, the relevancy graph comprising a plurality of nodes and a plurality of edges, wherein each node represents an individual and each edge connects two nodes and represents a relationship between the respective two nodes, the first table comprising information about each node in the relevancy graph and the second table comprising information about each edge in the relevancy graph; selecting a seed node from the plurality of nodes; generating a database query to obtain data for a relevancy table based on a set of neighbor nodes to the seed node and a set of corresponding edges connecting the seed node to each neighbor node of the set of neighbor nodes; distributing, to a plurality of computing nodes, portions of the database query to determine, in parallel, probability information for the relevancy table; generating the relevancy table comprising the seed node, the set of neighbor nodes, and, for the seed node and each neighbor node, a corresponding probability based on the probability information; receiving, from a remote computing device, a request related to an individual and a software service offered by a service provider; and determining and providing, to the remote computing device based on the relevancy table, information about a relationship between the individual and one or more individuals represented in the relevancy graph/