Hierarchical Bipartite Graph Processing for Web Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large bipartite graphs used in web search systems are challenging to process efficiently due to their size and lopsided nature, making it difficult to identify relevant search results and query recommendations in a fast and computationally efficient manner.
Innovation Solution
A distributed, hierarchical, and iterative graph processing method that partitions the graph into subgraphs, generates clusters based on node similarities, encodes these clusters with hashes, and distributes them across multiple processing machines for parallel processing, followed by a hierarchical merge process to recreate the bipartite graph, facilitating efficient data mining and improved query recommendations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If large bipartite graphs are processed using conventional methods, then complete graph analysis can be achieved, but processing time and computational resources increase significantly
Solution Approach 1:
The patent divides the large bipartite graph into multiple partitions, each containing a subset of nodes and edges. These partitions are then processed independently and in parallel across multiple computing machines, significantly reducing the time required to analyze the complete graph while maintaining analysis accuracy through subsequent merging of results.
2Ease of operation
If the entire bipartite graph is loaded into memory for processing, then random access to any node is enabled, but memory requirements become prohibitive
Solution Approach 1:
The graph is partitioned into multiple smaller subgraphs that can be loaded into memory individually. Each partition maintains the necessary node and edge information for its subset, enabling efficient processing within available memory constraints while still allowing access to the complete graph structure through coordinated processing of all partitions.
Solution Approach 2:
The patent introduces a distributed computing dimension by spreading graph processing across multiple machines. Instead of requiring all nodes to reside in a single memory space, the system uses networked memory and coordinated processing to achieve equivalent functionality with distributed memory resources.
3Device complexity
If graph processing is performed on a single machine, then data access is simplified, but processing throughput is limited
Solution Approach 1:
The graph processing workload is divided into independent partition processing tasks that can be executed simultaneously on multiple machines. Each machine processes its assigned partition with locally optimized algorithms, and results are merged to produce the final graph analysis, achieving linear scaling of throughput with the number of processing machines.
Solution Approach 2:
Multiple processing machines work in parallel on different graph partitions, and their individual processing results are merged to produce the complete graph analysis. This combining of parallel processing outcomes achieves high throughput while maintaining the simplicity of single-machine processing logic within each node.
4Measurement precision
If query recommendation systems use detailed graph analysis, then recommendation accuracy improves, but computational complexity increases
Solution Approach 1:
The computational complexity of graph analysis for query recommendations is reduced by processing the graph in partitions. Each partition is analyzed independently for local patterns and relationships, and these local analysis results are combined to derive global recommendation insights, maintaining accuracy while reducing the complexity of any single processing step.
Data Source
AI summary
Systems, methods, and apparatuses for implementing a distributed, hierarchical, and iterative data graph processing strategy are disclosed. A large data graph is partitioned and the partitions are clustered. Sets of keys and hashes are generated for each cluster and distributed to a plurality of processing machines. The processing machines recreate the partitions, based on received hashes and keys of the clusters of the first partitions, and the graph. Thereafter, the partitioning, clustering and processing is repeatable for the recreated graphs. The disclosed strategy enables processing of large and dynamic data graphs, including bipartite query-Uniform Resource Locator graphs.