Block-Based PageRank Computation for Memory-Limited Large Graphs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing size of graphs representing large datasets makes it difficult to load and calculate relative importance between vertices efficiently, requiring significant resources and time.
Innovation Solution
A method and apparatus that divide graph data into blocks, calculate weights of destination vertices using pagerank and out-degree in units of blocks, and update pagerank by accumulating weights, utilizing graphics processing units (GPUs) for parallel processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If graph data is loaded into memory for calculation, then calculation can be performed, but memory capacity is insufficient for large graphs with trillions of edges
Solution Approach 1:
The patent divides the large graph data into multiple blocks that can be processed separately. Each block contains a subset of vertices and edges, allowing the graph to be handled in manageable portions that fit within available memory capacity while still processing the entire graph over time.
Solution Approach 2:
The patent introduces a block index dimension to organize graph data, enabling efficient access and processing of specific graph portions without loading the entire graph into memory. This dimensional organization allows selective loading of required blocks based on processing needs.
2Reliability
If entire graph data is processed at once, then complete calculation is achieved, but processing time and resource consumption increase significantly
Solution Approach 1:
The patent segments the graph processing into block-level operations where weights are calculated for each block independently and then accumulated. This segmentation enables parallel processing of multiple blocks simultaneously, reducing overall processing time while maintaining complete graph analysis through systematic accumulation of block results.
Solution Approach 2:
The patent performs preliminary calculations of vertex weights within each block before accumulating them across blocks. This preliminary action within smaller units allows for optimized computation that can be parallelized, reducing the total time required compared to processing the entire graph as a single unit.
3Device complexity
If graph calculation is performed on a single machine, then system complexity is reduced, but processing capability is insufficient for large-scale graphs
Solution Approach 1:
The patent enables single-machine processing of large graphs by segmenting the computation into block-level operations that can be executed sequentially or with limited parallelism using available hardware resources. This segmentation allows efficient utilization of single-machine capabilities without requiring distributed system complexity.
Solution Approach 2:
The patent changes the processing parameters by working with block-sized subsets of graph data rather than the entire graph at once. This parameter change allows the single machine to handle larger graphs by adjusting the scope of each processing operation to match available computational resources while maintaining overall processing capability.
Data Source
AI summary
Provided are a graph data calculation method and apparatus. A computing device divides graph data including a source vertex and a destination vertex of an edge into a plurality of blocks, and identifies an out-degree of each vertex. Next, the computing device repeatedly performs a process of calculating, in units of blocks, a weight of a destination vertex of each edge by using a pagerank and an out-degree of a source vertex of each edge existing in a block, and updating a pagerank of each vertex of the graph data by accumulating a weight of a destination vertex of each block.