Virtual Node Compression for Web Graph Runtime Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Many web graph algorithms cannot run directly on compressed graphs, requiring decompression which adds a computational burden and is sub-optimal.
Innovation Solution
Extending web graph algorithms to operate directly on virtual node-type compressed web graphs using Black-box matrix computation and adjusting transition probabilities, allowing for faster eigenvector calculations and reduced runtime.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If web graph algorithms are run on uncompressed graphs, then algorithm accuracy is maintained, but runtime is excessive and computational resources are wasted
Solution Approach 1:
The patent segments the web graph into compressed and uncompressed portions. The compressed portion uses virtual nodes to represent dense subgraphs, while the uncompressed portion maintains original structure. This segmentation allows algorithms to run efficiently on the compressed portion while preserving accuracy requirements for the uncompressed portion.
Solution Approach 2:
The patent introduces virtual nodes as intermediaries between the compressed graph representation and the original graph structure. These virtual nodes act as mediators that allow standard web graph algorithms to operate on compressed data without requiring full decompression, thereby reducing runtime while maintaining algorithmic correctness.
2Quantity of substance
If compressed graphs are used, then memory and storage requirements are reduced, but standard web graph algorithms cannot run directly
Solution Approach 1:
The patent creates a universal graph representation that works with both compressed and uncompressed formats. By introducing virtual nodes that can represent either compressed subgraphs or original nodes, the system achieves multi-functionality where the same data structure supports both space-efficient storage and algorithm compatibility.
Solution Approach 2:
Virtual nodes serve as intermediaries that enable standard web graph algorithms to operate on compressed graphs. These intermediaries translate between the compressed representation and the algorithm's expectations, allowing algorithms like PageRank and HITS to run directly on compressed data without modification.
3Adaptability or versatility
If decompression is performed before algorithm execution, then algorithm compatibility is maintained, but additional computational burden is introduced
Solution Approach 1:
The patent performs preliminary compression of the web graph into a virtual node representation before algorithm execution. This preliminary action allows subsequent algorithm runs to operate directly on the compressed structure without requiring intermediate decompression, thereby eliminating the computational burden of repeated decompression operations.
Solution Approach 2:
Virtual nodes act as intermediaries that eliminate the need for decompression. Instead of converting compressed data back to full format, the virtual nodes allow algorithms to operate directly on the compressed representation, maintaining compatibility while improving efficiency.
Data Source
AI summary
Classes of web graph algorithms are extended to run directly on virtual node-type compressed web graphs where a reduction in runtime of the extended algorithms is realized which is approximately proportional to the compression ratio applied to the original (i.e., uncompressed) graph. In the virtual node compression technique, a succinct representation of a web graph is constructed by replacing dense subgraphs by sparse ones so that the resulting compressed graph has significantly fewer edges and a relatively small number of additional nodes.