Game-Based Streaming Graph Partitioning for Scalable Memory Management
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current graph partitioning methods, both non-streaming and streaming, face challenges such as high memory usage and limited scalability, with non-streaming methods being unsuitable for large datasets and streaming methods achieving inferior local optimal values due to single-round cost function calculations.
Innovation Solution
A game-based method and system for streaming graph partitioning that uses local information to dynamically adjust edge placements across partition blocks based on a cost function, allowing multiple rounds of strategy updates until convergence, thereby improving load balance and replication efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If non-streaming graph partitioning methods (e.g., METIS) are used to achieve good partitioning results, then partitioning quality is improved, but memory usage increases and scalability deteriorates
Solution Approach 1:
The patent segments the graph processing into multiple rounds where edges are processed in batches rather than loading the entire graph into memory at once. Each round processes a subset of edges and produces partial partitioning results, which are then merged. This segmentation allows the system to handle large graphs that exceed available memory while still achieving good partitioning quality through iterative refinement.
Solution Approach 2:
The patent performs preliminary actions by pre-processing edge lists, computing initial partition assignments, and preparing data structures before the main partitioning algorithm executes. This preliminary preparation optimizes the subsequent processing steps and reduces the memory footprint during the actual partitioning computation, enabling better scalability.
2Adaptability or versatility
If streaming graph partitioning methods are used to improve scalability and support dynamic graphs, then scalability is improved, but partitioning quality deteriorates due to single-round cost function calculations
Solution Approach 1:
The patent implements periodic action by executing the partitioning algorithm in multiple rounds rather than a single pass. Each round recalculates cost functions and refines partition assignments based on updated information from previous rounds. This periodic re-evaluation maintains partitioning quality comparable to non-streaming methods while preserving the scalability benefits of processing edges in manageable batches.
Solution Approach 2:
The patent incorporates feedback mechanisms where partitioning results from each round are fed back into the next round's cost function calculations. The algorithm uses the partition assignments and cost values from previous rounds to inform subsequent partitioning decisions, allowing continuous improvement of partitioning quality while maintaining the streaming processing model's scalability advantages.
Data Source
AI summary
The present invention relates to a game-based method and system for streaming-graph partitioning, the method comprises: partitioning a streaming graph using one or more processors, the one or more processors being configured to: read an edge streaming having a predetermined number of edges in an unpartitioned area of the streaming graph as a sub-graph; based on a first pre-partitioning model, pre-partition the edges of the sub-graph to at least two partition blocks as an initial state of a game process; and sequentially select an optimal partition block for each edge of the sub-graph through the game process until the game process becomes convergent, the disclosed method and system can partition streaming graph using local information only, without loading the whole streaming graph into the memory, thus have good scalability and support dynamic graph partitioning; the disclosed partitioning method and system can provide better partitioning results.