Streaming Graph Partitioning for GPU Memory-Limited Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing GPU-based graph partitioning methods face memory limitations and inefficiencies when processing large graphs, leading to difficulties in loading and processing data due to limited GPU memory capacity and slow data transfer rates.
Innovation Solution
A streaming clustering-based method that compresses graph data through clustering, utilizing CPU and GPU resources to efficiently partition large graphs by minimizing memory usage and optimizing data transmission, involving input, clustering, initial partitioning, and optimization operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If large graph data is loaded into GPU memory for processing, then processing speed and parallel computing capability are improved, but GPU memory capacity is exceeded causing out of memory errors
Solution Approach 1:
The patent applies segmentation by dividing the large graph data into multiple partitions that can be processed separately. The graph is split into smaller manageable chunks that fit within GPU memory constraints, allowing parallel processing of multiple partitions across different GPU streams or time periods, thus resolving the contradiction between processing speed and memory capacity
Solution Approach 2:
The patent implements nested doll by creating a hierarchical data structure where cluster-level data structures contain vertex-level data structures. This nesting allows the system to work with compressed representations of graph data at different levels of granularity, enabling large graphs to be processed within limited GPU memory by only loading necessary portions at each processing stage
2Quantity of substance
If graph data is compressed through clustering to fit GPU memory, then memory usage is reduced, but data transfer time and processing overhead increase
Solution Approach 1:
The patent applies preliminary action by performing clustering and data compression operations on the host CPU before transferring data to the GPU. The graph data is pre-processed, clustered, and organized into compressed formats on the host system, so that when data is transferred to the GPU, it is already in an optimized state that minimizes transfer time and maximizes memory efficiency during actual processing
Solution Approach 2:
The patent implements continuity of useful action by maintaining active GPU processing streams that continuously process multiple partitions of graph data. While one partition is being processed, the system prepares the next partition on the host, ensuring that the GPU remains continuously utilized without idle waiting time, thus balancing compression overhead with sustained processing throughput
3Quantity of substance
If streaming clustering is applied to compress graph data, then memory efficiency is improved, but algorithm complexity and computational overhead increase
Solution Approach 1:
The patent uses intermediary by introducing a host CPU as a mediator between the input graph data and the GPU processing system. The host CPU performs the complex streaming clustering algorithm and data compression operations, acting as an intermediary that prepares optimized data structures for GPU consumption. This separates the algorithmic complexity from the GPU execution, allowing the GPU to focus on parallel processing while the host handles the computationally intensive clustering logic
Data Source
AI summary
Provided is a large graph partitioning method using streaming clustering in a GPU environment. A CPU gradually compresses graph data by considering a structural environment of a GPU and then transfers the compressed graph data to a GPU memory to perform graph partitioning, thereby utilizing the memory of the GPU most efficiently and performing fast graph processing. As a result, the large graph partitioning method using streaming clustering in a GPU environment relates to a technology that can effectively process connection information for all clusters in parallel by utilizing the high computational amount of the GPU and overcome the memory limitations of the GPU by processing computations that require a large memory space in the CPU.