Distributed Graph Difference Encoding Without Remote Decoding Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing graph compression techniques based on difference encoding for graph analytics systems are suboptimal in distributed systems due to costly remote communication and potential incorrect decoding when graph datasets are distributed across multiple nodes.
Innovation Solution
A distributed-system-aware difference encoding scheme that encodes graph data without the need for remote accesses by considering the distribution of vertices and edges among multiple computing nodes, using variable-length codes to optimize compression ratio and decoding time, and allowing correct decoding even when the number of nodes is unknown.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If difference encoding is applied to graph data in distributed systems, then compression ratio is improved, but decoding correctness deteriorates due to node distribution issues
Solution Approach 1:
The patent segments the graph data encoding process by node index ranges. Each node is assigned a specific index range, and difference encoding is applied independently within each segment. This segmentation allows each node to decode its own data correctly without requiring information from other nodes, thus maintaining decoding correctness while achieving compression.
Solution Approach 2:
The patent applies local quality by making the encoding scheme adaptive to local node characteristics. Each node uses its own index range and local reference values for difference encoding, rather than a global reference system. This local approach ensures that each node can independently and correctly decode its portion of the graph data.
2Device complexity
If traditional difference encoding is used in distributed systems, then implementation simplicity is improved, but communication cost worsens due to remote accesses
Solution Approach 1:
The patent divides the graph data into segments assigned to different nodes based on index ranges. Each node only needs to access and process its local segment, eliminating the need for remote accesses to other nodes' data. This segmentation maintains encoding simplicity while removing communication overhead.
Solution Approach 2:
Each node performs self-service by independently encoding and decoding its own graph data segment using local reference values. Nodes do not need to communicate with or access data from other nodes, making the system scalable and eliminating inter-node communication costs for the decoding process.
3Adaptability or versatility
If graph data is distributed across multiple nodes, then system scalability is improved, but decoding performance deteriorates due to remote communication requirements
Solution Approach 1:
The patent segments graph data by node index ranges and distributes segments across multiple nodes. Each node decodes its local segment independently using only local data, eliminating remote communication requirements. This segmentation enables the system to scale to multiple nodes while maintaining high decoding performance through parallel independent processing.
Solution Approach 2:
The patent performs preliminary organization of graph data by node index ranges before distribution. This preliminary action ensures that each node receives a self-contained segment with all necessary information for independent decoding, eliminating the need for runtime communication and enabling immediate parallel processing upon data arrival.
Data Source
AI summary
A portion of a graph dataset is generated for each computing node in a distributed computing system by, for each subject vertex in a graph, recording for the computing node an offset for the subject vertex, where the offset references a first position in an edge array for the computing node, and for each edge of a set of edges coupled with the subject vertex in the graph, calculating an edge value for the edge based on a connected vertex identifier identifying a vertex coupled with the subject vertex via the edge. When the edge value is assigned to the first position, the edge value is determined by a first calculation, and when the edge value is assigned to position subsequent to the first position, the edge value is determined by a second calculation. In the computing node, the edge value is recorded in the edge array.