Property Graph Label Encoding for Memory-Efficient Distributed Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing graph processing systems face inefficiencies in storing and managing label information in property graphs, particularly due to memory-intensive string representations and suboptimal compression techniques that fail to scale with the number of labels.
Innovation Solution
A 2-tier dictionary encoding method is introduced, where a tier 1 dictionary maps individual labels to integer codes and a tier 2 dictionary maps label sets to integer codes, allowing labels to be represented as compressed label sets using a sequence of bits, reducing memory consumption and enabling efficient integer arithmetic operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If labels are stored as per-entity strings, then label information is easily accessible, but memory consumption increases significantly
Solution Approach 1:
The patent creates a dictionary mapping each unique label string to a compact integer code. Instead of storing repetitive label strings for each entity, the system stores integer codes that reference the dictionary. This copying approach replaces large string representations with compact integer equivalents, dramatically reducing memory usage while maintaining label accessibility through the dictionary mapping.
Solution Approach 2:
The patent transforms the representation parameter of labels from string type to integer code type. By changing the data type parameter from high-memory-consuming strings to low-memory-consuming integers, the system achieves significant memory savings. The dictionary provides the translation layer that maintains label semantics while using efficient integer parameters for storage.
2Quantity of substance
If dictionary encoding is used for labels, then memory usage is reduced, but string comparisons are replaced with integer operations
Solution Approach 1:
The patent performs preliminary encoding by building a complete dictionary mapping all label strings to integer codes before executing graph queries. This preliminary action consolidates the encoding complexity into a one-time setup phase, allowing the query execution phase to use simple integer comparisons. The dictionary construction is done once, and then the system benefits from efficient integer-based operations for all subsequent label comparisons.
3Quantity of substance
If standard compression techniques like Huffman compression are used, then individual strings are compressed efficiently, but sets of strings are not stored efficiently
Solution Approach 1:
The patent merges the compression of individual label strings into a unified dictionary-based integer coding system. Instead of compressing each label string independently using Huffman compression, the system merges all unique labels into a single dictionary and represents them with a unified integer code scheme. This merging approach enables efficient storage of sets of labels as collections of integers, which can be further compressed using bitmaps or other set-oriented compression techniques.
Solution Approach 2:
The patent changes the storage parameter from variable-length compressed strings to fixed-width integer codes. By standardizing the representation parameter to fixed-width integers (e.g., 4-byte integers), the system enables efficient set operations and indexing. This parameter change allows for the use of integer arrays, bitmaps, and other efficient data structures for storing and querying sets of labels, improving overall storage efficiency for label sets.
4Quantity of substance
If bitmap encoding is used for labels, then small number of labels are stored efficiently, but the approach does not scale as number of labels grows
Solution Approach 1:
The patent implements a dynamic encoding strategy that adapts to the number of labels. For entities with a small number of labels, the system uses compact representations such as sorted integer arrays or bitmaps when appropriate. For entities with larger label sets or when the total number of unique labels exceeds bitmap capacity, the system dynamically switches to more scalable representations like integer arrays with dictionary encoding. This dynamic adaptation allows the system to optimize storage for each case while maintaining scalability.
Solution Approach 2:
The patent segments the label storage approach into multiple representation strategies based on the characteristics of the label sets. Instead of using a single bitmap approach for all cases, the system segments the solution into: (1) bitmap encoding for small, fixed-size label sets with limited cardinality, (2) sorted integer arrays for medium-sized label sets, and (3) dictionary-encoded integer representations for large-scale scenarios. This segmentation allows each approach to be used where it is most effective, providing both efficiency for small sets and scalability for large sets.
Data Source
AI summary
Techniques are described herein for space-efficient encoding of label information of property graphs. In an embodiment, an input graph is received. The input graph comprises a plurality of entities and a plurality of label sets. Each entity of said plurality of entities is associated with a label set of the plurality of label sets and each label set of the plurality of label sets comprises zero or more labels of a plurality of labels. A first mapping is generated that maps each label of the plurality of labels to a label code. A second mapping is generated that maps each label integer set of a plurality of label integer sets to a label code. Each label integer set of the plurality of label integer sets corresponds to a label set of the plurality of label sets, wherein each label integer set of the plurality of label integer sets comprises label codes from the first mapping that are mapped to each label included in the corresponding label set. A compressed label set is generated for each entity of the plurality of entities. Each compressed label set comprises a plurality of bits that indicate a zeroth state, a first state, a second state, or a third state. The compressed label sets and the first and second mappings are used to efficiently evaluate graph label queries.