Block-Based Index for Graph Traversal Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Graph traversal in column-oriented storage systems is inefficient due to repetitive full column scans, especially for sparse graph topologies or vertices with small neighborhoods, leading to high computational expenses.
Innovation Solution
A block-based index is generated and maintained in the graph column, dividing it into non-overlapping blocks with associated value codes and block range vectors, allowing for efficient look-up and traversal operations by constraining the scan range to a fraction of the column.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If full column scans are performed for graph traversal, then completeness of data retrieval is ensured, but computational efficiency deteriorates
Solution Approach 1:
The column is divided into multiple non-overlapping blocks, each representing a segment of the data. This segmentation allows the system to traverse only relevant blocks instead of scanning the entire column, thus maintaining data completeness while improving computational efficiency.
Solution Approach 2:
Block range vectors are pre-computed and stored for each block, containing information about the range of values present in that block. This preliminary action enables quick determination of which blocks need to be scanned during graph traversal, avoiding full column scans and improving efficiency.
2Speed
If block-based indexing is implemented, then traversal speed is improved, but system complexity increases
Solution Approach 1:
The block range vectors serve multiple functions: they enable quick block identification during traversal, support sparsity detection, and facilitate health factor computation. This multi-functionality reduces the need for separate data structures, thereby limiting the increase in system complexity while maintaining traversal speed improvements.
Solution Approach 2:
The indexing system is designed to be self-maintaining through health factor computation. The system automatically identifies and re-partitions blocks that become inefficient, reducing the need for external intervention and complex maintenance mechanisms, thus balancing speed improvement with manageable system complexity.
3Reliability
If dynamic block management is performed, then index health is maintained, but computational overhead increases
Solution Approach 1:
Health factor computation is performed periodically or on-demand rather than continuously. This periodic action maintains index health by identifying blocks that need re-partitioning while avoiding unnecessary computational overhead from constant monitoring and adjustment of all blocks.
Solution Approach 2:
The system changes the granularity parameter of blocks dynamically based on health factor computation. When blocks are found to be inefficient (high health factor), they are re-partitioned into smaller blocks. This parameter change optimizes the index structure adaptively while controlling computational overhead by only modifying blocks that need improvement.
Data Source
AI summary
A system, computer-implemented method, and computer-readable storage medium for generating a block-based index, are provided. A block index is generated where the block index comprises a plurality of blocks and a block corresponds to a section of a graph column that stores a value. A block range vector is also generated for the index where the block range vector includes range information for the block that corresponds to the section of the graph and where the block-based index facilitates traversal of the graph column that searches for the value by constraining the traversal to the section of the graph.