Out-of-Core BFS With Disk-Spilling Queues for Large Graph Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing breadth first search (BFS) algorithms for shortest path queries are limited by memory constraints, particularly in memory-constrained systems, which restrict the size of datasets that can be processed, and there is a need for efficient solutions that utilize external storage to handle larger datasets.
Innovation Solution
The implementation of a disk-spilling hash-table (DSH) as the visited set and disk-spilling queues (DSQs) as the BFS frontier queue, utilizing out-of-core external storage such as hard drives, to manage memory resources efficiently and handle large datasets in memory-constrained systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional in-memory BFS algorithms are used, then algorithm simplicity and speed are maintained, but memory consumption limits the size of processable datasets
Solution Approach 1:
The patent segments the BFS data structures into multiple partitions that can be independently managed. The visited set and frontier queue are divided into partitions that can be loaded from external storage into memory in manageable chunks, allowing processing of datasets larger than available memory while maintaining algorithmic simplicity through partitioned data structures.
Solution Approach 2:
The patent transitions from a single-memory dimension to a multi-level storage dimension by introducing external storage (disk or network-attached storage) as an additional layer. This dimensional expansion allows the system to handle datasets exceeding memory capacity by leveraging the hierarchical storage architecture, effectively moving the bottleneck from memory size to storage size.
2Quantity of substance
If external storage is used to handle large datasets, then dataset size capability is improved, but access speed and performance may deteriorate
Solution Approach 1:
The patent implements preliminary loading of data partitions from external storage into memory before processing begins. By pre-loading necessary partitions and maintaining them in memory during the BFS execution, the system minimizes repeated external storage accesses during the algorithm execution, thereby reducing the performance penalty of using external storage.
Solution Approach 2:
The patent dynamically manages the boundary between memory and external storage by selectively loading and unloading data partitions based on memory availability and processing needs. This dynamic adjustment allows the system to optimize between memory speed and storage capacity, loading only necessary partitions into memory while keeping the rest on external storage.
3Quantity of substance
If memory resources are constrained, then hardware costs are reduced, but the size of processable datasets is limited
Solution Approach 1:
The patent introduces external storage as an intermediary between the limited memory resources and the large datasets. This intermediary layer allows the system to overcome the memory capacity bottleneck without requiring proportional increases in hardware cost, as external storage provides a cost-effective extension for data retention during processing.
Solution Approach 2:
The patent changes the fundamental parameter of data storage location from exclusively in-memory to a combination of in-memory and external storage. This parameter change allows the system to maintain processing capability with constrained memory by leveraging external storage for data retention, effectively decoupling dataset size from memory capacity requirements.
4Quantity of substance
If distributed computation is used to process large datasets, then dataset size capability is improved, but system complexity and overhead increase
Solution Approach 1:
The patent merges the concepts of in-memory processing and external storage access into a unified BFS algorithm implementation. By combining these approaches through partitioned data structures that can operate seamlessly across both memory and external storage, the system achieves large dataset processing without requiring distributed computation across multiple machines, thereby reducing system overhead.
Data Source
AI summary
A breadth first search (BFS) algorithm is provided that uses out-of-core external storage in a memory constrained system. Memory resources are used as long as they are available and external storage is used when necessary due to memory pressure. The BFS algorithm uses a disk-spilling hash-table (DSH) as the visited set and disk-spilling queues (DSQs) as the BFS frontier queue. To get the most out of the DSH, subsequent inserts and lookups must happen in the same DSH partition. To ensure that consecutive lookups happen in the same DSH partition, the BFS frontier queue is partitioned in a manner similar to the DSH partitions.