Streaming Dictionary with Tree-Structured Buckets for Disk I/O Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems face challenges in efficiently maintaining sorted data across disk drives due to performance disparities in accessing data, particularly when data is not localized, leading to suboptimal retrieval times.
Innovation Solution
A high-performance streaming dictionary system is implemented, utilizing a tree-structured data organization with pivot keys and memory hierarchy management to optimize data access and storage, allowing for efficient insertion, lookup, and range scans across non-adjacent levels of memory.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored in sorted order on disk to enable efficient lookup, then data retrieval speed is improved, but data access performance deteriorates when data is not localized due to disk locality constraints
Solution Approach 1:
The patent segments the sorted data into multiple buckets or partitions that are stored on disk. Each bucket contains a subset of the sorted data, allowing the system to access only relevant portions during lookup operations. This segmentation reduces the amount of data that needs to be read from disk while maintaining the sorted order within each bucket, thereby improving both retrieval speed and access performance.
Solution Approach 2:
The patent introduces an additional dimension to the data structure by creating a hierarchical organization with index levels. Instead of storing all sorted data at one level, the system creates multiple levels of indexing where higher levels provide coarse-grained navigation and lower levels provide fine-grained access. This dimensional approach allows efficient navigation to localized data regions on disk, resolving the contradiction between global sorted access and local access performance.
2Speed
If all sorted data is kept in memory for fast access, then data retrieval efficiency is improved, but memory usage increases significantly
Solution Approach 1:
The patent divides the large sorted dataset into smaller segments or buckets that can be individually loaded into memory. Instead of loading the entire dataset, the system loads only the necessary segments based on the query requirements. This segmentation enables efficient memory utilization while maintaining fast access speeds for the active data portion.
Solution Approach 2:
The patent pre-computes and stores index structures on disk that enable rapid identification of data locations. These preliminary index structures allow the system to quickly determine which data segments need to be loaded into memory, reducing the amount of data that must be in memory simultaneously while maintaining efficient retrieval performance.
Data Source
AI summary
A method, apparatus and computer program product for storing data in a disk storage system is presented. A high-performance dictionary data structure is defined. The dictionary data structure is stored on a disk storage system. Key-value pairs can be inserted and deleted into the dictionary data structure. Updates run faster than one insertion per disk-head movement. The structure can also be stored on any system with two or more levels of memory. The dictionary is high performance and supports with full transactional semantics, concurrent access from multiple transactions, and logging and recovery. Keys can be looked up with only a logarithmic number of transfers, even for keys that have been recently inserted or deleted. Queries can be performed on ranges of key-value pairs, including recently inserted or deleted pairs, at a constant fraction of the bandwidth of the disk.