Guide Word Paged Hash Table for Memory-Constrained Data Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
B-trees face inefficiencies due to high memory requirements and excessive disk access, particularly during operations like backups, where accessing disk data is time-consuming due to mechanical components, leading to a need for a more efficient data structure with reduced memory and disk access needs.
Innovation Solution
The guide word paged hash table data structure, which includes an in-memory portion with a tree or array of pages and an on-disk portion, uses hashing to store namespace data efficiently, leveraging locality of access by storing only hash values and employing guide words to quickly determine name ranges, thereby minimizing memory usage and disk accesses.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a b-tree is used to store and retrieve records, then data can be organized and accessed systematically, but memory requirements increase and disk access frequency increases
Solution Approach 1:
The patent extracts only the essential indexing information (guide words representing minimum and maximum name values) into a compact in-memory structure, while leaving the bulk data on disk. This selective extraction reduces memory requirements while maintaining reliable data access through the guide word index that directs queries to specific page ranges.
Solution Approach 2:
The patent segments the data structure into distinct components: a small in-memory guide word index and larger on-disk page structures. This segmentation allows the critical access path (guide words) to reside in fast memory while the bulk data remains on disk, resolving the contradiction between reliable access and memory consumption.
2Reliability
If a b-tree is used to store and retrieve records, then data can be organized and accessed systematically, but disk access frequency increases
Solution Approach 1:
The patent performs preliminary organization of data into pages with guide words that define name value ranges. This pre-organization allows queries to quickly determine which pages contain the desired data, eliminating the need to traverse entire b-tree structures and reducing disk access time while maintaining reliable data retrieval.
Solution Approach 2:
The patent introduces a new dimensional approach by organizing data in fixed-size pages with guide words that define ranges, rather than traditional hierarchical b-tree nodes. This dimensional change from hierarchical traversal to range-based page addressing reduces disk access frequency while preserving reliable data access.
3Adaptability or versatility
If the top level of a b-tree exceeds available memory, then all nodes can be stored on disk, but performance deteriorates due to excessive disk access
Solution Approach 1:
The patent changes the structural parameters of the data organization from traditional b-tree nodes to fixed-size pages with guide words. This parameter change enables the index to fit in memory even when the full dataset exceeds memory capacity, maintaining high data access speed by keeping the critical guide word index in fast memory while accommodating large datasets on disk.
Data Source
AI summary
A set of pages including a set of entries are stored on disk. A set of page headers are stored in memory. Each page header corresponds to a page on disk. A page header includes a page number identifying a corresponding page on disk, a first guide word, and a second guide word. The first and second guide words define minimum and maximum name values, respectively, of entries on the page. Upon a request to determine if an entry associated with a name value exists, the page headers are scanned to identify a page number in a header where the name value is between the guide words. The page, if not resident in memory, is located on disk, loaded into memory, and examined to determine whether the entry exists on the page.