SSD Indexing via Slicing and Buffer Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current SSD index designs lack flexibility and generality, leading to poor performance under varying workloads and incompatible data structures, which limits their applicability and scalability in data-intensive systems.
Innovation Solution
The proposed solution involves creating buffer indexes that cluster related hash type index data for efficient writing to flash memory, leveraging parallelism in SSD architecture, and using slicing to combine multiple reads into a single operation, allowing for optimal performance across different workloads and resource constraints.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If hash type indexes are compressed or reordered for space savings, then storage efficiency is improved, but the hash index becomes inoperable
Solution Approach 1:
The index is divided into two separate components: a hash table for fast key-based lookup and a sorted structure for space-efficient storage. This segmentation allows each component to serve its specific function optimally without compromising the other.
Solution Approach 2:
A mapping structure is introduced as an intermediary between the compressed sorted index and the actual data. This mapping layer preserves the operational integrity of the hash index while allowing the underlying storage to be compressed and reordered for space efficiency.
2Adaptability or versatility
If random page writes are performed on SSD, then write flexibility is improved, but write performance deteriorates
Solution Approach 1:
Data is pre-processed and organized into sorted runs before being written to SSD. This preliminary sorting action converts random write operations into sequential writes, dramatically improving write performance while maintaining the flexibility of random access through the hash table layer.
Solution Approach 2:
The system dynamically switches between hash table operations for fast random access and sorted structure operations for efficient sequential writes. This dynamic behavior allows the system to optimize for the current operation type, maintaining flexibility while improving overall productivity.
3Quantity of substance
If index size increases to accommodate growing data, then data capacity is improved, but access performance deteriorates
Solution Approach 1:
The growing index is segmented into multiple sorted runs that are stored sequentially on SSD. Each run maintains the sorted property, allowing binary search to remain effective even as the overall index size increases, thus preserving access performance while accommodating larger data capacities.
Solution Approach 2:
The system transitions from a single-dimension hash table to a two-dimensional structure combining hash-based key access with sorted multi-dimensional runs. This dimensional change allows the index to scale in capacity while maintaining fast access through the hash layer and efficient storage through the sorted runs.
4Quantity of substance
If SSD is used instead of DRAM for index storage, then cost and density are improved, but access latency increases
Solution Approach 1:
The index data is pre-sorted and organized into optimized runs before being written to SSD. This preliminary organization reduces the need for random access patterns during queries, allowing SSD to serve data more efficiently and reducing access latency while maintaining high density.
Solution Approach 2:
The system merges the fast key-based lookup capability of hash tables with the space-efficient sorted storage structure on SSD. This combination allows the system to leverage SSD's high density while minimizing access latency through intelligent data organization and access patterns.
Data Source
AI summary
Aspects of the present invention provide high-performance indexing for data-intensive systems in which “slicing” is used to organize indexing data on an SSD such that related entries are located together. Slicing enables combining multiple reads into a single “slice read” of related items, offering high read performance. Small in-memory indexes, such as hash tables, bloom filters or LSH tables, may be used as buffers for insert operations to resolve slow random writes on the SSD. When full, these buffers are written to the SSD. The internal architecture of the SSD may also be leveraged to achieve higher performance via parallelism. Such parallelism may occur at the channel-level, the package-level, the die-level and/or the plane-level. Consequently, memory and compute resources are freed for use by higher layer applications, and better performance may be achieved.