Flash Key-Value Storage Data Reduction via Compressibility-Aware RAID Stripes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage systems, particularly primary storage solutions, face challenges in implementing content-based storage schemes due to intensive computational load, block fragmentation, loss of sequential data location, and algorithmic difficulties in distributing tasks across multiple processing and storage elements, leading to performance degradation and inefficiencies in handling exponential data growth.
Innovation Solution
The method involves splitting empty RAID stripes into sub-stripes based on compressibility scores, compressing data, and distributing pages across these sub-stripes, allowing for parallel compression and decompression while maintaining logical block structure, and using content-addressable storage to ensure deduplication and optimal resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If content-based storage scheme is implemented, then storage capacity efficiency is improved, but computational load increases
Solution Approach 1:
The patent segments the storage system into multiple independent storage nodes that each maintain their own content-addressable storage tables. This distribution of the content-based mapping tasks across multiple nodes reduces the computational load on any single processor while maintaining the storage efficiency benefits of content-addressable storage at scale.
2Manufacturing precision
If large blocks are broken into smaller blocks for content addressing, then content addressing granularity is improved, but block fragmentation increases
Solution Approach 1:
The patent introduces a new dimension of organization by creating hierarchical content-addressable storage tables that map fine-grained content hashes to coarser-grained block locations. This allows the system to maintain content addressing at fine granularity while organizing data in larger contiguous blocks, avoiding fragmentation while preserving the benefits of content-based addressing.
3Productivity
If content-based mapping is used, then duplicate block writes are eliminated, but sequential location of data blocks is lost
Solution Approach 1:
The patent introduces content-addressable storage tables as an intermediary layer between the content-based deduplication mechanism and the physical storage medium. These tables store the mapping between content hashes and physical block locations, preserving sequential location information while maintaining the efficiency benefits of content-based mapping for eliminating duplicate writes.
4Quantity of substance
If content-based storage is implemented, then storage efficiency is improved, but algorithmic difficulty in distributing tasks increases
Solution Approach 1:
The patent segments the content-addressable storage namespace into multiple distributed tables across different storage nodes. Each node manages a portion of the content-to-location mappings, allowing the system to maintain high storage efficiency through content-based deduplication while reducing algorithmic complexity by localizing lookup operations to specific nodes based on hash distribution.
Data Source
AI summary
In one aspect, a method includes splitting empty RAID stripes into sub-stripes and storing pages into the sub-stripes based on a compressibility score. In another aspect, a method includes reading pages from 1-stripes, storing compressed data in a temporary location, reading multiple stripes, determining compressibility score for each stripe and filling stripes based on the compressibility score. In a further aspect, a method includes scanning a dirty queue in a system cache, compressing pages ready for destaging, combining compressed pages in to one aggregated page, writing one aggregated page to one stripe and storing pages with same compressibility score in a stripe.