Solid State Deduplication Index Hierarchical Cache
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current storage systems face challenges in achieving stable latency and efficient deduplication, especially when dealing with out-of-order writes, which leads to increased memory consumption and performance degradation, as existing techniques like bloom filters and prefetching require significant memory and are ineffective when the order of duplicates is not preserved.
Innovation Solution
A storage system utilizing a Solid State Deduplication Index (SSDI) structure, which includes a data management unit and a duplication determination unit, employs feature data and index data to determine duplicate storage target data, with index data being stored in a hierarchical write cache across main and auxiliary storage devices to manage memory and performance effectively.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If a disk-resident hash table is used for deduplication, then deduplication capacity is sufficient, but lookup latency becomes unstable and requires random disk reads
Solution Approach 1:
The patent segments the hash table into multiple regions (hot region, warm region, cold region) based on access frequency. The hot region is kept in main storage device for fast access, while less frequently accessed regions are in auxiliary storage devices. This segmentation allows the system to maintain sufficient deduplication capacity across all regions while providing fast lookup for frequently accessed hashes, resolving the contradiction between capacity and latency.
2Speed
If bloom filter and prefetch techniques are used, then lookup speed improves, but memory consumption increases significantly
Solution Approach 1:
The patent changes the physical storage parameter of the hash table from entirely disk-resident to a hierarchical structure with portions in main storage. By adjusting the size and distribution of the hot region in main storage versus auxiliary storage, the system optimizes the balance between lookup speed and memory consumption, avoiding the need for large in-memory bloom filters while maintaining fast lookup performance.
3Speed
If prefetch is used assuming order preservation, then positive answer lookup speeds up, but performance drops dramatically when duplicates are out-of-order
Solution Approach 1:
The patent implements a dynamic hash table structure that adapts to changing access patterns. The hot region is dynamically updated based on actual access frequency, allowing the system to efficiently handle both in-order and out-of-order duplicates. This dynamic adaptation eliminates the performance penalty associated with rigid prefetch assumptions, maintaining high throughput regardless of duplicate order.
4Loss of time
If all hashes are kept in RAM, then lookup latency is minimized, but memory size requirements become unreasonably large
Solution Approach 1:
The patent applies local quality by providing different storage media for different portions of the hash table based on their access characteristics. The hot region, which requires fastest access, is placed in main storage device with higher speed. Less frequently accessed regions are placed in auxiliary storage devices. This local optimization allows fast lookup for critical operations while avoiding the need to keep the entire hash table in high-speed memory.
Data Source
AI summary
The storage system includes a first auxiliary storage device, a second auxiliary storage device, and a main storage device, and also includes a data management unit which stores and keeps, in the main storage device, index data based on feature data by referring to the feature data of storage target data stored in the first auxiliary storage device, and if the index data stored and kept in the main storage device reaches a preset amount, stores and keeps, in the second auxiliary storage device, the index data stored and kept in the main storage device, and deletes the index data stored and kept in the second auxiliary storage device from the main storage device.