Composite Storage Data Migration Using Probabilistic Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for managing composite disk systems, combining fast and slow non-volatile storage devices, are not space efficient and require large amounts of memory to allocate and migrate data effectively between the two components.
Innovation Solution
A method that uses a first data structure to track recency of access on the fast SSD and a probabilistic hash table or Bloom filter to track recent access on the slower HDD, reducing memory overhead while maintaining efficient data allocation and migration between the two storage devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional data structures (LRU, CLOCK, ClockPro) are used to manage composite disk systems, then data allocation and migration between fast and slow storage devices is improved, but memory overhead increases significantly
Solution Approach 1:
The patent changes the parameter of data structure type from traditional deterministic structures (LRU, CLOCK) to probabilistic structures (Bloom filter, probabilistic hash table). This parameter change allows the system to achieve comparable data allocation efficiency while dramatically reducing memory overhead from gigabytes to kilobytes, directly resolving the technical contradiction between productivity and quantity of substance.
Solution Approach 2:
The patent employs probabilistic data structures that are space-efficient and can be easily reconstructed. These structures act as disposable, low-cost alternatives to traditional memory-intensive data structures, providing sufficient functionality for tracking recently evicted blocks without requiring persistent large memory allocations.
2Productivity
If large amounts of main memory are allocated to store data structures for data migration, then data allocation between storage devices is improved, but space efficiency deteriorates
Solution Approach 1:
The patent fundamentally changes the memory usage parameter by replacing traditional data structures with probabilistic alternatives. The Bloom filter and probabilistic hash table require only kilobytes of memory compared to the gigabytes required by LRU or CLOCK algorithms, while maintaining sufficient data migration efficiency through probabilistic tracking of recently evicted blocks.
Solution Approach 2:
The patent uses probabilistic hash tables and Bloom filters that create compact probabilistic copies of access pattern information rather than maintaining full deterministic records. These probabilistic copies provide sufficient information for data migration decisions without requiring complete duplication of access history in memory.
3Quantity of substance
If probabilistic hash table or Bloom filter is used to track recent access on slower storage device, then memory overhead is reduced, but measurement precision of access tracking deteriorates
Solution Approach 1:
The patent accepts a parameter change from deterministic to probabilistic measurement precision in exchange for dramatic memory overhead reduction. The Bloom filter and probabilistic hash table provide sufficiently accurate access tracking for the specific use case of identifying recently evicted blocks, where approximate information is adequate and false positives can be handled by checking actual access patterns.
Solution Approach 2:
The probabilistic data structures provide partial information about access patterns rather than complete accuracy. This partial information is sufficient for the specific task of identifying candidates for data migration, where excessive precision would require prohibitively large memory allocations. The system performs the necessary action with adequate rather than perfect precision.
Data Source
Figure 1~2
Figure 3~4
Figure 5
AI summary
Approaches to managing a composite, non-volatile data storage device are described. In one embodiment, a method for managing a composite storage device made up of fast non-volatile storage, such as a solid state device, and slower non-volatile storage, such as a traditional magnetic hard drive, can include maintaining a first data structure, which stores instances of recent access to each unit in a set of units in the fast non-volatile storage device, such as the SSD device and also maintaining a second data structure that indicates whether or not units in the slower storage device, such as the HDD, have been accessed at least a predetermined number of times. In one embodiment, the second data structure can be a queue of Bloom filters.