Hierarchical Block Storage Deduplication via Object Store Hashing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In a scenario where multiple block storage devices share an object storage device without deduplication functionality, the same data from these block storage devices are duplicated and stored in the object storage, leading to inefficiencies and increased storage costs.
Innovation Solution
A block storage device configuration that includes a hierarchical logical disk structure with physical devices of varying access performance, a deduplication unit, and a hierarchy changing unit. The deduplication unit searches the object storage for data with reduced access frequency and, if not found, transmits it to the object storage, while omitting transmission if the data is already present, thus preventing duplication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple block storage devices share an object storage device without deduplication functionality, then storage capacity is increased, but data duplication occurs leading to increased storage costs and reduced efficiency
Solution Approach 1:
The block storage device performs preliminary deduplication checks before transmitting data to the object storage device. The deduplication unit calculates hash values of data blocks and checks against previously stored hash values in the deduplication table, preventing duplicate data from being transmitted to object storage in the first place.
Solution Approach 2:
The system uses hash value copies instead of actual data copies for deduplication comparison. By storing only hash values in the deduplication table and comparing these lightweight copies, the system efficiently identifies duplicate data blocks without requiring actual data comparison, reducing overhead while maintaining storage efficiency.
2Reliability
If data is frequently transferred between block storage and object storage, then data availability is maintained, but network bandwidth is consumed and transfer time increases
Solution Approach 1:
The system performs preliminary hash value comparison before actual data transfer. By checking the deduplication table for existing hash values upfront, the system determines whether data transfer is necessary, avoiding unnecessary network transmissions and reducing overall data transfer time.
Solution Approach 2:
The deduplication table acts as an intermediary between block storage and object storage operations. It stores hash value mappings that mediate data transfer decisions, allowing the system to quickly determine data uniqueness without direct data comparison, thus reducing transfer overhead and time.
3Quantity of substance
If all data blocks are stored in object storage, then storage capacity is maximized, but access performance for frequently accessed data deteriorates
Solution Approach 1:
The storage system is segmented into two distinct layers: block storage for frequently accessed data requiring high performance, and object storage for archival data requiring high capacity. The hierarchical logical disk structure divides data blocks between these layers based on access patterns, with the active layer providing fast access and the archive layer providing extensive capacity.
Solution Approach 2:
Different storage locations are assigned different quality characteristics: block storage provides high-speed access quality for active data, while object storage provides high-capacity quality for archival data. The system dynamically assigns data blocks to appropriate storage locations based on their access requirements, ensuring each data block resides in the most suitable storage environment.
Data Source
AI summary
A first block storage device sharing an object storage device with a second block storage device has: plural kinds of physical devices having different access response performances; and a hierarchical logical disk created by using the storage areas of the physical devices and the storage area of the object storage device. When the access frequency of a logical block stored in a hierarchy one level above a lowest hierarchy falls below a threshold, the first block storage device searches the object storage device for a logical block having the hash of the logical block as a content address. When the search fails, the first block storage device transmits the logical block to the object storage device to change a hierarchy to store the logical block to the lowest hierarchy, whereas when the search succeeds, omits the transmission and changes a hierarchy to store the logical block to the lowest hierarchy.