Virtualization Storage De-duplication via Read-Time Fingerprinting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In virtualization environments, existing approaches to storage management are inefficient due to coarse-grained administration and the inability to effectively de-duplicate data within high-performance tiers, leading to suboptimal resource utilization and performance.
Innovation Solution
Implementing a content cache on high-performance storage tiers, such as RAM and flash, that uses fingerprints (like SHA1) to associate and de-duplicate data chunks, allowing for efficient read operations while writing data without initial de-duplication, thereby optimizing storage utilization and performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is written without initial de-duplication to high-performance storage tiers, then write operations are faster, but storage space is wasted due to duplicated data
Solution Approach 1:
The system performs de-duplication during read operations rather than write operations. When data is read from high-performance storage tiers, the system checks for duplicate copies and consolidates them, thereby eliminating storage waste without impacting write speed. This preliminary action is deferred to the read phase, allowing writes to proceed at full speed.
Solution Approach 2:
The de-duplication process is integrated into the read operation itself, where the system serves data requests while simultaneously performing de-duplication. The read operation serves dual purposes: delivering data to the user and identifying/consolidating duplicates, thereby eliminating the need for separate de-duplication processing.
2Loss of substance
If de-duplication is performed during read operations in high-performance tiers, then storage utilization improves, but read operation complexity increases
Solution Approach 1:
The system merges the de-duplication function with the read operation, so that a single read operation simultaneously retrieves data and performs de-duplication checking. This consolidation eliminates the need for separate de-duplication processing steps, reducing overall system complexity while achieving storage optimization.
Solution Approach 2:
The read operation is designed to perform multiple functions: data retrieval, duplicate detection, and duplicate consolidation. This multi-functional approach eliminates the need for specialized de-duplication hardware or software components, thereby reducing system complexity while achieving storage optimization.
3Speed
If a content cache is implemented on high-performance storage tiers, then read performance improves, but storage capacity is reduced
Solution Approach 1:
The system extracts duplicate data copies from high-performance storage tiers and consolidates them into a single copy. This extraction process eliminates wasted storage space while maintaining the content cache functionality, as the cache retains only unique data blocks that need to be cached for performance optimization.
Solution Approach 2:
The system changes the state of stored data from duplicated multiple copies to a single de-duplicated copy, thereby changing the storage capacity parameter. This parameter change is achieved through fingerprint-based identification and consolidation, which allows the system to maintain cache effectiveness while reducing total storage requirements.
Data Source
AI summary
The present application provides an improved approach for managing performance tier de-duplication in a virtualization environment. A content cache is implemented on high performance tiers of storage in order to maintain a working set for the user virtual machines accessing the system, and associates fingerprints with data stored therein. During write requests from the user virtual machines, fingerprints are calculated for the data to be written. However, no de-duplication is performed during the write. During read requests, fingerprints corresponding to the data to be read are retrieved and matched with the fingerprints associated with the data in the content cache. Thus, while multiple pieces of data having the same fingerprints may be written to the lower performance tiers of storage, only one of those pieces of data having that fingerprint will be stored in the content cache for fulfilling read requests.