Host Cache Flush Deduplication for Distributed Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed storage systems, existing technologies face inefficiencies in managing data deduplication and cache flushing, leading to increased resource consumption and network load, as they lack effective mechanisms for host-side caching and deduplication before data is written to front-end storage elements.
Innovation Solution
Implementing host-side caching with a deduplicating cache manager that aggregates unique data from cached writes into a 'change set file' and sends commands to the distributed storage system to share or copy only the unique data, reducing the load on network resources and storage space by leveraging deduplication across flush boundaries.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If data is written directly to front-end storage elements without host-side caching, then storage operations are simple, but network resource consumption and storage space usage increase due to lack of deduplication
Solution Approach 1:
The host performs deduplication operations on cached writes before the data is supplied to front-end storage elements. This preliminary deduplication action identifies duplicate data blocks and creates a change set file containing only unique data, thereby reducing network resource consumption during the cache flush operation while avoiding the complexity of implementing deduplication in the storage system itself
2Quantity of substance
If all cached writes are supplied to front-end storage elements, then data integrity is maintained, but storage space is wasted due to lack of deduplication
Solution Approach 1:
The host performs deduplication operations on cached writes before the data is supplied to front-end storage elements. This preliminary deduplication action identifies duplicate data blocks and creates a change set file containing only unique data, thereby reducing network resource consumption during the cache flush operation while avoiding the complexity of implementing deduplication in the storage system itself
Solution Approach 2:
A change set file is introduced as an intermediary data structure between the host cache and the distributed storage system. This change set file aggregates unique data from multiple cached writes and serves as the intermediary through which deduplicated data is transferred to storage, reducing both storage space consumption and network resource usage
3Productivity
If multiple individual writes are processed separately, then data management is simple, but network load and processing time increase
Solution Approach 1:
Multiple individual cached writes are merged and aggregated into a single change set file based on deduplication results. This merging process combines multiple data streams into one consolidated file that contains only unique data blocks, thereby improving cache flush efficiency by reducing the number of individual write operations to the distributed storage system while minimizing processing time through efficient deduplication algorithms
Data Source
AI summary
In addition to caching I/O operations at a host, at least some data management can migrate to the host. With host side caching, data sharing or deduplication can be implemented with the cached writes before those writes are supplied to front end storage elements. When a host cache flush to distributed storage trigger is detected, the host deduplicates the cached writes. The host aggregates data based on the deduplication into a “change set file” (i.e., a file that includes the aggregation of unique data from the cached writes). The host supplies the change set file to the distributed storage system. The host then sends commands to the distributed storage system. Each of the commands identifies a part of the change set file to be used for a target of the cached writes.