Change Rate Estimation in Deduplicated Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In data storage systems with deduplication, estimating the rate of change in data storage requirements is complex due to the lack of a mapping from segments to files, making it difficult to determine the amount of changed data without reconstituting files and leading to inefficient storage allocation and increased costs.
Innovation Solution
A change rate estimator generates file measurements that approximate the number of unique segments belonging to a file at any two points in time, using combined bitmaps to estimate the change rate in physical bytes stored, and employs bloom filters to efficiently represent and store these measurements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data deduplication is implemented to optimize storage capacity, then storage utilization is improved, but the ability to accurately determine changed data is worsened
Solution Approach 1:
The patent introduces segment-to-file mapping as an intermediary data structure that connects deduplicated segments back to their source files. This mapping enables the system to track which segments belong to which files without compromising the deduplication efficiency, thus resolving the contradiction between storage optimization and change detection capability
Solution Approach 2:
The patent replaces the traditional mechanical approach of reconstituting entire files to determine changes with a data structure-based approach. By maintaining segment-to-file mappings in metadata, the system can efficiently identify changed segments through logical operations on the mapping data rather than physical file reconstruction
2Measurement precision
If segment-to-file mapping is maintained to enable change detection, then measurement precision is improved, but device complexity increases
Solution Approach 1:
The patent segments the file system into discrete units (segments) and maintains mappings at the segment level rather than file level. This segmentation allows for granular tracking of changes while keeping individual mapping entries simple and manageable, reducing overall system complexity while improving measurement precision
Solution Approach 2:
The patent implements segment-to-file mapping for all segments, which appears excessive at first glance. However, this partial mapping approach (mapping only what's necessary at the segment level) actually simplifies the overall system by avoiding the need for complex file-level reconstruction logic, thus improving precision without proportionally increasing complexity
Data Source
AI summary
File measurements are computed and stored in persistent memory of a deduplicated storage system as files are written or on demand, where the file measurements are used to estimate storage requirements for storing a subset of files. The file measurements are accumulated into an initial measurement at a first point in time and a final measurement at a second point in time to obtain an estimate of any change in a quantity of unique segments required to store the subset of files in the deduplicated storage system between the first and second points in time. Future storage requirements can be estimated based on a computed rate of change in the amount of storage required to store the subset of files between the first and second points in time.