Efficiency Sets for Unique Data Estimation in Distributed Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed storage systems, determining the amount of memory reclaimed by deleting a data object is challenging due to shared data blocks referenced by multiple client addresses, making it difficult to accurately estimate the memory recovery.
Innovation Solution
A content manager uses efficiency sets to estimate the amount of memory freed up by deleting a data object by generating sets of block identifiers and calculating set differences, allowing for reduced computation time and improved user experience.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional methods are used to determine memory reclaimed by deleting data objects, then measurement precision may be improved, but computation time increases significantly
Solution Approach 1:
The patent creates efficiency sets that are simplified representations or copies of the actual data block reference structures. These efficiency sets contain the essential information needed to estimate memory recovery without requiring complete analysis of all data block references across the distributed storage system, thus providing accurate estimates with reduced computation time
Solution Approach 2:
The patent divides the complex problem of memory recovery calculation into smaller manageable segments by creating efficiency sets that focus on specific subsets of data blocks. This segmentation allows the system to process and analyze only the relevant portions of data needed for estimation, rather than examining the entire storage system state
2Measurement precision
If complete data block reference analysis is performed to accurately determine unique data, then measurement precision is improved, but device complexity increases
Solution Approach 1:
The patent introduces efficiency sets as intermediary structures that mediate between the complex distributed storage system and the memory recovery calculation process. These efficiency sets serve as simplified intermediaries that capture the essential reference information without requiring direct analysis of the complete system state, thus reducing complexity while maintaining accuracy
Data Source
AI summary
A system, method, and machine-readable storage medium for determining an amount of unique data in a distributed storage system are provided. In some embodiments, a combined efficiency set for a first data set stored in the distributed storage system, such as at a volume, may be generated. The first data set may include a first subset of data and a second subset of data in the distributed storage system. Additionally, a set of efficiency sets for the first subset of data may be generated. A set difference based on the combined efficiency set and the set of efficiency sets may be computed. An amount of memory used for storing unique data of the second subset of data may be estimated based on the set difference. The unique data may be present in the second subset of data but absent from the first subset of data.