Storage Capacity Estimation Using Milli-Indices and Deep Reference Counts
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deduplicated storage systems face challenges in estimating available storage capacity and resource management due to the complexity introduced by data deduplication, making it difficult to determine the resources that can be freed up or consumed when moving datasets between systems.
Innovation Solution
The use of shallow reference counts and milli-indices with deep reference count information, along with multiset operations, allows for accurate estimation of storage system capacity consumption, supporting data management and deduplication by sampling and tracking object references efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data deduplication is implemented in distributed storage systems, then storage efficiency and resource optimization are improved, but the complexity of estimating available storage capacity and managing resources increases
Solution Approach 1:
The patent segments the reference counting mechanism into shallow reference counts (tracking immediate parent metadata objects) and deep reference counts (tracking all transitive references through the entire tree). This segmentation allows the system to manage deduplication complexity by maintaining multiple levels of reference tracking with different scopes, enabling accurate capacity estimation while preserving storage efficiency benefits
Solution Approach 2:
The patent introduces a new dimension of reference counting by transitioning from simple shallow reference counts to deep reference counts that traverse the entire metadata tree. This dimensional change in tracking depth enables the system to accurately estimate capacity consumption even in complex deduplicated storage environments, resolving the contradiction between storage efficiency and estimation accuracy
2Speed
If shallow reference counts are used to track object references, then the speed of operations is improved, but the precision of capacity estimation deteriorates
Solution Approach 1:
The patent performs preliminary actions by maintaining both shallow reference counts (for fast operations) and deep reference counts (for accurate estimation) concurrently. The shallow reference counts provide immediate operational speed while deep reference counts, pre-computed and maintained, provide the precision needed for accurate capacity estimation when required
Solution Approach 2:
The patent uses deep reference counts as an intermediary mechanism that bridges the gap between fast shallow reference counting and accurate capacity estimation. The deep reference counts are maintained as an additional layer that can be queried when precision is needed, allowing the system to achieve both speed and precision without compromising either
3Measurement precision
If deep reference count information is maintained for all objects, then the accuracy of storage capacity estimation is improved, but the device complexity and overhead increase
Solution Approach 1:
The patent applies local quality by differentiating the depth of reference counting based on the specific needs of different objects and operations. Deep reference counts are maintained for objects where accurate capacity estimation is critical, while shallow reference counts suffice for operations where speed is prioritized. This localized application of different reference counting depths reduces overall system complexity while maintaining estimation accuracy where needed
Data Source
AI summary
Techniques and architectures to estimate storage system capacity are disclosed. Deduplicated data and an index of object entries are maintained in a storage system. The entries contain shallow reference counts. The shallow reference count values indicate a number of parent metadata objects that hold a reference to corresponding object. One or more tree milli-indices and one or more node milli-indices of object entries are maintained in the storage system. The entries correspond to a subset of objects stored in the storage system. The entries also have truncated object signature values and deep reference count values for the corresponding objects in the storage system. A capacity utilization of the storage system is determined based on analysis utilizing the deep reference count values to perform various multiset operations.