External Storage Estimation Using HyperLogLog for Deduplication Savings
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
It is challenging for external host systems to determine the efficacy of deduplication and compression in network storage systems, making it difficult to assess the performance and cost savings of these systems.
Innovation Solution
The implementation of a method using a hyperloglog algorithm to estimate deduplication and compression ratios by generating fingerprints from data blocks, compartmentalizing them into buckets defined by precision bits or compression ratios, allowing for the calculation of deduplication and effective compression ratios.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data deduplication and compression techniques are implemented in network storage systems, then storage efficiency and cost savings are improved, but the ability of external host systems to determine and measure the efficacy of these techniques deteriorates
Solution Approach 1:
The patent introduces an intermediary measurement mechanism that allows external host systems to estimate deduplication and compression ratios without internal access to the storage system. By using sampling techniques and statistical algorithms, the host can obtain meaningful metrics about storage efficiency while maintaining system encapsulation and security boundaries.
Solution Approach 2:
The patent replaces direct internal measurement mechanisms with external statistical estimation methods. Instead of requiring internal sensors or meters within the storage system, the solution uses algorithmic approximation through sampling and hashing techniques that can be performed externally to achieve similar measurement objectives.
2Measurement precision
If detailed internal monitoring of deduplication and compression operations is implemented, then measurement precision is improved, but device complexity increases
Solution Approach 1:
The patent applies partial action by using sampling rather than complete monitoring of all data blocks. Instead of analyzing every single block for deduplication and compression metrics, the system selectively samples representative portions of data, achieving sufficient measurement precision while avoiding the complexity and overhead of exhaustive internal monitoring.
Solution Approach 2:
The patent uses temporary, disposable data structures such as hash tables and sampling buffers that are created during measurement and discarded afterward. These short-lived objects enable precise local measurements without requiring permanent complex infrastructure within the storage system.
Data Source
AI summary
Methods for estimating cost savings in a storage system using an external host system. One method includes accessing over a communication network data from a unit of storage of a data storage system, wherein each of the blocks of data is uncompressed. A plurality of blocks is parsed from the data. A plurality of fingerprints is generated from the blocks using a hash algorithm. A deduplication ratio is estimated for the plurality of blocks stored in the unit of storage using a hyperloglog algorithm and a first plurality of buckets compartmentalizing the plurality of blocks, wherein the first plurality of buckets is defined by precision bits of the plurality of fingerprints. An effective compression ratio is estimated for the plurality of blocks stored in the unit of storage using the hyperloglog algorithm and a second plurality of buckets compartmentalizing the plurality of blocks, wherein the second plurality of buckets is defined by ranges of compression ratios.