Backend Capacity Evaluation for Deduplicated Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data de-duplication storage systems face challenges in accurately determining the amount of storage space used by individual customers, especially when data is shared across multiple customers, leading to inefficiencies in billing and resource usage.
Innovation Solution
Implementing a sub-sampling method to estimate the storage capacity used by each customer, based on a sub-sample ratio that ensures minimum acceptable accuracy, leveraging billing rates and incremental billing increments to reduce processing resources required.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the service provider tracks all hashes associated with a particular user without counting duplicate hashes, then the billing accuracy is improved, but the processing resources and calculation time required increase significantly
Solution Approach 1:
The patent applies partial action by using sub-sampling to count only a portion of the hashes (e.g., 10%) associated with a user rather than tracking all hashes. This partial counting approach provides sufficient accuracy for billing purposes while dramatically reducing the processing resources and calculation time required, thus resolving the contradiction between billing accuracy and processing efficiency.
Solution Approach 2:
The patent changes the parameter of sampling ratio to balance accuracy and efficiency. By adjusting the sub-sampling ratio (e.g., counting 10% of hashes), the system achieves acceptable billing accuracy with significantly reduced processing overhead. This parameter change allows the system to trade off some measurement precision for improved productivity.
2Adaptability or versatility
If the service provider moves customer data to a different de-duplication domain, then data flexibility is improved, but the storage space requirement calculation becomes uncertain
Solution Approach 1:
The patent applies preliminary action by calculating the backend storage space requirements before actually moving the data. The system performs sub-sampling calculations to estimate the storage capacity needed for the customer's data in the target de-duplication domain. This preliminary estimation allows the service provider to verify storage availability before data migration, ensuring that the move can be completed successfully without causing storage shortages.
3Measurement precision
If the service provider charges customers based on exact storage usage, then billing precision is improved, but the calculation complexity and processing time increase
Solution Approach 1:
The patent applies partial action by using sub-sampling to count only a portion of the data hashes (e.g., 10%) needed for billing calculations. This approach provides sufficient billing precision for most commercial purposes while dramatically reducing the calculation time and processing resources required compared to counting all hashes. The sub-sampling method strikes a practical balance between billing precision and computational efficiency.
Data Source
AI summary
One example method is directed to generating a data sampling parameter, and includes identifying an incremental billing rate for data storage, where the data storage includes a data set having a multiple data pieces that each correspond with a respective unique indicator, and the unique indicators reside in a space. The example method further includes identifying a threshold billing increment, and determining an acceptable error in the threshold billing increment. Finally, the method includes determining a sub sample ratio based upon the threshold billing increment and the acceptable error. The sub sample ratio has a variable value ‘X’ which indicates that sampling of one unique indicator in every ‘X’ unique indicators in the space is sufficient to determine the size of the entire data set.