Object Store Size Estimation Using Histograms
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems for estimating the total size of objects in cloud-based object stores are inefficient, leading to potential overcharging or undercharging, as they either consume excessive resources or result in severe overestimation of storage size due to insufficient methods for tracking object sizes.
Innovation Solution
Implementing data structures such as histograms, zone maps, and quantization algorithms that provide upper and lower bound guarantees on estimated storage size, allowing for accurate estimation with low overhead, and ensuring crash recoverability and eventual consistency in distributed database environments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a list operation is used to traverse all objects in the object store, then the total size can be accurately calculated, but it consumes hours of time in the case of a large object store
Solution Approach 1:
The system pre-calculates and stores size information in a data structure (such as a Bloom filter or histogram) when objects are created or modified. This preliminary action allows the estimation to be retrieved quickly without traversing all objects, thus resolving the contradiction between accuracy and time consumption.
Solution Approach 2:
Instead of directly traversing all objects to calculate total size, the system maintains a copied representation of size information in a compact data structure. This copy allows fast estimation while the original object store remains unchanged, achieving both accuracy and efficiency.
2Measurement precision
If heavyweight data structures are implemented to track the total size as objects are created and destroyed, then the estimation accuracy is improved, but the system complexity and resource consumption increase
Solution Approach 1:
The system changes the parameter representation from tracking individual object sizes to using aggregated size distributions (histograms) or probabilistic structures (Bloom filters). This parameter transformation reduces complexity while maintaining estimation accuracy through mathematical properties of these structures.
Solution Approach 2:
Instead of uniformly tracking all objects with the same detailed level, the system uses local quality by applying different tracking granularities to different size ranges or object types, reducing overall complexity while maintaining necessary accuracy for pricing decisions.
3Productivity
If it is assumed that each object has a fixed size, then only the total number of objects needs to be tracked, but this results in severe overestimation of the total storage size
Solution Approach 1:
The system transitions from a static fixed-size assumption to a dynamic size distribution model. The data structure is updated as objects are created, modified, or deleted, allowing the size estimation to adapt to actual object size variations while maintaining tracking efficiency through incremental updates.
Solution Approach 2:
Instead of tracking every single object size detail, the system uses partial action by maintaining only the necessary aggregated size information (such as size buckets or probability distributions) that is sufficient for accurate estimation, avoiding the excessive complexity of full tracking.
Data Source
AI summary
A system includes detection of a first allocation of a first memory size in an object store for storage of a first logical page, in response to detection of the first allocation, incrementing a count associated with the first memory size by a first data structure associating a respective count with each of a plurality of memory sizes, detection of a first deallocation of the first logical page, in response to detection of the first deallocation, decrementing a count associated with a second one of the plurality of memory sizes by the first data structure, and determination of a memory usage associated with the object store based on the counts associated with each of the plurality of memory sizes by the first data structure wherein the second one of the plurality of memory sizes is different from the first memory size.