Distributed Storage Metering via Extent Manifests
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed data storage systems, accurately tracking and metering data consumption across multiple storage devices is complex due to time lags and variations in data replication, making it difficult to correctly charge customers for usage.
Innovation Solution
A method is introduced where each storage device generates a manifest of data attributes, such as size, and sends it to a shared records storage device, which processes and combines valid reports to determine derivative attributes like effective size, ensuring accurate billing and data management across replicated data extents.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is replicated across multiple storage devices to increase durability and scalability, then storage reliability and capacity are improved, but tracking data consumption and metering accuracy become more difficult
Solution Approach 1:
The patent segments data into extents that can be independently tracked across multiple storage devices. Each extent is assigned a unique identifier and is replicated to multiple devices, allowing the system to track consumption at the extent level rather than at the individual data level, thus maintaining metering accuracy while enabling replication for improved reliability
Solution Approach 2:
The patent introduces a metering service as an intermediary component that coordinates between storage devices and billing systems. This service collects metadata from storage devices, resolves replication inconsistencies, and provides accurate consumption data to billing systems, thereby solving the measurement precision problem without compromising storage reliability
2Adaptability or versatility
If data is replicated across multiple storage devices to improve scalability, then storage capacity and performance are improved, but tracking consumption becomes increasingly complex
Solution Approach 1:
The patent divides the storage system into independent extent units that can be replicated across multiple devices. This segmentation allows the system to scale by adding more storage devices without increasing the complexity of tracking individual data items, as extents serve as atomic units for both storage and metering
Solution Approach 2:
The patent uses metadata copies to track data consumption. Instead of tracking actual data movement, the system creates and maintains metadata records that represent the state of replicated data across storage devices. This copying approach enables scalability while simplifying consumption tracking to metadata manipulation rather than data-level monitoring
3Reliability
If data replication occurs with time lags, then data availability and durability are improved, but obtaining accurate data quantity information becomes difficult
Solution Approach 1:
The patent implements preliminary action by having storage devices report their data quantity information proactively to the metering service at regular intervals, rather than waiting for consumption events or requests. This continuous reporting ensures that the metering service has up-to-date information even during replication time lags, maintaining information accuracy while preserving data availability
Solution Approach 2:
The patent establishes a feedback mechanism where storage devices continuously report their state to the metering service, which then uses this information to resolve replication inconsistencies and provide accurate consumption data. This feedback loop ensures that time lags in data replication do not result in information loss, as the system actively monitors and reconciles data quantity information across all storage devices
Data Source
AI summary
Techniques described and suggested include systems, methods, and processes for determining data-related attributes, such as size, for data and/or data containers (such as volumes and file systems) resident on distributed storage systems. For example, a shared records storage device may be utilized to gather information related to file systems stored on a distributed storage system having a plurality of storage servers. A data processor may be used to generate information records and meter data storage size for file systems resident on the distributed storage system, and such records may be provided to requestors, such as customers associated with the file systems.