Reducing Cloud Storage Metadata Size via Object Identifier Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud storage systems face inefficiencies due to large data object identifiers that burden storage nodes, consume valuable metadata space, and result in computationally intensive and time-intensive data retrieval, exacerbated by erasure coding that repeats metadata across multiple storage nodes.
Innovation Solution
A system that uses a reduction function, such as a hashing function, to generate a smaller second object identifier from a larger first object identifier, reducing metadata size and improving data retrieval speed by fitting more metadata in cache memory and reducing comparison operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If large data object identifiers are used to uniquely identify data objects, then identification accuracy is improved, but metadata size increases and storage efficiency deteriorates
Solution Approach 1:
The system segments the metadata storage by separating the large object identifier from the metadata structure. Instead of storing the complete large identifier in every metadata instance, the system divides the identifier into a stored portion (in metadata) and a client-held portion, reducing metadata size while preserving full identification capability through combination of both portions.
Solution Approach 2:
The system transitions from a single-dimension storage approach (storing complete identifiers in metadata) to a multi-dimensional approach where identifiers are distributed across multiple dimensions: client devices hold one portion, servers hold another portion, and metadata stores a reduced portion. This dimensional distribution reduces metadata size while maintaining identification accuracy.
2Adaptability or versatility
If large data object identifiers are stored in metadata across multiple storage nodes, then data object distribution is improved, but data retrieval speed deteriorates due to computationally intensive comparisons
Solution Approach 1:
The system applies partial action by storing only a portion of the object identifier in metadata rather than the complete identifier. This partial storage reduces the comparison workload during data retrieval operations while still enabling accurate identification when the client portion is combined with the stored portion, thus improving retrieval speed without sacrificing distribution capability.
3Reliability
If erasure coding is applied to distribute data objects across multiple storage nodes, then system reliability is improved, but computational overhead increases due to repeated metadata presence
Solution Approach 1:
The system extracts the computationally burdensome portion of the identifier from the metadata structure and relocates it to client devices. This extraction eliminates the need for repeated full identifier comparisons across multiple storage nodes during erasure coding operations, reducing computational overhead while maintaining system reliability through the distributed identifier verification mechanism.
Data Source
AI summary
Metadata sizes for data objects in cloud storage systems can be reduced. For example, a computing system can receive, at a client device of a cloud storage system, a first object identifier associated with a data object in the cloud storage system. The computing system can use a reduction function to generate a second object identifier associated with the data object. The second object identifier can have a smaller byte size than the first object identifier. The computing system can transmit, to a server of the cloud storage system, the second object identifier to be stored in metadata associated with the data object in the cloud storage system.