Distributed Metadata Architecture for Cloud Storage Scalability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional cloud storage systems lack scalability for small files, have limited metadata capacity, and do not provide end-to-end data integrity and inline distributed deduplication, which restricts their deployment for applications requiring small files and high metadata handling.
Innovation Solution
The system distributes metadata between object-tracking and chunk-tracking metadata, using a federated file system for object metadata and a distributed database for chunk metadata, optimized with solid state drives and tailored storage requirements, enabling scalable metadata processing and inline deduplication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional cloud storage systems use a centralized metadata server, then system simplicity is maintained, but metadata capacity and scalability are limited
Solution Approach 1:
The patent segments metadata into two distinct types: object-tracking metadata stored in a federated file system and chunk-tracking metadata stored in a distributed database. This segmentation allows each metadata type to be managed independently with optimized storage mechanisms, dramatically increasing overall metadata capacity without creating a single point of failure or bottleneck
2Adaptability or versatility
If cloud storage systems support a large number of clients, then service coverage is improved, but metadata handling scalability deteriorates
Solution Approach 1:
By dividing metadata responsibilities between object-tracking (federated file system) and chunk-tracking (distributed database), the system can handle metadata operations in parallel across multiple nodes, enabling both increased client capacity and maintained metadata handling scalability
Solution Approach 2:
The patent introduces a two-dimensional metadata architecture where object metadata and chunk metadata operate in separate dimensions with different storage optimizations. This dimensional separation allows the system to scale clients in one dimension while maintaining efficient metadata handling in the other dimension through specialized storage paths
3Reliability
If cloud storage systems use traditional file systems, then ease of operation is maintained, but end-to-end data integrity and inline distributed deduplication are not provided
Solution Approach 1:
The patent introduces an intermediary layer between the federated file system and chunk servers that implements end-to-end data integrity checking and inline distributed deduplication. This intermediary handles cryptographic verification and deduplication operations without requiring fundamental changes to the underlying file system architecture, maintaining operational simplicity while adding reliability features
Data Source
AI summary
A method and system is disclosed for providing a cloud storage system supporting existing APIs and protocols. The method of storing cloud storage system (CSS) object metadata separates object metadata that describes each CSS object as a collection of named chunks with chunk locations specified as a separate part of the metadata. Chunks are identified using globally unique permanent identifiers that are never re-used to identify different chunk payload. While avoiding the bottleneck of a single metadata server, the disclosed system provides ordering guarantees to clients such as guaranteeing access to the most recent version of an object. The disclosed system also provides end-to-end data integrity protection, inline data deduplication, configurable replication, hierarchical storage management and location-aware optimization of chunk storage.