Hybrid Metadata Storage Using Reversemap for Consistency at Scale
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed database systems face challenges in managing metadata across multiple geographic locations, requiring a balance between consistency and scalability, which existing solutions often fail to address effectively, leading to high computational and financial costs and latency issues.
Innovation Solution
A hybrid metadata storage architecture that separates metadata storage into a strongly consistent and non-scalable system for critical operations and a highly scalable, eventually consistent system for non-critical operations, using a reversemap to manage metadata efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If metadata is stored in a distributed database system across multiple geographic locations, then scalability is improved, but consistency deteriorates
Solution Approach 1:
The patent segments metadata into two categories: critical metadata requiring strong consistency and non-critical metadata allowing eventual consistency. This segmentation allows the system to store different types of metadata in different locations with different consistency guarantees, thereby achieving both scalability and consistency where needed.
Solution Approach 2:
The patent applies local quality by implementing different consistency models for different metadata types based on their specific requirements. Critical metadata operations use strong consistency guarantees while non-critical operations use eventual consistency, optimizing the system for each metadata type's specific needs rather than applying a uniform approach.
2Reliability
If strong consistency is enforced for all metadata operations, then consistency is improved, but performance deteriorates due to latency
Solution Approach 1:
The patent applies partial action by enforcing strong consistency only for critical metadata operations where it is truly needed, while allowing eventual consistency for non-critical operations. This partial application of strong consistency reduces the overall latency impact while maintaining consistency guarantees where necessary.
3Device complexity
If a unified metadata storage system is used, then device complexity is reduced, but adaptability deteriorates
Solution Approach 1:
The patent implements a unified metadata storage system that serves multiple functions: storing both critical and non-critical metadata, supporting both strong and eventual consistency models, and handling different types of metadata operations. This multi-functional approach maintains system simplicity while providing the adaptability needed for different metadata requirements.
Solution Approach 2:
The patent changes the consistency parameter based on metadata type and operation criticality. The system dynamically adjusts consistency requirements, using strong consistency for critical operations and eventual consistency for non-critical operations, thereby achieving adaptability within a unified storage framework.
Data Source
AI summary
Techniques are disclosed for managing metadata of a distributed database system in a hybrid manner. A computer system may receive, from a computing device, a request to access a set of data stored in nodes of a distributed storage system that is a caching layer of the system. The system retrieves metadata for a set of data specified in the request, including accessing a reversemap storing a reverse-ordered copy of original metadata stored in a metadata store of the system, where the reversemap is stored on durable storage of the system. Based on retrieving the metadata from the reversemap, the system accesses nodes of the distributed storage system, where the reversemap specifies the nodes of the distributed storage system that store the set of data. The system transmits, to the computing device, information indicating a result of accessing data stored in nodes of the distributed storage system.