Persistent Metadata Element Trees for Distributed Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed storage systems face inefficiencies in data access and scalability due to resource coupling, locking issues, and high hardware costs, leading to performance bottlenecks as they scale.
Innovation Solution
A storage system utilizing persistent metadata structures in the form of element trees with hash table, range, bitmap, and content blocks, allowing dynamic growth, reduced storage space, and minimized lock contention, enabling efficient data access and scalability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If locking mechanisms are implemented to ensure consistency during concurrent access operations, then data consistency is improved, but lock overhead and lock contention increase
Solution Approach 1:
The patent divides the namespace into multiple namespaces, with each namespace having its own independent root inode. This segmentation allows different namespaces to be accessed concurrently without requiring global locks, thereby reducing lock contention while maintaining data consistency within each namespace.
Solution Approach 2:
The patent introduces copy-on-write inodes as an intermediary mechanism. When metadata needs to be modified, a copy of the inode is created rather than modifying the original in place. This intermediary approach eliminates the need for locking during metadata operations, as readers continue to access the original inode while writers operate on the copy.
2Reliability
If high performing persistent storage technologies such as Flash and NVRAM are used for metadata and data, then write-erase cycle reliability and access speed are improved, but hardware costs increase significantly
Solution Approach 1:
The patent applies different storage media to different parts of the storage system based on their specific requirements. High-performance persistent storage (Flash/NVRAM) is used specifically for the element store containing metadata structures, while capacity-oriented storage (HDD) is used for bulk data storage. This local quality approach ensures reliability where needed while controlling overall costs.
Solution Approach 2:
The patent segments the storage system into distinct layers: an element store for metadata (using persistent storage) and a data store for bulk data (using capacity-oriented storage). This segmentation allows optimization of each layer independently, using expensive high-performance storage only where it provides the most value (metadata access) and cheaper storage for bulk capacity.
3Quantity of substance
If the number of servers is increased to add storage capacity in distributed file systems, then storage scalability is improved, but resource coupling causes computing resources to also increase
Solution Approach 1:
The patent extracts the metadata management function from the coupled compute-storage server model and places it in a dedicated element store. This extraction allows storage capacity to be scaled independently by adding more storage nodes without requiring proportional increases in computing resources, as metadata operations are separated from data storage operations.
Data Source
AI summary
A storage system. The storage system includes a data store including a plurality of data blocks; and an element store, the element store including a plurality of element trees, wherein each element tree includes a plurality of metadata blocks containing pointers, wherein data of the plurality of data blocks is read by navigating at least one of the plurality of element trees, wherein the element store includes a first level and at least one second level, wherein the first level is shared among the plurality of plurality of element trees.