Scale Out Storage Metadata Management via Dual Mapping and Caching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Scale out storage systems face inefficiencies in metadata management, leading to performance issues, high network latency, and resource exhaustion due to large metadata sizes and lack of effective caching, especially during scale out operations and logical device mobility.
Innovation Solution
A storage system is configured with a plurality of nodes connected by a network, utilizing first and second metadata to map logical addresses to content identifiers and physical addresses, respectively, with redundant storage and caching strategies to accelerate access and performance, including tree data structures and local caching to optimize metadata management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If metadata is stored persistently on storage devices with caching in memory, then data durability is improved, but access speed deteriorates due to lack of effective caching
Solution Approach 1:
The patent segments metadata into multiple copies distributed across different storage devices. Each node maintains redundant metadata copies, allowing the system to serve metadata requests from multiple sources simultaneously, thereby improving access speed while maintaining data durability through redundancy.
Solution Approach 2:
The patent implements extensive copying of metadata across storage devices and nodes. Metadata is replicated multiple times throughout the storage system, enabling any node to serve as a metadata source. This copying strategy ensures both data durability (through redundancy) and fast access (through local caching and multiple access points).
2Manufacturing precision
If metadata size increases to support larger capacity and smaller data blocks, then storage precision is improved, but memory caching effectiveness deteriorates
Solution Approach 1:
The patent applies local quality by allowing different nodes to cache different portions of metadata locally. Each node maintains a local cache tailored to its specific access patterns and workload, rather than requiring all nodes to cache the entire metadata. This enables effective caching even as total metadata size increases.
Solution Approach 2:
The patent implements partial caching strategies where nodes cache only the metadata portions they frequently need, rather than caching all metadata. The system accepts that not all metadata will be cached at all nodes simultaneously, but through redundancy and distributed caching, ensures that any required metadata is available from some node with acceptable latency.
3Adaptability or versatility
If reverse map metadata is accessed frequently during scale out operations, then scale out capability is improved, but system resources are exhausted due to thrashing
Solution Approach 1:
The patent performs preliminary actions by pre-loading and caching reverse map metadata into memory before scale out operations are needed. Nodes proactively maintain reverse map metadata in their local caches, so when scale out operations occur, the metadata is already available in memory rather than requiring disk access during the operation, preventing thrashing.
Solution Approach 2:
The patent ensures continuous maintenance of reverse map metadata in memory through background processes that keep the metadata fresh and available. Rather than loading metadata only when needed, the system continuously maintains it in cache, ensuring uninterrupted access during scale out operations and eliminating resource exhaustion from repeated loading.
4Adaptability or versatility
If logical device mobility is implemented by moving data blocks between nodes, then device versatility is improved, but access time increases due to network latency
Solution Approach 1:
The patent introduces metadata as an intermediary layer that abstracts the physical location of data blocks. When logical device mobility occurs, the metadata is updated to reflect new locations, but user applications continue to access data through the same logical addresses. The metadata translation layer handles the physical movement transparently, minimizing access time by avoiding direct network operations for every data access.
Data Source
AI summary
A method for managing metadata in a scale out storage system is disclosed. The system includes a plurality of nodes, a storage pool, first metadata that maps logical addresses of logical data blocks to corresponding content identifiers, and second metadata that maps content identifiers to corresponding physical addresses of physical data blocks in the storage pool and maintains a reference count. During an add-a-node operation, the processors are configured to move from the existing nodes to the new node some of its physical data blocks, their content identifiers and reference counts in the second metadata without accessing or altering the first metadata. A method is disclosed to move a logical device from one node to another by de-activating the logical device's first metadata on the first node and activating and retrieving the logical device's first metadata on the second node.