Metadata Prefetching in Cloud Storage Nodes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud storage systems experience significant delays in reading large data objects due to the time-consuming process of fetching and assembling metadata, which involves multiple input/output operations and can lead to substantial delays in servicing read requests.
Innovation Solution
Implementing metadata prefetching, where metadata for data objects is cached in a node cache prior to receiving a read request, allowing for quicker retrieval and prioritization based on estimated usage or expiration time, thereby reducing the need for subsequent I/O operations and enhancing read performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If metadata is fetched from node storage when a read request is received, then the storage system can service read requests, but significant delay occurs due to multiple I/O operations required to fetch and assemble metadata
Solution Approach 1:
The system performs preliminary actions by prefetching metadata into the node cache before read requests arrive. The gateway sends metadata to storage nodes in advance, and nodes store it in their caches, so when read requests come in, the metadata is already available and no time-consuming I/O operations are needed at request time.
Solution Approach 2:
The node cache acts as an intermediary between the gateway and node storage. Instead of directly fetching metadata from node storage when a read request arrives, the system uses the cache as a buffer that holds pre-fetched metadata, eliminating the need for time-consuming I/O operations during request processing.
2Productivity
If metadata is cached in node cache prior to read requests, then read performance is improved, but additional I/O operations are required to prefetch metadata
Solution Approach 1:
The storage nodes perform self-service by automatically caching metadata in their own caches when they receive data shards. The nodes proactively fetch and store their own metadata without waiting for external requests, enabling them to quickly serve read requests later without requiring additional I/O operations at request time.
Solution Approach 2:
The system maintains continuous useful action by having nodes continuously prefetch and cache metadata as data shards are written. This ongoing caching process ensures that metadata is always ready in the cache when read requests arrive, eliminating idle time and keeping the system continuously productive.
3Loss of information
If multiple I/O operations are performed to fetch and assemble metadata, then complete metadata is obtained, but the process becomes time-consuming and reduces overall system efficiency
Solution Approach 1:
The system creates copies of metadata and stores them in node caches. Instead of performing multiple I/O operations to fetch and assemble complete metadata from node storage when needed, the system maintains duplicate copies in fast cache memory, allowing immediate retrieval without time-consuming operations while ensuring complete metadata is available.
Data Source
AI summary
Aspects and features of the present disclosure can prefetch metadata in the nodes of a cloud-based storage system. At a node that stores at least one shard of a data object, metadata for the data object can be fetched from node storage and saved in a node cache prior to processing a read request for the shard. The metadata can be cached in response to a prefetch request transmitted to the node, for example, by the gateway that shards data objects for the storage system. Thus, the metadata can be available in the node cache when the read request for the data shard arrives later, reducing delays in accessing data object shards from the storage nodes of the system.