Remote Storage Device Handling With Metadata Caching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Remote replication in software-defined data storage systems incurs significant network and disk access latency due to per-write metadata retrieval from remote storage nodes, which is inefficient and adds to overall latency.
Innovation Solution
Implementing an application programming interface (API) on the production node to copy all filesystem metadata associated with remote storage objects to memory upon connection establishment, allowing for cached metadata usage to generate remote replication input-output commands, thereby eliminating the need for per-IO metadata retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If per-write metadata retrieval from remote storage nodes is used, then data consistency is maintained, but network and disk access latency increases significantly
Solution Approach 1:
The patent applies preliminary action by copying all filesystem metadata from the remote storage node to the production node's memory before replication operations begin. This pre-loading of metadata eliminates the need for repeated disk accesses during replication, as the metadata is already available in memory when write operations occur.
Solution Approach 2:
The patent uses copying by creating a local copy of the filesystem metadata on the production node. Instead of accessing metadata on the remote storage node for every write operation, the system copies the metadata to local memory, allowing fast access without network delays while maintaining data consistency.
2Measurement precision
If per-write metadata retrieval is implemented, then accurate storage object identification is achieved, but replication efficiency decreases
Solution Approach 1:
The system performs preliminary action by pre-copying metadata to memory before replication operations. This ensures that when write operations occur, the metadata is already available for accurate storage object identification without delaying replication efficiency.
Solution Approach 2:
The patent introduces an intermediary mechanism by using a local metadata cache on the production node. This intermediary layer provides fast access to metadata information needed for accurate storage object identification, eliminating the need for slow remote disk accesses while maintaining precision.
3Loss of time
If metadata is cached in production node memory, then network and disk access latency is reduced, but memory resource consumption increases
Solution Approach 1:
The patent uses copying to create a local metadata cache on the production node. This copy of the metadata enables fast access without network or disk delays. The system balances memory consumption by only caching the metadata portion rather than entire datasets.
Solution Approach 2:
The patent applies parameter changes by altering the storage location of metadata from remote disk to local memory. This parameter change (location) improves access speed while the scope is limited to metadata size rather than full data, managing memory resource consumption effectively.
Data Source
AI summary
In a software-defined data storage system with a production node and local and remote storage nodes that replicate data generated by an application running on the production node, all filesystem metadata associated with remotely replicated devices is copied from the remote storage node to memory of the production node in response to establishment of a communication link between the production node and remote storage node. That metadata is maintained in the memory for as long as the link is active and used to generate multiple remote replication IOs. Local replication is implemented by reading only necessary metadata from the local storage node on a per-IO basis. The cached metadata is flushed from memory in response to loss of the communication link.