Metadata Proxy Server for Asynchronous RPC Forwarding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing client agent and metadata server (MDS) architecture incurs significant latency due to the roundtrip communication required for file system requests, particularly in geographically separated setups, which affects the performance of metadata operations like create, rename, delete, and query file attributes during backup and restore processes.
Innovation Solution
The introduction of a metadata proxy server that forwards remote procedure calls (RPCs) from the client agent to an RPC client, which then sends them to the MDS asynchronously, allowing the client agent to respond instantly and reducing the perceived latency by short-circuiting the roundtrip communication path.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a client agent and metadata server architecture is used for file system operations, then data backup and restore functionality is provided, but significant latency is incurred due to roundtrip communication between geographically separated components
Solution Approach 1:
An MDS proxy is introduced as an intermediary component that receives RPCs from the client agent and forwards them to the MDS. The proxy acts as a local representative that can immediately acknowledge receipt of requests to the client agent, while the actual MDS processing occurs asynchronously in the background. This intermediary architecture allows the client agent to perceive low latency (instant acknowledgment) while still maintaining reliable communication with the remote MDS for data backup and restore operations.
2Reliability
If synchronous roundtrip communication is used between client agent and MDS, then correctness in file system operations is maintained, but performance of metadata operations deteriorates due to waiting for remote responses
Solution Approach 1:
The MDS proxy performs preliminary action by immediately acknowledging receipt of RPCs from the client agent before the actual MDS processing is complete. The proxy sends an instant acknowledgment to the client agent, allowing it to proceed with operations without waiting for the remote MDS response. The actual metadata operations are then performed asynchronously in the background, maintaining correctness while improving performance.
Solution Approach 2:
The system dynamically switches between synchronous and asynchronous communication modes. The MDS proxy provides synchronous-like behavior (instant acknowledgment) to the client agent while using asynchronous communication with the actual MDS. This dynamic approach allows the system to adapt to the different requirements of client responsiveness and remote processing reliability.
Data Source
AI summary
In one example, a method is performed by a client agent and includes receiving a metadata request from an application, generating a remote procedure call (RPC) that identifies metadata requested by the application, transmitting the RPC to a metadata server (MDS) proxy, receiving, from the MDS proxy, an acknowledgement that the RPC has been received and logged, receiving metadata identified in the RPC, and providing the metadata to the application.