RPC Framework Extent Transfer via Out-of-Band Port
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing mechanisms, such as remote procedure call (RPC) frameworks, are inefficient and memory-intensive for transferring large data objects like extents in cloud-based storage systems, making it impractical for high availability and fault tolerance.
Innovation Solution
A data storage system that uses an RPC framework to initiate a data transfer by opening a port outside the framework for streaming large data objects, computes checksums for validation, and returns a private identifier for accessing the transferred extent, facilitating efficient and secure bulk data transfers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If an RPC framework is used to transfer large data objects like extents, then data transfer can be performed through a standardized interface, but the transfer becomes extremely time-consuming and memory-intensive due to serialization, deserialization, copying, and validation operations
Solution Approach 1:
The patent segments the data transfer process into two distinct parts: (1) control plane operations through RPC for metadata exchange, port allocation, and coordination; and (2) data plane operations through a separate socket connection for actual bulk data streaming. This segmentation allows each part to be optimized independently, resolving the contradiction between standardized interface and transfer speed.
Solution Approach 2:
The patent introduces an intermediary socket connection that acts as a mediator between the RPC framework and the actual data transfer. The RPC framework handles control operations while the socket connection handles data streaming, allowing both to operate in parallel without interfering with each other's performance characteristics.
2Reliability
If an RPC framework is used to transfer large data objects like extents, then data validation and integrity checking can be performed, but the process becomes extremely memory-intensive due to multiple copies and validation operations
Solution Approach 1:
The patent extracts the data validation function from the RPC framework and implements it separately at the application layer using checksums computed during the socket-based data stream transfer. This allows validation to occur without requiring the RPC framework's serialization and copying operations, significantly reducing memory consumption while maintaining reliability.
3Reliability
If traditional RPC mechanisms are used for extent transfer, then fault tolerance and high availability can be implemented through replication, but the replication process becomes impractical due to time-consuming operations
Solution Approach 1:
The patent segments replication into rapid socket-based data streaming for bulk transfer and separate RPC calls for coordination and verification. This allows multiple replicas to be created in parallel with minimal overhead, making fault tolerance practical for large data objects.
Data Source
AI summary
A system that stores sets of data blocks in extents located in storage devices is described. During operation, a receiving device receives, through an RPC framework, a first call asking to transfer an extent from a sending device to the receiving device. In response, the receiving device opens a port for a data connection that operates outside the RPC framework. The receiving device makes a second call, to the sending device through the RPC framework, asking to stream the extent to the port. The receiving device subsequently receives the extent from the sending device through the port and computes a checksum for the extent. The receiving device also receives a return from the second call, including a checksum for the extent computed by the sending device. If the computed checksum matches the received checksum, the receiving device returns the first call to indicate the transfer operation completed successfully.