Distributed Storage Replication via Separate Metadata and Data Channels
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Legacy storage systems in cloud computing are inefficient in replicating data between remote data centers due to high latency and unreliability caused by transmitting data over client networks, which introduces significant delays and potential failures in disaster recovery and redundancy processes.
Innovation Solution
The replication process is split into two coordinated parts, where the upper storage layer transmits metadata over a client network and the lower storage layer transmits actual data via faster and more reliable internal networks like Storage Area Networks (SAN) or Fiber Channel, reducing the burden on client networks and minimizing latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is transmitted over client networks for replication between remote data centers, then network accessibility is maintained, but latency and reliability deteriorate significantly
Solution Approach 1:
The patent segments the data into two distinct types: metadata and actual data. Metadata is transmitted over the client network to maintain accessibility, while actual data is transmitted separately over dedicated storage networks. This segmentation resolves the contradiction by allowing each type of data to use the most appropriate transmission path for its specific requirements.
Solution Approach 2:
The patent introduces an intermediary mechanism where the storage system itself manages the separation and coordinated transmission of metadata and data through different networks. The storage controller acts as an intermediary that orchestrates which components go through which network paths, enabling simultaneous use of client networks and storage networks without conflict.
2Speed
If all data is transmitted over internal storage networks, then transmission speed improves, but network complexity and client accessibility worsen
Solution Approach 1:
The patent divides the network transmission architecture into two segments: client networks for metadata and storage networks for actual data. This segmentation allows the system to leverage the speed of internal storage networks for bulk data transfer while maintaining the simplicity and accessibility of client networks for control operations, thereby improving speed without proportionally increasing complexity.
Solution Approach 2:
The patent creates a multi-functional network architecture where the client network handles metadata operations and the storage network handles data operations. This universality allows each network to be optimized for its specific function, enabling high-speed data transmission through storage networks while client networks maintain their role in providing accessibility and control.
3Productivity
If metadata and data are transmitted together over the same network, then system complexity is reduced, but transmission efficiency and reliability worsen
Solution Approach 1:
The patent segments the replication process into two independent streams: metadata replication over client networks and data replication over storage networks. This segmentation improves productivity by allowing both streams to proceed simultaneously without blocking each other, while the complexity is managed through automated coordination mechanisms that track and synchronize the two streams.
Solution Approach 2:
The patent ensures continuous replication by maintaining independent transmission channels for metadata and data. This continuity allows the replication process to proceed without interruption even if one network experiences delays or issues, as the other network continues to operate, thereby improving overall productivity through parallel continuous operations.
Data Source
AI summary
Storage data and metadata in separate storage layers of a distributed storage system can be replicated via separate channels. For example, a first gateway at a first storage layer in a first data center can identify a data object for replication to a second data center. The first gateway can access metadata to determine that the data object corresponds to a set of storage data on a first storage node in the first data center. The first gateway can then transmit the metadata to a second gateway in the second data center via a first network. The first gateway can also transmit a command to the first storage node for causing the first storage node to transmit the set of storage data to a second storage node in the second data center via a second network.