Cluster Data Replication via Remote Storage Snapshots
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data replication methods in clusters for search engines result in delays and resource consumption during query processing and data movement, as direct data transfer between nodes is required, leading to inefficiencies in data transfer and resource usage.
Innovation Solution
Implementing a system where data snapshots are stored remotely, allowing nodes to download copies from the remote storage instead of each other, thereby reducing direct data transfer and resource consumption within the cluster.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If direct data transfer between nodes is used for replication, then data consistency is maintained, but processing delays and resource consumption increase
Solution Approach 1:
The patent introduces a remote storage system as an intermediary component that stores data snapshots. Instead of direct peer-to-peer data transfer between cluster nodes, nodes retrieve data from the remote storage intermediary, which reduces network congestion and processing delays while maintaining data consistency through snapshot-based replication.
Solution Approach 2:
The system performs preliminary actions by creating and storing data snapshots in remote storage before they are needed for replication. This advance preparation allows nodes to quickly retrieve data without waiting for real-time transfers from other nodes, thereby reducing processing delays while ensuring data consistency.
2Reliability
If direct data transfer between nodes is used for replication, then data is replicated accurately, but resource consumption (CPU, memory, I/O, network) increases
Solution Approach 1:
The remote storage system acts as an intermediary that handles data storage and retrieval operations. This eliminates the need for source nodes to expend CPU, memory, I/O, and network resources continuously for data transfer, as the remote storage independently manages snapshot storage and serves data requests, thereby reducing overall resource consumption while maintaining replication accuracy.
Solution Approach 2:
The system creates copies of data in the form of snapshots stored in remote storage. These snapshots serve as standalone copies that can be retrieved without accessing the original source nodes, reducing the resource burden on source nodes while ensuring accurate data replication through the snapshot copy mechanism.
3Speed
If data is copied from source node to target node directly, then replication speed may be fast for small datasets, but scalability is limited
Solution Approach 1:
The remote storage system provides a universal data service that all cluster nodes can access. This multi-functional architecture allows the same remote storage infrastructure to serve multiple nodes simultaneously, enabling the system to scale to larger cluster sizes without requiring proportional increases in direct peer-to-peer connectivity, thus improving scalability while maintaining replication speed.
Data Source
AI summary
Provided are methods and systems for peer recovery from remote storage. An example method includes storing, to a remote storage, a data snapshot of a plurality of nodes of a cluster, determining, by the cluster, that a piece of data stored on a first node of the cluster needs to be copied to a second node of the cluster, and in response to the determination, causing, by the cluster, the second node to download a copy of the piece of data from the data snapshot in the remote storage, determining, by the cluster, that the piece of data stored on the first node differs from the copy of the piece of data stored on the data snapshot, and in response to the determination, causing copying of the piece of data directly from the first node to the second node.