Snapshot Data Replication in Distributed Analytic Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional distributed computing systems incur high costs in terms of network bandwidth and processing power due to the need for extensive data replication, which is necessary for reliability but inefficient in large data sets.
Innovation Solution
Implementing snapshot replication in distributed analytic computing systems, where data nodes access snapshots as independent virtual volumes rather than physical copies, reducing the need for server-to-server data replication and leveraging an external storage array to manage and create these snapshots.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If server-to-server data replication is performed to ensure data reliability, then data reliability is improved, but network bandwidth consumption and processing power requirements increase significantly
Solution Approach 1:
The patent uses storage array snapshots instead of traditional server-to-server replication. The storage array creates point-in-time copies of data blocks and distributes these snapshot references to multiple data nodes. This approach maintains data reliability through redundancy while avoiding the need to physically copy large amounts of data across the network, thereby significantly reducing network bandwidth consumption.
Solution Approach 2:
The storage array acts as an intermediary between the master node and data nodes. Instead of data nodes directly replicating data from each other (server-to-server replication), the storage array serves as a central mediator that provides snapshot data to multiple nodes simultaneously. This eliminates the need for multiple data transfer paths and reduces overall network bandwidth consumption.
2Reliability
If multiple copies of data are maintained across data nodes to ensure reliability, then data reliability is improved, but processing power and computer bus bandwidth are consumed during data transfers
Solution Approach 1:
The system creates snapshot copies at the storage array level rather than copying data through processor memory buffers. The snapshots are block-level copies that can be rapidly created and distributed without involving the CPU-intensive data transfer processes of traditional replication, thereby reducing processing power consumption.
Solution Approach 2:
The patent replaces the mechanical data transfer process (reading data from source node, transferring through network, writing to destination node) with a storage-array-managed snapshot distribution mechanism. The storage array directly provides snapshot data to multiple nodes without requiring intermediate processing through CPU and memory systems, eliminating the processing overhead associated with conventional replication.
3Reliability
If conventional server-to-server replication is used to create data copies, then data reliability is improved, but the time required for data replication increases
Solution Approach 1:
The storage array pre-creates snapshots of the data before distribution to data nodes. These snapshots are prepared in advance and can be rapidly mounted by multiple nodes simultaneously. This preliminary action eliminates the need for time-consuming data transfer and processing during runtime, significantly reducing the overall replication time while maintaining data reliability.
Solution Approach 2:
The storage array creates block-level snapshot copies that can be rapidly distributed to multiple data nodes. Unlike traditional file-level or application-level replication, block-level snapshots can be created and distributed in parallel, dramatically reducing the time required to provide redundant data copies across the distributed system.
Data Source
AI summary
A computer program product having a computer readable medium tangibly recording computer program logic for performing analytics on data at a data node, the computer program product including code to instruct a storage array to create a snapshot of the data, code to access the snapshot, by the data node, as an independent virtual volume, code to receive, at the data node, a command mapping a processing task to the data node, in which the processing task includes analysis on the data, and code to perform the processing task on the data by accessing the data through the snapshot.