Scale-out Data Backup with Double Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data backup systems face challenges in scalability, particularly with the scale-up method, which limits performance and increases storage costs due to reliance on a single controller and lack of effective multi-task management, making it difficult to handle growing data requirements effectively.
Innovation Solution
A scale-out data backup architecture using ZooKeeper and Ceph, with a double deduplication mechanism that distributes deduplication tasks across multiple computing nodes and performs secondary deduplication at the management node to avoid redundant storage and optimize storage space, allowing for easier expansion and management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If scale-up method is used with single controller, then system simplicity is maintained, but scalability and performance are limited
Solution Approach 1:
The backup system is segmented into multiple independent computing nodes, each capable of handling deduplication tasks autonomously. This segmentation enables the system to scale horizontally by adding more nodes without requiring a complete system redesign, thus improving scalability while managing complexity through modular architecture
Solution Approach 2:
The system transitions from vertical scaling (scale-up with single controller) to horizontal scaling (scale-out with multiple nodes). This dimensional change allows the system to expand capacity by adding more computing nodes across the network, bypassing the performance limitations of single-controller architectures
2Productivity
If deduplication tasks are centralized, then coordination is simplified, but processing speed and efficiency decrease
Solution Approach 1:
Deduplication tasks are segmented and distributed across multiple computing nodes rather than being centralized. Each node independently processes data segments and performs deduplication operations, significantly increasing overall processing speed while the index-based coordination mechanism manages the distributed operations efficiently
Solution Approach 2:
An index structure serves as an intermediary mechanism that enables coordination between distributed computing nodes without requiring complex inter-node communication. The index allows nodes to independently determine whether data segments have been previously stored, simplifying the coordination of distributed deduplication tasks
3Quantity of substance
If data is stored without deduplication, then storage speed is faster, but storage space is wasted
Solution Approach 1:
The system performs preliminary deduplication operations by computing hashes of data segments before final storage. This preliminary action identifies duplicate segments that can be referenced rather than stored again, optimizing storage space utilization while the incremental nature of this process maintains acceptable backup speeds
Solution Approach 2:
Instead of storing duplicate data segments, the system creates references (copies of metadata) to the original segments through hashing. This copying approach dramatically reduces storage space requirements while maintaining data integrity and retrieval capability, with the hash computation overhead being minimal compared to full data processing
Data Source
AI summary
Embodiments of the present disclosure relate to method, device and computer program product for data backup. The method comprises: in response to receiving from a backup server a data stream to be backed up, dividing the data stream into a plurality of data segments; distributing the plurality of data segments to at least one computing node; in response to receiving an index of a corresponding data segment from a first computing node of the at least one computing node, looking up the index in a global index cache, the index being generated by the first computing node to uniquely identify the data segment, the global index cache storing indexes of data in a backup storage device; in response to the missing index in the global index cache, adding the index into the global index cache; and sending to the first computing node an indication to store the data segment in the backup storage device.