Multi-threaded Deduplicated Backup for Collaboration Server Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing de-duplicated backup methods for collaboration server data, such as IBM Lotus Domino, often take a long time to complete due to the use of a single backup thread, especially for large datasets, leading to inefficiencies in data transfer and processing.
Innovation Solution
Implementing a multi-threaded backup process that divides data into size-based or volume-based subsets, allowing multiple threads to work in parallel, with each thread having its own local cache to maximize de-duplication cache hits and efficiently transfer only unique data to the remote backup server, while also backing up transaction logs in parallel.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a single backup thread is used for de-duplicated backup, then data integrity is ensured through centralized cache management, but backup time becomes excessively long for large datasets
Solution Approach 1:
The backup process is divided into multiple independent threads, each handling a separate cache and portion of the data. This segmentation allows parallel processing while maintaining data integrity through individual cache management for each thread, resolving the contradiction between centralized reliability and parallel speed.
Solution Approach 2:
The system transitions from a single-threaded sequential approach to a multi-threaded parallel approach, adding the dimension of concurrency. Each thread operates independently with its own cache, enabling simultaneous data processing that dramatically reduces backup time while preserving data integrity through structured cache management.
2Productivity
If multiple backup threads are implemented in parallel, then backup time is reduced through parallel processing, but cache management complexity increases
Solution Approach 1:
The cache is segmented into multiple independent local caches, one for each backup thread. This eliminates the need for complex inter-thread cache coordination and synchronization, as each thread manages its own cache independently. The segmentation simplifies cache management while enabling parallel processing that improves backup speed.
Solution Approach 2:
Each backup thread independently manages its own local cache without requiring coordination with other threads. This self-service approach to cache management eliminates complex synchronization mechanisms and inter-thread communication overhead, simplifying the system while maintaining high productivity through parallel operations.
3Productivity
If data is divided into multiple subsets for parallel processing, then backup efficiency is improved, but the overhead of managing multiple threads and caches increases
Solution Approach 1:
Data is segmented into multiple subsets that are processed in parallel by separate threads, each with its own cache. This segmentation improves backup efficiency by utilizing multiple processing units simultaneously. The overhead is minimized because each thread operates independently with its own cache, avoiding the need for complex data sharing and synchronization that would consume additional energy.
Solution Approach 2:
The system changes the parameter of cache organization from a single shared cache to multiple independent caches. This parameter change enables parallel processing that improves productivity while the independent cache structure reduces the overhead of cache coordination and synchronization, thereby reducing energy loss from processing overhead.
Data Source
AI summary
Techniques to backup collaboration server data are disclosed. A plurality of separate threads are used to back up a prescribed number of subsets of a dataset. Each subset of the prescribed number of subsets was determined by walking an associated directory in the prescribed order at a prior time with which a prior backup is associated. A separate thread is spawned to backup transaction logs associated with the dataset. A first pass backup of the transaction logs is performed in parallel with the backup of the prescribed number of subsets.