Single-Source Parallel Refresh of Multiple Secondary Copies
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing copy data management systems face inefficiencies in refreshing multiple secondary copies due to serial processing, which leads to prolonged refresh times and performance impacts, especially when multiple requests compete for a single source.
Innovation Solution
Implementing a storage API that allows multiple secondary copies to be refreshed in parallel from a single source using multi-threaded operations, optimizing the read from the source and write to targets, thereby avoiding serial inefficiencies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If serial refresh operations are used to refresh multiple secondary copies from a single source, then the source is not overloaded and operations are simple to manage, but the total refresh time increases significantly and productivity decreases
Solution Approach 1:
The refresh operation is segmented into multiple independent parallel threads, each handling a different secondary copy. The system divides the single refresh workflow into N concurrent threads that can simultaneously read from the source and write to different target copies, thereby increasing throughput while maintaining source reliability through controlled concurrency.
Solution Approach 2:
The system transitions from a single-dimensional serial processing model to a multi-dimensional parallel processing model by introducing thread-level concurrency. Multiple refresh operations that were previously executed sequentially in one dimension are now executed simultaneously across multiple threading dimensions, dramatically improving refresh throughput.
2Productivity
If parallel refresh operations are implemented to improve productivity, then refresh throughput increases, but the source becomes overloaded and operation complexity increases
Solution Approach 1:
A thread management intermediary layer is introduced between the refresh requests and the actual data operations. This intermediary manages thread creation, synchronization, and termination, abstracting the complexity of parallel operations from the user while enabling high-throughput parallel refresh operations. The intermediary coordinates access to the source to prevent overload.
Solution Approach 2:
The system dynamically adjusts parallelism parameters such as thread count, batch size, and concurrency depth based on source capacity and target requirements. By changing these operational parameters, the system optimizes throughput while preventing source overload, managing complexity through adaptive parameter control rather than fixed complex configurations.
3Device complexity
If refresh requests are queued and processed sequentially to maintain simplicity, then system complexity remains low, but the time required to process multiple requests accumulates significantly
Solution Approach 1:
The system performs preliminary thread initialization and resource allocation when refresh requests are received, preparing the parallel execution environment in advance. Threads are pre-configured with source and target information, and synchronization mechanisms are established beforehand, allowing rapid parallel execution without the overhead of dynamic thread creation during processing.
Data Source
AI summary
One example method includes refreshing secondary or repurposed copies. A copy data management system issues a single request that allows multiple secondary copies to be refreshed using an API server. The server allows a storage controller to read data from a single source and then write the data read from the source to multiple targets simultaneously, thereby refreshing multiple copies in parallel.