Restore Secondary Data Using Thread Pooling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing information management systems face challenges in efficiently handling a large number of concurrent restore requests due to limitations in processing capacity, leading to inefficiencies and resource underutilization, especially when adding more media agents is not practical.
Innovation Solution
Implementing thread pooling for media agents, where a pool of restore threads is created to handle multiple requests concurrently, and using look-ahead threads to optimize data retrieval by consolidating reads for files, allowing dynamic adjustment based on performance metrics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a restore thread is created for each restore request, then each request can be handled independently, but the number of threads increases quickly when many requests are received within a short period, exceeding processing capacity
Solution Approach 1:
Multiple restore requests are merged and assigned to a single restore thread within a thread pool. The thread pool acts as a container that holds multiple requests and processes them sequentially or in parallel depending on availability, thereby reducing the total number of threads needed while maintaining the ability to handle multiple concurrent requests.
Solution Approach 2:
A single restore thread in the pool can serve multiple clients and handle multiple restore requests sequentially. This universal approach allows one thread to perform the work of multiple dedicated threads, reducing overall thread count while maintaining high request handling capacity through shared resources.
2Productivity
If another media agent is added to scale the system, then more concurrent restore requests can be handled, but resource utilization becomes inefficient and it may not be practical due to lack of resources
Solution Approach 1:
The thread pool size is made dynamic and can be adjusted based on the actual number of concurrent restore requests. When requests are few, the pool uses fewer threads; when requests increase, the pool can expand or reuse existing threads. This dynamic adjustment eliminates the need to provision for maximum capacity always, improving resource efficiency while maintaining scalability.
Solution Approach 2:
The system changes the parameter of thread pool size rather than adding more media agents. By adjusting the number of threads in the pool based on load conditions, the system achieves scalability without the practical and efficiency issues of adding physical media agents, thereby improving ease of manufacture and resource utilization.
3Productivity
If data is retrieved from secondary storage devices for multiple files, then restore operations can be performed, but the same file may be accessed multiple times, reducing retrieval efficiency
Solution Approach 1:
The look-ahead thread performs preliminary actions by reading files in advance and caching them in memory before they are actually needed for restore operations. This allows the main restore threads to retrieve data faster from the cache rather than accessing the secondary storage device repeatedly, thereby reducing file access time and improving retrieval efficiency.
Solution Approach 2:
A caching mechanism acts as an intermediary between the secondary storage device and the restore threads. Instead of restore threads directly accessing the storage device for every file read, they first query the cache. The cache serves as a mediator that stores frequently accessed files, reducing redundant storage accesses and improving overall data retrieval speed.
Data Source
AI summary
A system according to certain aspects may include a secondary storage controller computer configured to: in response to a first instruction to obtain a first secondary copy of a first data set from a secondary storage device(s), the first instruction associated with a first restore operation: instantiate a first restore thread on a processor of the secondary storage controller computer; using the first restore thread, retrieve the first secondary copy from the secondary storage device(s); and forward the retrieved first secondary copy to a primary storage subsystem for storage; and in response to a second instruction to obtain a second secondary copy of a second data set from the secondary storage device(s), the second instruction associated with a second restore operation: using the first restore thread, retrieve the second secondary copy from the secondary storage device(s); and forward the retrieved second secondary copy to the primary storage subsystem for storage.