Archival Thread Management for Cloud Data Transfer Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale computing and storage systems face inefficiencies in data archiving due to a high ratio of files to be moved compared to available storage controller data movement threads, leading to overloaded threads and idle resources, which reduces the overall efficiency of data transfer to cloud object storage devices.
Innovation Solution
A system and method that optimize data throughput by employing a global serialization algorithm, where archival managers assign and manage storage controller data movement threads, using threshold values to determine when to obtain exclusive locks or acquire threads intermittently, thereby serializing access and reducing thread overloading and idling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a finite number of storage controller data movement threads are used to transfer data, then the system can operate with limited processing resources, but the threads become overloaded and idle resources increase, reducing overall data transfer efficiency
Solution Approach 1:
The patent segments data files into different categories (first data type requiring thread assignment, second data type not requiring thread assignment) and processes them through different pathways. This segmentation allows the system to handle varying quantities of files efficiently without overloading the有限 number of threads, as only critical files need thread resources while less critical files can be transferred through alternative means.
Solution Approach 2:
The patent implements dynamic thread assignment where thread resources are allocated and deallocated based on real-time needs. The system dynamically determines which files require thread assignment and which don't, allowing flexible adaptation to changing data quantities and transfer priorities. This dynamic approach prevents thread overload by selectively assigning threads only when necessary.
2Productivity
If all data files are processed sequentially through limited threads, then thread overload occurs, but parallel processing is constrained by the finite number of available threads
Solution Approach 1:
The patent divides data files into distinct segments (first data type and second data type) with different processing requirements. This segmentation simplifies thread management by clearly defining which files need thread resources and which don't, reducing the complexity of managing thread assignments while maximizing throughput through parallel processing of different file types.
Solution Approach 2:
The patent introduces an intermediary mechanism (archival manager with thread assignment logic) that mediates between the data files and the limited thread resources. This intermediary layer manages thread allocation intelligently, assigning threads only when needed and using threshold values to determine assignment criteria, thereby reducing management complexity while maintaining high productivity.
3Reliability
If threshold values are used to determine thread assignment, then thread overloading is reduced, but additional processing logic is required to evaluate and manage thresholds
Solution Approach 1:
The patent uses threshold values as key parameters to dynamically determine thread assignment decisions. By changing the threshold parameters based on system state and file characteristics, the system achieves reliable thread allocation without requiring overly complex logic. The thresholds provide simple, rule-based decision-making that improves reliability while keeping the evaluation logic manageable.
Solution Approach 2:
The patent implements feedback mechanisms where the system monitors thread usage patterns and adjusts threshold values accordingly. This feedback loop allows the system to learn from previous operations and optimize thread assignment decisions, improving reliability over time. The feedback mechanism enables automatic adaptation to changing workloads without requiring manual reconfiguration or complex adaptive algorithms.
Data Source
AI summary
Embodiments relate to a system, program product, and method for optimizing the throughput of an archival application through storage volume and file selection and assignment of a finite number of storage controller data movement threads associated with a plurality of storage controllers. Data directed to be transferred from one or more storage controllers to non-cloud-based storage is transferred through standard I/O features. Data directed to be transferred from the storage controllers to cloud object storage devices is managed through an archival application that manages the finite number of storage controller data movement threads to provide efficient and effective transfer of the data.