Storage Cluster GC Scheduling for Lower Service Disruption
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing garbage collection (GC) process in distributed storage systems is inefficient and affects the performance of the storage cluster, consuming significant system resources and impacting data service performance.
Innovation Solution
A method and system for data collection that identifies storage units not in a data collection state, determines a target storage unit based on historical collection time, stops its data processing service, performs data collection, and updates its state, while stopping and recovering data processing services of other units, allowing full-load data collection and reducing performance impact.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the storage logical unit performs the GC process spontaneously, then the GC operation can be executed, but the system resources are consumed greatly and the data service performance of the storage cluster is affected
Solution Approach 1:
The patent implements dynamic scheduling of GC operations by the scheduling unit, which determines when and which storage units perform GC based on current system state and service requirements. This dynamic control allows the system to balance GC needs with service performance, preventing spontaneous GC from degrading overall system performance while ensuring GC operations are executed when appropriate.
Solution Approach 2:
The scheduling unit acts as an intermediary between the storage units and the GC process. It receives GC requests, determines the appropriate storage units to perform GC, and coordinates the execution timing. This intermediary layer prevents direct spontaneous GC execution by storage units, thereby controlling resource consumption and minimizing impact on data service performance.
2Productivity
If multiple storage units perform data collection simultaneously, then the total data collection throughput increases, but the system resources are overwhelmed and service performance degrades
Solution Approach 1:
The scheduling unit selectively activates data collection on specific storage units based on current system conditions and service requirements, rather than allowing all storage units to perform data collection simultaneously. This partial action approach ensures sufficient data collection throughput while preventing system resource exhaustion.
Solution Approach 2:
The system implements periodic data collection scheduling where the scheduling unit determines intervals and timing for data collection operations on different storage units. This periodic control prevents continuous simultaneous data collection across all units, balancing throughput generation with resource conservation.
Data Source
AI summary
Disclosed in the embodiments of the present application are a data recovery method, system and apparatus, and a computer-readable storage medium and a program product. The method comprises: in the current scheduling period, acquiring use information of a storage unit in a storage unit cluster, wherein the use information comprises a recovery state and a historical recovery time; on the basis of the recovery state, determining, from the storage unit cluster, at least one first storage unit which is not in a data recovery state; when the recovery state represents being in the data recovery state, determining, to be a second storage unit, the storage unit corresponding to the recovery state; on the basis of the historical recovery time, determining a target storage unit from the at least one first storage unit; stopping a data processing service of the target storage unit, executing a data recovery operation by means of the target storage unit, and updating the recovery state of the target storage unit; and stopping a data recovery operation of the second storage unit, and recovering the data processing service of the second storage unit. By means of the present solution, the garbage recycling efficiency and the storage cluster performance can be improved.


