Cache Pool Queueing for Priority Writes in Distributed Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed storage systems face inefficiencies due to the use of ordinary SATA disks for data storage, leading to insufficient performance, while using SSDs for long-term storage is costly, and there is a need to balance performance and cost effectively.
Innovation Solution
A method for data processing that determines priority levels for placement groups in a cache pool, constructs a target queue based on priority, and iteratively traverses and writes objects from high-priority groups to a backend pool, optimizing disk pressure balance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If SSDs are used for cache pool to store frequently accessed data, then data access performance is improved, but storage cost increases
Solution Approach 1:
The storage system is segmented into two distinct pools: a cache pool using SSDs for frequently accessed data and a backend pool using SATA disks for infrequently accessed data. This segmentation allows the system to leverage the high speed of SSDs for hot data while using cost-effective SATA disks for cold data, thereby resolving the contradiction between performance and cost.
Solution Approach 2:
Different storage media are assigned to different locations based on data access characteristics. The cache pool location uses SSDs for data that requires high access speed, while the backend pool location uses SATA disks for data that can tolerate lower access speed. This local quality assignment optimizes both performance and cost by matching storage media characteristics to data requirements.
2Productivity
If placement groups are processed in priority order, then writing efficiency is improved, but disk pressure distribution becomes unbalanced
Solution Approach 1:
Placement groups are segmented into multiple waiting queues based on their priority levels. Each waiting queue processes placement groups of the same priority together, which allows the system to maintain priority-based writing efficiency while distributing the processing load across multiple queues. This prevents any single disk from being overwhelmed by high-priority writes alone.
Solution Approach 2:
The system dynamically manages placement group processing by iteratively traversing waiting queues and selectively retrieving placement groups based on priority. This dynamic approach allows the system to adapt to varying write patterns and balance disk pressure in real-time while maintaining overall writing efficiency through priority-based processing.
Data Source
AI summary
Data processing method and apparatus, an electronic device, and a storage medium are disclosed, which is in the fields of artificial intelligence, such as distributed storage and cloud computing. The method includes: determining a priority of each placement group in a cache pool respectively, and dividing placement groups with the same priority into a same waiting queue; constructing a target queue which is initially empty, and in response to determining that a supplementary trigger condition is met, determining placement groups to be retrieved based on the principle that a placement group in a waiting queue with higher priority is retrieved first, retrieving the placement groups to be retrieved from the corresponding waiting queue and adding the placement groups to be retrieved to the target queue; and in response to determining that the target queue is not empty, iteratively traversing each placement group in the target queue, wherein when traversing each placement group, the placement group is used as a target placement group respectively, and the number of writable objects is determined as a first quantity, and the first quantity of objects retrieved from the target placement group is written to a backend pool.