Cloud Storage Resource Pool Management for Workload Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing cloud storage systems face a heavy workload for management nodes due to the complexity of calculating and managing metadata across storage nodes, leading to inefficient data storage, reading, and cleaning processes.
Innovation Solution
The implementation of grouping storage blocks into resource pools, where data is stored, read, or cleaned in units of resource pools, simplifies the management by determining the number of occupied storage blocks and sending cleaning instructions only when a preset condition is met, thereby reducing the workload of the management node.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the management node receives and processes metadata from each storage node individually to calculate remaining capacity and send cleaning instructions, then data management coverage is complete, but the workload and complexity of the management node increases significantly
Solution Approach 1:
The patent merges multiple storage nodes into storage node groups, where each group is managed as a single unit. The management node receives metadata from storage node groups rather than individual storage nodes, combining the management of multiple nodes into one consolidated process. This reduces the number of individual metadata processing operations while maintaining comprehensive data management coverage through the group management mechanism.
Solution Approach 2:
The patent introduces storage node groups as intermediary entities between the management node and individual storage nodes. These groups act as mediators that aggregate metadata from multiple storage nodes and present it to the management node in a consolidated form, reducing the direct processing burden on the management node while ensuring complete data management coverage.
2Reliability
If the management node processes metadata from all storage nodes to determine cleaning needs, then data cleaning coverage is thorough, but the processing time and operational efficiency decrease
Solution Approach 1:
The patent combines the processing of metadata from multiple storage nodes into a single group-level operation. Instead of evaluating each storage node separately for cleaning needs, the management node evaluates storage node groups as unified entities, maintaining thorough cleaning coverage while significantly reducing the number of individual processing operations required.
Solution Approach 2:
The patent segments the storage system into storage node groups that can be managed independently. This segmentation allows the management node to process groups in batches or prioritized sequences rather than handling all storage nodes simultaneously, improving operational efficiency while ensuring comprehensive cleaning coverage across all groups.
3Adaptability or versatility
If individual storage nodes are managed separately for data allocation and cleaning, then fine-grained control is achieved, but the system complexity and management overhead increase
Solution Approach 1:
The patent merges the management of multiple storage nodes into group-level operations while preserving the ability to manage individual nodes within each group. This hierarchical approach combines the simplicity of group-level control with the flexibility of individual node management when needed, reducing overall system complexity while maintaining fine-grained control capabilities.
Solution Approach 2:
The patent implements a dynamic management structure where storage nodes can be organized into groups for bulk operations but can also be individually accessed when specific fine-grained control is required. This dynamic approach allows the system to switch between group-level and individual-level management modes based on operational needs, balancing control precision with management simplicity.
Data Source
Figure 1~2
Figure 3~5
Figure 6~8
AI summary
Embodiment of the present application disclose data storage, data reading, and data cleaning methods and devices, and a cloud storage system. Storage blocks of storage nodes are grouped into resource pools. The data is stored, read, and cleaned in units of resource pools, thereby reducing the workload of management. In particular, during data cleaning, only the data in a resource pool is cleaned. Specifically, the method includes: for each resource pool, determining a first number of occupied storage blocks in the resource pool (S401); determining whether the first number meets a preset data cleaning condition (S402); if so, sending a data cleaning instruction to a storage node where each occupied second storage block in the resource pool is located, such that the storage node cleans the data stored on the second storage block in the storage node (S403). That is to say, the management node determines the number of occupied storage blocks of a resource pool. When the number meets a preset data cleaning condition, the data in the storage block is cleaned. Such a process is simple, reducing the workload of management node.