Cluster Volume Table for Space Consolidation in Federated Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems face challenges in scalability, as vertical capacity expansion only increases storage capacity without improving processing power, memory, or bandwidth, while horizontal expansion adds computing resources but not efficiently managing data distribution across nodes.
Innovation Solution
The method involves striping volumes across multiple storage nodes, maintaining a cluster volume table to track logical block distribution, and intelligently redistributing data based on usage ratios to optimize storage utilization, allowing for efficient addition of new nodes and handling I/O operations during space crunches.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If vertical capacity expansion is used to add disk drives to increase storage capacity, then storage capacity is improved, but processing power, main memory, and bandwidth are not improved
Solution Approach 1:
The system segments storage capacity and computing resources into separate expandable components. Storage can be expanded by adding disk drives to existing nodes, while computing resources (processing power, memory, bandwidth) can be expanded by adding new nodes to the cluster, allowing independent optimization of each resource type.
Solution Approach 2:
The patent introduces horizontal expansion as a new dimension for scaling storage systems. Instead of only adding storage vertically to existing nodes, the system allows expansion by adding new nodes to the cluster, creating a two-dimensional expansion model that simultaneously addresses storage capacity and computing resource needs.
2Quantity of substance
If horizontal capacity expansion is used to add new storage nodes, then storage capacity and computing resources are improved, but data distribution management becomes more complex
Solution Approach 1:
The data distribution mechanism is designed to be universal and automatically adapts to any number of nodes in the cluster. The same striping and redistribution algorithms work whether there are 3 nodes or 30 nodes, eliminating the need for complex node-specific management logic.
Solution Approach 2:
The system implements self-service data redistribution where the data distribution mechanism automatically detects new nodes and redistributes data without manual intervention. The load balancer and data striping algorithms autonomously manage data placement and migration across the cluster.
3Quantity of substance
If data is striped across multiple storage nodes in a horizontally federated system, then storage capacity and performance are improved, but space consolidation becomes difficult when nodes are added
Solution Approach 1:
The data distribution scheme is dynamic and automatically adapts when nodes are added or removed from the cluster. The system continuously monitors storage usage across nodes and automatically redistributes data to maintain balanced utilization, making space consolidation seamless rather than a manual operation.
Solution Approach 2:
The system implements feedback mechanisms where storage usage metrics from all nodes are continuously monitored and fed back to the data distribution controller. This feedback drives automatic redistribution decisions, ensuring data is consistently placed on nodes with available capacity and consolidating space efficiently.
Data Source
AI summary
Systems and methods for handling input/output operations during a space crunch are described herein. An example method includes striping a volume across a plurality of storage nodes, and maintaining a cluster volume table (CVT) storing information regarding distribution of the logical blocks. Additionally, the CVT includes a plurality of entries, where each of the entries includes information identifying a respective owner storage node of a respective logical block. The method also includes receiving a write I/O operation directed to an unallocated logical block owned by a landing storage node, where the landing storage node lacks free storage capacity, and locking the unallocated logical block. The method further includes updating the CVT to identify a storage node having free storage capacity as owner storage node of the unallocated logical block, and unlocking the unallocated logical block, wherein the write I/O operation proceeds at the storage node having free storage capacity.