Distributed Storage Volume Allocation via Thin-Provisioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed storage systems face challenges in maintaining data consistency and efficiency due to their distributed nature, leading to delays and storage overhead in managing large amounts of data across multiple storage nodes.
Innovation Solution
A method is introduced that involves a storage manager coordinating the allocation and reassignment of logical storage volumes across storage nodes, using thin-provisioning to scale storage gradually, and maintaining data consistency through efficient mapping and reassignment of slices, ensuring that data is written and read effectively across the network.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is distributed across multiple storage nodes, then storage capacity and scalability are improved, but data consistency and access efficiency deteriorate due to coordination overhead and network delays
Solution Approach 1:
The storage system divides data into slices that are distributed across multiple storage nodes. Each slice is independently managed and can be accessed without requiring coordination across all nodes, thereby maintaining data consistency while enabling scalable storage capacity.
Solution Approach 2:
A storage manager acts as an intermediary between applications and storage nodes, coordinating slice allocation and reassignment. The storage manager maintains metadata about slice locations and manages the distribution logic, enabling data consistency without requiring direct coordination between all storage nodes.
2Reliability
If traditional storage allocation methods are used, then data consistency is maintained, but storage overhead and access delays increase
Solution Approach 1:
The system pre-allocates storage slices to storage nodes before data is actually written. This preliminary allocation establishes the storage structure in advance, allowing data to be written directly to predetermined locations without real-time allocation decisions, thereby reducing access delays while maintaining consistency through the storage manager's coordination.
3Productivity
If storage volumes are allocated immediately, then productivity and resource utilization are improved, but system complexity and management overhead increase
Solution Approach 1:
Storage nodes autonomously manage their allocated slices without requiring complex centralized coordination for each operation. Each node independently handles read and write operations for its local slices, reducing management overhead while maintaining high resource utilization through the storage manager's initial allocation decisions.
Data Source
AI summary
A storage scheme allocates portions of a logical volume to storage nodes in excess of the capacity of the storage nodes. Slices of the storage nodes and segments of slices are allocated in response to write requests such that actual allocation on the storage nodes is only in response to usage. Segments are identified with virtual segment identifiers that are retained when segments are moved to a different storage node. Logical volumes may therefore be moved seamlessly to different storage nodes to ensure sufficient storage capacity. Data is written to new locations in segments having space and a block map tracks the last segment to which data for a given address is written. Garbage collection is performed to free segments that contain invalid data, i.e. data for addresses that have been subsequently written to.