Data Storage System Segmentation for Low Latency HPC
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High performance computing environments face challenges in data storage architectures that require efficient data migration with low latency and fault tolerance, but existing systems incur overhead and latency due to continuous arbitration between storage controllers for 'write' I/O operations, limiting scalability.
Innovation Solution
A data storage system that dynamically divides storage devices into separate pools, assigns each storage controller to a pool for exclusive 'write' operations, and employs a multi-level selection routine to select healthy storage entities without explicit communication, ensuring data shards are written across different devices to prevent collisions and ensure data integrity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If continuous arbitration between storage controllers is implemented for write I/O operations, then data integrity is protected, but operation latency and overhead increase
Solution Approach 1:
The storage system is segmented into multiple storage controllers, each assigned to a specific storage pool. This segmentation allows independent write operations for each pool without requiring arbitration between controllers, thereby reducing latency while maintaining data integrity through dedicated pool assignment.
Solution Approach 2:
A centralized controller acts as an intermediary to allocate storage pools to compute nodes before data migration begins. This pre-allocation eliminates the need for continuous arbitration during data writes, as each compute node has a predetermined storage pool assignment that avoids controller conflicts.
2Quantity of substance
If multiple storage controllers share multiple storage devices, then storage capacity is utilized, but data placement collisions occur
Solution Approach 1:
Storage devices are segmented into separate storage pools, with each pool dedicated to a specific storage controller. This segmentation ensures that data placed by different controllers goes to different pools, eliminating data placement collisions while still allowing multiple controllers to share the overall storage infrastructure.
Solution Approach 2:
Each storage controller has exclusive write access to its assigned storage pool, creating a local quality where data placement is guaranteed to be collision-free within that pool. This local exclusivity prevents conflicts while maintaining global storage utilization across multiple controllers.
3Stability of the object's composition
If storage controllers arbitrate for write operations, then data consistency is maintained, but scalability is limited
Solution Approach 1:
The system is divided into independent storage pools, each managed by a dedicated storage controller. This segmentation allows the system to scale by simply adding more pools and controllers without introducing arbitration complexity, as each new pool operates independently with its own controller.
Solution Approach 2:
Each storage controller independently manages its assigned storage pool without requiring coordination or arbitration with other controllers. This self-service capability enables linear scalability, where adding more controllers and pools increases system capacity without increasing the complexity of inter-controller communication.
4Productivity
If data shards are placed on the same storage device, then storage efficiency is improved, but fault tolerance is reduced
Solution Approach 1:
Before data migration, the system pre-determines the target storage pool for each compute node based on its assigned controller. This preliminary action ensures that data shards are distributed across different storage devices within the same pool, achieving both storage efficiency and fault tolerance by avoiding concentration of data on single devices.
Solution Approach 2:
Within each storage pool, data is distributed across multiple storage devices with exclusive write access to the pool's controller. This local quality ensures that even though data shards are placed on different devices for fault tolerance, the unified pool management maintains storage efficiency through coordinated access patterns.
Data Source
AI summary
A data migration system supports a low-latency and reduced overhead data storage protocol for data storage sharing in a non-collision fashion which does not require inter-communication and permanent arbitration between data storage controllers to decide on the data placement/routing. The multiple data fragments of data sets are prevented from routing to the same storage devices by a multi-step selection protocol which selects (in a first phase of the selection routine) a healthy highest ranked drive enclosure, and further selects (in a second phase of the selection routine) a healthy highest-ranked data storage controller residing in the selected drive enclosure, for routing data fragments to different storage pools assigned to the selected data storage devices for exclusive “writing” and data modification. The selection protocol also contemplates various failure scenarios in a data placement collision free manner.