Burst Buffer Appliance for Cluster File System Storage Tiering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional cluster file systems, such as Lustre, face challenges in balancing storage capacity and IO throughput, leading to suboptimal performance or excessive costs due to mismatched storage devices and IO operations.
Innovation Solution
Implementing a burst buffer appliance with storage tiering control functionality that transparently includes a flash storage tier, allowing dynamic balancing of storage capacity and IO throughput requirements without significant changes to clients, object storage servers, or metadata servers, by controlling data movement between different storage tiers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional Lustre implementations use object storage servers with back-end storage arrays directly, then storage capacity can be provided, but IO throughput performance becomes suboptimal due to mismatched storage devices
Solution Approach 1:
The patent segments the storage system into multiple storage tiers (first storage tier with object storage servers, second storage tier with additional storage resources) that can be independently configured and managed. This segmentation allows each tier to be optimized for specific storage needs while working together to provide both capacity and performance.
Solution Approach 2:
The patent introduces a storage tiering controller as an intermediary component that coordinates data movement between storage tiers and manages storage operations. This mediator enables the system to achieve optimal IO throughput by directing operations to appropriate storage tiers without requiring significant changes to existing object storage servers or clients.
2Quantity of substance
If storage capacity is increased using additional storage resources, then more data can be stored, but IO throughput performance may deteriorate due to storage device mismatches
Solution Approach 1:
The patent implements dynamic storage tiering where data can be moved between storage tiers based on access patterns and performance requirements. The storage tiering controller dynamically manages data placement and movement, allowing the system to adapt to changing storage demands and maintain optimal IO throughput as storage capacity scales.
Solution Approach 2:
The patent applies local quality by assigning different storage resources to different storage tiers with optimized characteristics for specific workloads. The first storage tier uses object storage servers for general storage, while the second storage tier provides additional capacity with coordinated access management, ensuring that each layer contributes its optimal performance characteristics.
3Speed
If a flash storage tier is included in the cluster file system, then IO operations are accelerated, but the system requires significant changes to clients, object storage servers, metadata servers or applications
Solution Approach 1:
The storage tiering controller serves as an intermediary that transparently manages the flash storage tier, handling data movement and access coordination without requiring modifications to clients, object storage servers, or metadata servers. This intermediary approach enables IO acceleration while maintaining system simplicity.
Solution Approach 2:
The patent designs the storage tiering controller to provide multiple functions: it manages the flash storage tier, coordinates data movement between tiers, and interfaces with existing Lustre components without requiring them to change. This multi-functional design achieves performance improvement without increasing overall system complexity.
4Productivity
If storage devices are well-matched to current system needs, then IO throughput is optimized, but storage capacity may be insufficient for future growth
Solution Approach 1:
The patent implements a dynamic storage tiering system that can adapt to growing storage needs. The storage tiering controller enables seamless expansion by adding resources to the second storage tier and automatically managing data distribution and movement, allowing the system to maintain optimal IO throughput while scaling storage capacity.
Data Source
AI summary
A cluster file system comprises a burst buffer appliance coupled to a plurality of object storage servers via a network. The burst buffer appliance comprises a controller configured to implement storage tiering control functionality for at least first and second storage tiers comprising respective disjoint subsets of the plurality of object storage servers. The burst buffer appliance is thereby configured to control movement of data between the first and second storage tiers. The object storage servers in the first storage tier may be configured to interface with object storage targets of a first type and the object storage servers in the second storage tier may be configured to interface with object storage targets of a second type different than the first type. For example, the object storage targets of the first type may comprise non-volatile electronic storage devices such as flash storage devices, and the object storage targets of the second type may comprise disk storage devices.