Burst Buffer Appliance for Partitioned Key-Value Store Tiering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing partitioned key-value stores face challenges in efficiently managing large amounts of data across multiple nodes, requiring improved parallelism and storage solutions to balance storage capacity and IO throughput while maintaining performance at a lower cost.
Innovation Solution
The implementation of a burst buffer appliance that stores key-value data across multiple storage tiers, including flash and disk storage, with local and global sorting capabilities, allowing for batch IO operations and dynamic tiering to optimize storage and performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a single-node key-value store is used, then simplicity is maintained, but storage capacity and performance are insufficient for large-scale workloads
Solution Approach 1:
The key-value store is divided into multiple partitions distributed across different nodes in the cluster. Each node maintains a portion of the key-value pairs, allowing the system to scale storage capacity while keeping individual node complexity manageable. The partitioning is based on key ranges, with each node responsible for a specific subset of keys.
Solution Approach 2:
The system transitions from a single-node architecture to a multi-node distributed architecture, adding the dimension of spatial distribution. This allows the system to handle larger workloads by distributing data and operations across multiple nodes, effectively solving the scalability problem while maintaining operational simplicity through standardized node interfaces.
2Quantity of substance
If distributed key-value stores are implemented across multiple nodes, then storage capacity increases, but data transfer and processing overhead increases
Solution Approach 1:
The system implements local caching and buffering mechanisms at each node to reduce network traffic. Frequently accessed key-value pairs are kept in local memory or fast storage, minimizing the need for repeated network transfers. This local optimization reduces data transfer overhead while maintaining distributed storage capacity.
Solution Approach 2:
The system performs preliminary data localization by pre-fetching and caching data that is likely to be accessed soon. By anticipating access patterns and preparing data in advance at local nodes, the system reduces the need for frequent network transfers during actual access operations, thereby reducing data transfer overhead.
3Productivity
If flash storage tier is added to improve IO throughput, then performance increases, but storage cost increases
Solution Approach 1:
The system implements a hierarchical storage architecture where flash storage and disk storage serve different purposes. Flash storage is used for frequently accessed data and caching operations requiring high IO throughput, while disk storage handles less frequently accessed data and bulk storage requirements. This differentiated approach optimizes performance for critical operations while controlling overall storage costs.
Solution Approach 2:
Instead of using flash storage for the entire key-value store, the system applies flash storage selectively to only the portion of data that requires high-performance access. This partial application of expensive storage technology achieves the necessary IO throughput improvement for critical operations while avoiding the excessive cost of deploying flash storage across the entire storage capacity.
Data Source
AI summary
Parallel storage systems are provided with a burst buffer appliance for storage of a partitioned key-value store across a plurality of storage tiers. The burst buffer appliance is configured to store key-value data on first and second storage tiers comprising respective disjoint subsets of storage. One or more servers in the partitioned key-value store provide (i) an interface between one or more local key-value stores and one or more additional storage architectures, and/or (ii) a key value interface that provides access to at least one hardware key-value store. The local key-values are optionally sorted both locally by the local store and globally by the overall MDHIM framework. The local and global sorting can be leveraged to provide batch input/output (IO) operations that aggregate multiple requests into batch operations on one or more sorted ranges.