Shared Memory Burst Buffer System for Data Consistency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional burst buffer systems lack data consistency and redundancy, leading to irrecoverable data loss in case of failures, which is a concern for the data storage industry aiming to minimize complexity and cost while ensuring efficient data management.
Innovation Solution
Implementing a data storage system with multiple burst buffer nodes that provide shared memory, enabling data redundancy and consistency through I/O Forwarder Servers, data management modules, and fast non-volatile storage, allowing communication between nodes for data synchronization and recovery.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional burst buffer systems are used, then implementation complexity is reduced, but data consistency and reliability deteriorate leading to irrecoverable data loss
Solution Approach 1:
The system divides the burst buffer functionality into multiple independent nodes, each maintaining its own storage and processing capabilities. This segmentation allows data to be distributed across nodes with redundancy, ensuring that if one node fails, data can be recovered from other nodes. The segmentation principle directly addresses the reliability issue by creating fault-tolerant distributed storage while keeping individual node complexity manageable.
Solution Approach 2:
Multiple burst buffer nodes are merged into a unified system that presents a single shared memory namespace to applications. The I/O Forwarder Server combines data from multiple nodes and manages consistency across them. This merging approach maintains reliability through redundancy while abstracting the complexity away from applications, as they see a unified interface rather than multiple distributed nodes.
2Reliability
If data redundancy is implemented across multiple nodes, then data recovery capability is improved, but system complexity increases
Solution Approach 1:
The I/O Forwarder Server acts as an intermediary between applications and the distributed burst buffer nodes. It receives I/O requests from applications, determines which nodes store the required data, and aggregates data from multiple nodes before returning it to the application. This intermediary approach enables data redundancy and recovery across nodes while hiding the communication complexity from applications, as they interact only with the forwarder server.
Solution Approach 2:
The I/O Forwarder Server performs multiple functions: it manages I/O requests, coordinates data retrieval from multiple nodes, ensures data consistency, and handles failure recovery. By consolidating these diverse functions into a single universal component, the system achieves robust data recovery capability without requiring applications to implement complex node-to-node communication logic.
3Productivity
If shared memory is provided by multiple burst buffer nodes, then data accessibility is improved, but coordination overhead increases
Solution Approach 1:
The system creates a virtual copy of the shared memory namespace at each burst buffer node, allowing multiple nodes to serve identical data simultaneously. When data is written to shared memory, the I/O Forwarder Server replicates it to relevant nodes. This copying approach improves data accessibility by allowing any node to serve requests without complex real-time coordination, while maintaining consistency through controlled replication rather than continuous synchronization.
Data Source
AI summary
A computer-executable method, computer program product, and system for managing I/Os from an Application on a host, wherein the host is in communication with a data storage system including one or more burst buffer servers, the computer-executable method, computer program product, and system comprising providing a portion of shared memory using the one or more burst buffer servers of the data storage system, wherein the portion of shared memory is enabled to be visible from each of the one or more burst buffer servers, receiving, at a first burst buffer server of the one or more burst buffer servers, a data I/O request from the application on the host, wherein the data I/O request is associated with data on the portion of shared memory, determining whether data associated with the data I/O request is located on the first buffer server, and processing the data I/O request based on the determination.