Burst Buffer Parity Group Migration via Distributed Hash Table
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage architectures for high-performance computing lack efficient methods for migrating Parity Group structures between high-performance compute nodes and long-term storage, particularly in handling unstructured data and ensuring fault tolerance and redundancy.
Innovation Solution
A data migration system utilizing a Burst Buffer tier with Non-Volatile Memory (NVM) arrays and a Distributed Hash Table, where Parity Group structures are ingested in an unstructured manner and reordered for structured long-term storage, using Parity Group Information descriptors for tracking and fault tolerance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored directly from high-performance compute nodes to long-term storage, then data migration speed is limited by storage bottlenecks, but system complexity increases and fault tolerance is reduced
Solution Approach 1:
The patent introduces a Burst Buffer tier as an intermediary storage layer between high-performance compute nodes and the Parallel File System. This Burst Buffer tier absorbs write bursts from compute nodes and smooths data flow to the underlying file system, eliminating storage bottlenecks without requiring complex architecture changes. The Burst Buffer tier acts as a buffer that decouples the high-speed compute side from the slower storage side, improving data migration speed while maintaining manageable system complexity.
2Reliability
If Parity Group structures are maintained with strict ordering and structure, then data integrity is ensured, but data ingestion speed decreases
Solution Approach 1:
The patent segments the storage system into distinct tiers with different requirements: the Burst Buffer tier accepts unstructured, high-speed data ingestion without enforcing ordering or structure, while the Parallel File System tier maintains structured Parity Group formats for long-term storage. This segmentation allows each tier to optimize for its specific function - the Burst Buffer for speed and the file system for integrity - resolving the contradiction between data integrity and ingestion speed.
Solution Approach 2:
The system performs preliminary data ingestion into the Burst Buffer tier without immediately organizing data into structured Parity Groups. Data is first captured in the Burst Buffer in an unstructured manner at high speed, and only later is it reorganized into structured Parity Group formats when written to the Parallel File System. This preliminary unstructured storage action enables fast data capture while maintaining the option for structured organization later.
3Productivity
If an intermediate storage tier is introduced to enable fast unstructured data ingestion, then data ingestion speed improves, but metadata management complexity increases
Solution Approach 1:
The patent extracts metadata management responsibilities from the Burst Buffer tier and places them in the Parallel File System tier. The Burst Buffer tier focuses solely on fast data ingestion and stores only minimal information needed for immediate operations. Complex metadata including Parity Group Information descriptors, data location tracking, and structured information are managed exclusively by the Parallel File System, which is designed to handle such complexity. This extraction of metadata management to the appropriate tier enables fast unstructured data ingestion in the Burst Buffer without burdening it with complex metadata management.
4Reliability
If distributed Parity Group Information descriptors are used for tracking data locations, then fault tolerance and data reconstruction capability improve, but system overhead increases
Solution Approach 1:
The patent implements local Parity Group Information descriptors at each Burst Buffer node that contain only the specific metadata needed for that node's data blocks, rather than maintaining complete global metadata at every node. Each descriptor is tailored to the local context, containing information about data block locations, parity information, and reconstruction capabilities specific to that node's responsibilities. This local optimization reduces the quantity of metadata overhead compared to a fully distributed global descriptor approach, while still providing the necessary fault tolerance and data reconstruction capability through the coordinated system of local descriptors.
Data Source
AI summary
The present invention is directed to data migration, and particularly, Parity Group migration, between high performance data generating entities and data storage structure in which distributed NVM arrays are used as a single intermediate logical storage which requires a global registry/addressing capability that facilitates the storage and retrieval of the locality information (metadata) for any given fragment of unstructured data and where Parity Group Identifier and Parity Group Information (PGI) descriptors for the Parity Groups' members tracking, are created and distributed in the intermediate distributed NVM arrays as a part of the non-deterministic data addressing system to ensure coherency and fault tolerance for the data and the metadata. The PGI descriptors act as collection points for state describing the residency and replay status of members of the Parity Groups.