Virtual Shards for Distributed Storage Metadata Management
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High Performance Computing (HPC) environments face challenges in efficiently storing and managing data and metadata across sharded storage arrays, particularly in large-scale distributed systems where existing techniques struggle with complexity and performance.
Innovation Solution
The implementation of a method that uses addressable virtual shards within sharded distributed data storage systems to store data and metadata, with specific shards dedicated to storing checksum values, allowing for efficient metadata retrieval and storage across multiple storage nodes, utilizing an I/O Dispatcher to manage the burst buffer tier and replace traditional storage interfaces.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If data and metadata are stored together in the same shard, then storage simplicity is improved, but metadata retrieval efficiency deteriorates due to increased complexity in managing large data sets
Solution Approach 1:
The patent divides each physical shard into multiple virtual shards, where different virtual shards store different types of data (e.g., data virtual shards for actual data, metadata virtual shards for metadata, checksum virtual shards for integrity verification). This segmentation allows metadata to be stored separately from large data sets, improving retrieval efficiency while maintaining storage organization.
2Productivity
If shards are created on a large number of storage devices to parallelize I/O operations, then I/O performance is improved, but system complexity deteriorates due to distributed metadata management
Solution Approach 1:
The patent creates a unified virtual shard namespace that works across all physical storage devices in the distributed system. The virtual shard abstraction layer provides consistent data structures and access methods regardless of the underlying physical distribution, allowing the same virtual shard interface to be used across multiple storage devices without increasing application-level complexity.
Solution Approach 2:
The virtual shard layer acts as an intermediary between the application layer and the physical distributed storage system. It translates high-level virtual shard operations into low-level physical storage operations, managing the complexity of distributed metadata storage and retrieval while presenting a simplified interface to applications.
3Adaptability or versatility
If traditional storage interfaces are used in sharded storage arrays, then compatibility is improved, but performance deteriorates due to interface overhead in large-scale distributed systems
Solution Approach 1:
The patent replaces traditional block-based storage interfaces with a file system-based interface that operates on virtual shards. This substitution eliminates the overhead of traditional storage interfaces by using higher-level file system abstractions that are more efficient for distributed environments, while maintaining compatibility through standard file system protocols.
Data Source
AI summary
Techniques are provided for storing data and metadata on sharded storage arrays. In one embodiment, data is processed in a sharded distributed data storage system that stores data in a plurality of shards on one or more storage nodes by providing a plurality of addressable virtual shards within each of the shards, wherein at least a first one of the addressable virtual shards stores the data, and wherein at least a second one of the addressable virtual shards stores the metadata related to the data; obtaining the data from a compute node; and providing the data and the metadata related to the data stored to the sharded distributed data storage system for storage in the respective first and second addressable virtual shards. The metadata related to the data is stored together at a portion of a corresponding stripe for the data in the second one of the addressable virtual shards. A third one of the addressable virtual shards optionally stores a checksum value related to the data.