List-Based Index for Parallel Storage Replica Management
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing parallel storage systems in high-performance computing environments face inefficiencies in storing replica copies, particularly in hierarchical storage tiering systems, where data replication methods do not effectively manage bandwidth and disk space usage.
Innovation Solution
The implementation of a list-based index to identify file replicas, which includes pointers to storage locations and checksum values for validation, allows for efficient storage and retrieval of files and their replicas across different storage tiers, reducing data processing and transfer costs while preserving disk space.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data replication is used to store the same data on multiple storage devices, then reliability and fault-tolerance are improved, but disk space consumption increases
Solution Approach 1:
The patent creates replica copies of files and stores them in a hierarchical storage tiering system with different storage types (flash memory, HDDs, tape drives). The list-based index tracks these replicas, allowing the system to maintain multiple copies for fault-tolerance while efficiently managing the disk space through intelligent placement and retrieval mechanisms.
2Speed
If storage tiering techniques are used to combine NVRAM and HDDs, then bandwidth requirements are satisfied, but system complexity increases
Solution Approach 1:
The patent segments the storage system into different hierarchical tiers (flash memory, HDDs, tape drives) with distinct characteristics. Each tier serves specific purposes: flash memory for high-speed access and bandwidth-intensive operations, HDDs for capacity, and tape drives for archival. The list-based index manages these segmented storage locations, simplifying the overall system by providing a unified view of the heterogeneous storage architecture.
3Ease of operation
If multiple copies of the same data are stored using the same type of storage, then accessibility is improved, but disk space is not preserved
Solution Approach 1:
The patent applies local quality by placing replica copies in different storage tiers based on their access patterns and requirements. Frequently accessed replicas are placed in faster storage (flash memory), while less frequently accessed replicas are placed in slower, higher-capacity storage (HDDs, tape drives). The list-based index tracks these location-specific qualities, enabling the system to maintain accessibility for critical replicas while preserving disk space through intelligent placement decisions.
Data Source
AI summary
Improved techniques are provided for storing files in a parallel computing system using a list-based index to identify file replicas. A file and at least one replica of the file are stored in one or more storage nodes of the parallel computing system. An index for the file comprises at least one list comprising a pointer to a storage location of the file and a storage location of the at least one replica of the file. The file comprises one or more of a complete file and one or more sub-files. The index may also comprise a checksum value for one or more of the file and the replica(s) of the file. The checksum value can be evaluated to validate the file and/or the file replica(s). A query can be processed using the list.