Parallel-Logged File System Metadata Compression via Pattern Merging
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional Parallel-Logged File Systems (PLFS) face performance issues due to increasing metadata overhead, which slows down data storage systems as data volume grows, and are not scalable to exa-flop supercomputers, leading to inefficient read and write operations.
Innovation Solution
Implementing a method to analyze checkpoint data for patterns and compress metadata indices using a discovery engine, which buffers raw indices, identifies and merges pattern structures, and stores them separately from non-pattern entries, reducing the metadata footprint and optimizing index access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional PLFS stores all metadata indices individually, then complete data tracking is achieved, but metadata size increases and performance deteriorates
Solution Approach 1:
The patent merges identical or similar metadata indices into single consolidated entries. Instead of storing each index individually, the system combines redundant indices into unified representations, reducing metadata overhead while preserving complete data tracking capability. This directly addresses the contradiction by maintaining reliability through complete tracking while improving productivity through reduced metadata processing.
Solution Approach 2:
The patent creates universal metadata entries that can represent multiple individual indices simultaneously. These consolidated metadata structures serve multiple functions: tracking individual data blocks, representing groups of identical indices, and enabling efficient compression operations. This multi-functionality allows the system to maintain complete data tracking while achieving performance improvements through reduced metadata size.
2Reliability
If traditional PLFS stores all metadata indices individually, then complete data tracking is achieved, but metadata size increases
Solution Approach 1:
The patent merges identical or similar metadata indices into single consolidated entries. Instead of storing each index individually, the system combines redundant indices into unified representations, reducing metadata overhead while preserving complete data tracking capability. This directly addresses the contradiction by maintaining reliability through complete tracking while improving productivity through reduced metadata processing.
Solution Approach 2:
The patent uses representative copying where a single consolidated metadata entry represents multiple individual indices. Rather than copying and storing each index separately, the system creates a unified metadata structure that references or represents multiple data blocks, significantly reducing the quantity of metadata stored while maintaining the ability to track all underlying data.
3Quantity of substance
If PLFS processes large datasets, then data storage capacity increases, but metadata overhead increases and scalability is limited
Solution Approach 1:
The patent merges identical or similar metadata indices into single consolidated entries. Instead of storing each index individually, the system combines redundant indices into unified representations, reducing metadata overhead while preserving complete data tracking capability. This directly addresses the contradiction by maintaining reliability through complete tracking while improving productivity through reduced metadata processing.
Solution Approach 2:
The patent changes the structural parameters of metadata from individual discrete entries to consolidated grouped entries. By transforming the metadata organization from one entry per data block to one entry representing multiple blocks, the system reduces metadata overhead complexity while maintaining the ability to scale to large data storage capacities.
4Ease of manufacture
If PLFS uses traditional metadata structure, then implementation is straightforward, but read and write operations become inefficient with large datasets
Solution Approach 1:
The patent merges identical or similar metadata indices into single consolidated entries. Instead of storing each index individually, the system combines redundant indices into unified representations, reducing metadata overhead while preserving complete data tracking capability. This directly addresses the contradiction by maintaining reliability through complete tracking while improving productivity through reduced metadata processing.
Data Source
AI summary
A computer-executable method, system, and computer program product for managing a Parallel-Logged File System (PLFS) on a data storage system, wherein the data storage system is in communication with an application that updates the data storage system with checkpoint data, the computer-executable method, system, and computer program product comprising receiving a first checkpoint including data from the application; analyzing the data in the first checkpoint to determine whether a pattern exists in the checkpoint data; and upon a positive determination, adding the pattern into PLFS.