Parallel Data Chunk Compression in Log-Structured File Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In high-performance computing environments, existing parallel storage systems face inefficiencies in compressing and decompressing large datasets due to the lack of parallel techniques for compressing data chunks being written to shared objects, leading to increased resource usage and transmission requirements.
Innovation Solution
Implementing a Log-Structured File System (LSFS) that compresses data chunks on compute nodes or burst buffer nodes before writing them to storage nodes, and decompresses them on read operations, leveraging the parallelism of concurrent writes and high interconnect speeds in supercomputer networks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If compression is applied after data is sent to storage system using pre-defined file system block sizes, then storage space is reduced, but transmission time and computational overhead increase due to sequential processing of large datasets
Solution Approach 1:
The patent divides the shared data object into multiple data chunks that can be processed in parallel. Each compute node works on separate chunks simultaneously, enabling parallel compression operations. This segmentation allows the system to reduce transmission time while maintaining storage space efficiency through distributed parallel processing rather than sequential compression of entire datasets.
2Quantity of substance
If compression is applied to entire shared data objects using pre-defined block sizes, then storage efficiency improves, but device complexity and resource usage increase due to sequential processing requirements
Solution Approach 1:
The data object is segmented into chunks that can be independently compressed in parallel across multiple compute nodes. This reduces processing complexity by allowing independent compression operations rather than requiring complex sequential processing of entire datasets, while still achieving storage efficiency through compression of all chunks.
Solution Approach 2:
Each compute node independently performs compression on its assigned data chunk without requiring centralized coordination for the compression operation itself. The nodes autonomously compress their local chunks and store results, reducing overall system complexity while maintaining storage efficiency.
3Productivity
If parallel compression of data chunks is implemented, then transmission efficiency and storage resource usage improve, but requires coordination overhead for managing parallel operations across distributed nodes
Solution Approach 1:
By segmenting data into independent chunks assigned to different compute nodes, the system enables parallel processing that improves transmission efficiency. The coordination overhead is minimized because each node operates independently on its chunk without requiring continuous communication during compression operations.
Data Source
AI summary
Techniques are provided for parallel compression of data chunks being written to a shared object. A client executing on a compute node or a burst buffer node in a parallel computing system stores a data chunk generated by the parallel computing system to a shared data object on a storage node by compressing the data chunk; and providing the data compressed data chunk to the storage node that stores the shared object. The client and storage node may employ Log-Structured File techniques. The compressed data chunk can be de-compressed by the client when the data chunk is read. A storage node stores a data chunk as part of a shared object by receiving a compressed version of the data chunk from a compute node; and storing the compressed version of the data chunk to the shared data object on the storage node.