Sub-File Dictionary Compression for Lower Memory Overhead
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data compression algorithms, such as LZ4 and Snappy, struggle to meet the increasing demands of massive data production and storage due to exponential data volume growth, while public dictionary compression leads to memory occupation issues due to long-tail problems where local data files reference a global dictionary, causing performance deterioration.
Innovation Solution
Implementing a data compression method where a single persistent file maintains a dictionary affinity-bound to a sub-file, ensuring synchronized life cycles, and using pre-allocated data and metadata areas to reduce metadata expansion and improve performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If public dictionary compression is used to increase data reduction ratios, then compression ratio is improved, but memory occupation increases sharply due to the public dictionary occupying memory for a long time
Solution Approach 1:
The patent divides the global public dictionary into multiple sub-dictionaries, each associated with a specific data file. This segmentation allows each sub-dictionary to be independently managed and released when its corresponding data file is deleted, preventing the long-tail memory occupation problem while maintaining compression effectiveness.
Solution Approach 2:
The patent implements dynamic dictionary management where sub-dictionaries are created, updated, and released based on the lifecycle of corresponding data files. This dynamic approach allows the system to adapt memory usage to actual needs, releasing dictionary memory when no longer required while maintaining compression ratios when data is active.
2Loss of substance
If a global public dictionary is used for multiple data files, then compression ratio is improved, but system performance deteriorates due to frequent swap-in and swap-out operations
Solution Approach 1:
By segmenting the global dictionary into file-specific sub-dictionaries, the patent enables the storage system to load only the sub-dictionary corresponding to the currently accessed data file into memory. This eliminates frequent swap-in and swap-out operations for unrelated dictionary data, improving system performance while maintaining compression effectiveness.
Solution Approach 2:
The patent applies local quality by making each sub-dictionary specialized for its corresponding data file rather than using a general global dictionary. This localization improves cache hit rates and reduces memory swap operations since each sub-dictionary contains only the relevant compression patterns for its associated file.
3Quantity of substance
If data volume increases exponentially, then storage capacity requirements increase, but existing compression algorithms become insufficient
Solution Approach 1:
The patent implements a dynamic sub-dictionary system that can adapt to different data files and their characteristics. As data volume increases, the system dynamically creates and manages sub-dictionaries for each file, maintaining compression effectiveness without being constrained by the limitations of fixed global dictionaries or traditional compression algorithms.
Data Source
AI summary
This application provides a data compression method, including: obtaining a to-be-persisted data file, where the to-be-persisted data file includes a plurality of sub-files; and during persistence of a first sub-file in the plurality of sub-files, generating, based on the first sub-file, a dictionary corresponding to the first sub-file, and compressing the first sub-file by using the dictionary, to obtain a first compressed file; generating a first persistent file based on the first compressed file and the dictionary; and writing the first persistent file into a persistent storage medium. A single persistent file independently maintains a dictionary corresponding to a sub-file. The dictionary and the sub-file may implement same life cycle management.