Sub-File Dictionary Compression to Cut Storage Memory Occupancy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data compression algorithms, such as LZ4 and Snappy, struggle to meet the increasing demands of massive data production and storage due to exponential data volume growth, while public dictionary compression leads to memory occupancy issues due to long-tail problems caused by life cycle differences between local data files and global dictionaries, resulting in performance deterioration.
Innovation Solution
A data compression method where a dictionary is affinity-bound to a sub-file, with both sharing the same life cycle, and is stored in a pre-allocated metadata area within a persistent file, ensuring efficient space reservation and management, reducing metadata overflow, and improving system performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If public dictionary compression is used to increase data reduction ratios, then compression ratio is improved, but memory occupancy increases sharply due to the public dictionary occupying memory for a long time
Solution Approach 1:
The patent divides the global public dictionary into multiple sub-dictionaries, each associated with a specific data file. This segmentation allows each sub-dictionary to be independently managed and released when its corresponding data file is deleted, preventing the long-tail memory occupancy problem while maintaining compression effectiveness.
Solution Approach 2:
The patent implements dynamic dictionary management where sub-dictionaries are created, updated, and released based on the lifecycle of corresponding data files. This dynamic approach allows the system to adapt memory usage to actual needs, releasing dictionary memory when no longer required while maintaining compression ratios.
2Loss of substance
If a global public dictionary is used for compression, then compression effectiveness is improved, but system performance deteriorates due to frequent swap-in and swap-out operations
Solution Approach 1:
By segmenting the global dictionary into file-specific sub-dictionaries, the patent enables the storage system to load only the sub-dictionary corresponding to the currently accessed data file into memory. This eliminates frequent swap-in and swap-out operations of the entire global dictionary, improving system performance while maintaining compression effectiveness.
Solution Approach 2:
The patent applies local quality by making each sub-dictionary specialized for its corresponding data file rather than using a single global dictionary for all files. This local optimization allows the system to maintain highly effective compression for each file type while reducing overall memory management overhead.
3Adaptability or versatility
If metadata area is not pre-allocated, then storage flexibility is maintained, but metadata overflow occurs causing performance deterioration
Solution Approach 1:
The patent implements preliminary action by pre-allocating the metadata area when the data file is created. This ensures that sufficient space is reserved for dictionary storage from the beginning, preventing metadata overflow and associated performance issues while maintaining storage flexibility through the pre-planned allocation strategy.
Data Source
Figure 1A
Figure 1B
Figure 1C
AI summary
This application provides a data compression method, including: obtaining a to-be-persisted data file, where the to-be-persisted data file includes a plurality of sub-files; and during persistence of a first sub-file in the plurality of sub-files, generating, based on the first sub-file, a dictionary corresponding to the first sub-file, and compressing the first sub-file by using the dictionary, to obtain a first compressed file; generating a first persistent file based on the first compressed file and the dictionary; and writing the first persistent file into a persistent storage medium. A single persistent file independently maintains a dictionary corresponding to a sub-file. The dictionary and the sub-file may implement same life cycle management. That is, a life cycle of data in the sub-file and a life cycle of the dictionary are affinity-bound. This solves a long-tail problem that a public dictionary cannot be released because a local data file references a global public dictionary. In this way, the following problem is avoided: Metadata expansion is caused because the public dictionary occupies a memory for a long time and cannot be released; consequently, performance deterioration of a storage system is further caused.