Compressed Data Block Indexing for Low-Amplification Random Reads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data compression methods for read/write file systems suffer from random read amplification and low read efficiency, particularly in scenarios where data and metadata updates are required.
Innovation Solution
A data compression method that compresses data blocks using a preset algorithm, establishing a first index to identify storage locations and attribute information, allowing for efficient random reads with minimal amplification, and supporting data and metadata updates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data blocks are compressed using a fixed-size minimum compressible unit, then data storage capacity is reduced, but random read efficiency deteriorates and read amplification increases
Solution Approach 1:
The patent segments compressed data into fixed-size data blocks with separate indexing. Each data block is independently addressable through an index structure that maps logical block addresses to physical compressed data locations. This segmentation allows random access to individual blocks without reading entire compressed files, resolving the contradiction between compression efficiency and random read performance.
Solution Approach 2:
The patent introduces an index as an intermediary layer between the compressed data and the access interface. The index contains metadata about each data block including its position in the compressed stream, allowing the system to directly locate and retrieve specific blocks without sequential scanning. This intermediary structure enables efficient random reads while maintaining compressed storage.
2Quantity of substance
If compressed data is stored without an index structure, then storage overhead is reduced, but data retrieval time increases
Solution Approach 1:
The patent performs preliminary organization of compressed data into fixed-size blocks with pre-computed indexes during the compression phase. The index structure is built in advance, storing metadata about each block's position and characteristics. This preliminary action enables fast retrieval during read operations without incurring time penalties during actual data access.
3Stability of the object's composition
If a fixed output compression manner is used, then compression consistency is improved, but adaptability to data and metadata updates deteriorates
Solution Approach 1:
The patent implements a dynamic compression system where the fixed output block size provides structural consistency, but the index structure allows flexible insertion, deletion, and modification of individual blocks. When data or metadata needs to be updated, the system can modify specific blocks and update corresponding index entries without re-compressing the entire dataset, thus maintaining both consistency and adaptability.
Data Source
Figure 1a
Figure 1b
Figure 2
AI summary
This application discloses a data compression method and apparatus. The method includes: obtaining m data blocks in a data area of a readable and writable file system; compressing the m data blocks by using a preset compression algorithm, to sequentially obtain n compressed data blocks, where first capacities of all compressed data blocks are the same, and the first capacity represents a quantity of bytes of compressed data that can be included in the compressed data block; and establishing a first index of each of j data blocks corresponding to an ith compressed data block in the n pieces of compressed data, and recording a mapping relationship between first indexes and the j data blocks. The first index is used to identify a storage location, in a storage medium, of each data block included in the j data blocks, and attribute information included in each of the j data blocks. When a data block is read, read efficiency can be effectively improved, and it can be ensured that data is read in a random read scenario with a small read amplification coefficient.