Block Re-Pair Front Coding for Faster Database Decompression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database compression methods, such as Re-Pair Front Coding (RPFC), face challenges with high compression times and memory consumption, especially when dealing with large string dictionaries, making them impractical for multi-terabyte in-memory database systems.
Innovation Solution
The proposed solution is a block re-pair front coding (BRPFC) method that splits the dictionary into blocks, applies front coding, and then uses Re-Pair compression on each block individually, optimizing compression times and memory usage by limiting the number of symbols and using vectorized operations for decompression.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If Re-Pair compression is applied to the entire string dictionary, then compression ratio is improved, but compression time and memory consumption increase significantly
Solution Approach 1:
The string dictionary is divided into multiple blocks, where each block contains a subset of the dictionary strings. Re-Pair compression is applied independently to each block rather than the entire dictionary at once. This segmentation reduces the computational complexity and memory requirements while maintaining overall compression effectiveness.
Solution Approach 2:
The patent introduces a hierarchical block structure dimension, organizing the dictionary into blocks of blocks. This multi-dimensional organization allows parallel processing of blocks and reduces the linear time complexity associated with processing the entire dictionary as a single unit.
2Loss of substance
If Re-Pair compression is applied to the entire string dictionary, then compression ratio is improved, but memory consumption increases
Solution Approach 1:
By segmenting the dictionary into blocks, the patent reduces the memory footprint required for Re-Pair compression. Each block can be processed independently with its own compressed representation, avoiding the need to load the entire dictionary into memory simultaneously. This enables processing of large dictionaries that would otherwise exceed available memory resources.
3Loss of substance
If front coding is applied to compress strings, then compression is achieved, but decompression speed decreases due to complex processing
Solution Approach 1:
The patent applies front coding during the compression phase to pre-process and simplify the string representations. By performing the computationally intensive front coding operations during compression rather than decompression, the patent achieves good compression ratios while keeping decompression operations simpler and faster.
Data Source
AI summary
A method, a system, and a computer program product for decompressing data. One or more compressed blocks in a set of stored compressed blocks responsive to a request to access data in the set of stored compressed blocks are identified. String prefixes inside the identified compressed blocks are decompressed using front coding. String suffixes inside the identified compressed blocks are decompressed using a re-pair decompression. Uncompressed data is generated.