Block-Based String Dictionary Compression for Faster In-Memory Databases
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database compression methods, such as Re-Pair Front Coding (RPFC), face challenges with high compression times and memory consumption, especially when dealing with large string dictionaries, making them impractical for multi-terabyte in-memory database management systems (IMDBMS).
Innovation Solution
The implementation of a block re-pair front coding (BRPFC) method, which splits the dictionary into blocks, applies front coding, and uses Re-Pair compression on each block individually, optimizing compression times and reducing memory footprint by storing offsets and grammar within each block.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If Re-Pair Front Coding (RPFC) is applied to compress string dictionaries, then compression ratios are improved, but compression time and memory consumption increase significantly
Solution Approach 1:
The patent divides the string dictionary into multiple blocks, where each block contains a subset of the dictionary strings. This segmentation allows the compression algorithm to process smaller portions independently and in parallel, reducing overall compression time while maintaining the compression effectiveness of the full dictionary.
2Loss of substance
If Re-Pair Front Coding (RPFC) is applied to compress string dictionaries, then compression ratios are improved, but memory consumption increases
Solution Approach 1:
By segmenting the dictionary into blocks, the patent reduces the memory footprint required for the compression data structures. Each block can be processed and stored independently, allowing the system to handle large dictionaries with limited memory resources.
3Loss of substance
If the entire dictionary is processed as a single unit, then compression ratios are maximized, but processing time and memory usage become impractical for multi-terabyte databases
Solution Approach 1:
The patent segments the dictionary into blocks that can be processed independently and in parallel. This enables the system to achieve practical compression speeds for large-scale databases while maintaining acceptable compression ratios, making the approach feasible for multi-terabyte in-memory database management systems.
Solution Approach 2:
After compressing individual blocks independently, the patent merges them into a complete compressed dictionary representation. This combining approach allows the system to achieve both the speed benefits of parallel processing and the compression effectiveness of a unified dictionary structure.
Data Source
AI summary
A method, a system, and a computer program product for executing a database compression. A compressed string dictionary having a block size and a front coding bucket size is generated from a dataset. Front coding is applied to one or more buckets of strings in the dictionary having the front coding bucket size to generate one or more front coded buckets of strings. One or more portions of the generated front coded buckets of strings are concatenated to form one or more blocks having the block size. Each block is compressed. A set of compressed blocks is stored. The set of the compressed blocks stores all strings in the dataset.