Block Dictionary Reuse for Repeated-Value Table Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing methods for large database tables with repeated values require significant hardware resources due to the need for large memory spaces and data storage, leading to inefficiencies in processing and storage.
Innovation Solution
Implementing dictionary-based compression techniques to generate value identifiers, block dictionaries, and block offset columns, which allow for efficient compression and sorting of data, reducing memory requirements and enabling cost-effective in-memory processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If large amounts of data are stored in database tables, then data completeness is maintained, but memory space requirements increase significantly
Solution Approach 1:
The patent segments the data storage structure by dividing tables into blocks and further dividing blocks into groups. This hierarchical segmentation allows efficient compression by identifying and exploiting repeated patterns at multiple levels (block repetition, group repetition) without losing data completeness. The segmented structure enables selective compression of repeated blocks while maintaining random access capability.
Solution Approach 2:
The patent uses copy-on-write techniques where repeated blocks are stored once and referenced multiple times through pointers or offsets. When a block is repeated, instead of storing duplicate data, the system creates a reference to the original block. This copying approach dramatically reduces memory space requirements while preserving data integrity and allowing efficient retrieval through the reference mechanism.
2Volume of stationary object
If data is compressed to reduce memory requirements, then storage efficiency improves, but processing speed may deteriorate
Solution Approach 1:
The patent performs preliminary compression and organization of data into blocks and groups before processing queries. The data is pre-processed into a compressed format with metadata structures that enable fast navigation. This preliminary action ensures that when queries are executed, the system can quickly locate and access relevant data without performing expensive compression/decompression operations during query processing, thus maintaining high processing speed.
Solution Approach 2:
The patent introduces intermediary data structures such as block headers, group metadata, and offset tables that mediate between the compressed data and the query processing system. These intermediaries provide fast lookup mechanisms and navigation paths, allowing the system to efficiently access compressed data without full decompression. The intermediaries act as translators that enable quick identification of relevant data locations in the compressed structure.
3Quantity of substance
If block compression with dictionaries is implemented, then compression ratio improves, but system complexity increases
Solution Approach 1:
The patent changes the parameter representation by using compact block identifiers and offset values instead of full data copies. The system transforms the data representation parameters to include block IDs, group IDs, and offset pointers that use fewer bits than the original data. This parameter change enables high compression ratios while the structured nature of these parameters keeps the complexity manageable through consistent encoding rules.
Data Source
AI summary
Methods and apparatus, including computer program products, for block compression of tables with repeated values. In general, value identifiers representing a compressed column of data may be sorted to render repeated values contiguous, and block dictionaries may be generated. A block dictionary may be generated for each block of value identifiers. Each block dictionary may include a list of block identifiers, where each block identifier is associated with a value identifier and there is a block identifier for each unique value in a block. Blocks may have standard sizes and block dictionaries may be reused for multiple blocks.