Block Dictionary Compression for Repeated-Value Tables
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing methods for large database tables with repeated values require significant hardware resources due to the need for large memory spaces and data storage, leading to inefficiencies in data compression and processing.
Innovation Solution
Implementing dictionary-based compression techniques to generate value identifiers, block dictionaries, and block vectors, which allow for efficient compression and sorting of data, reducing memory requirements and enabling cost-effective in-memory processing by representing repeated values with minimal bit coding and reusing dictionaries across blocks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional data storage methods are used for large database tables, then data can be stored and accessed, but significant hardware resources and large memory spaces are required
Solution Approach 1:
The patent segments the data storage into multiple components: a compressed column storing only unique values, a dictionary mapping compressed values to original values, and block offset information. This segmentation allows the system to store large tables with repeated values using minimal space by only preserving unique values in the compressed column.
Solution Approach 2:
The patent creates a dictionary that copies and stores the mapping between compressed values and original values. Instead of storing redundant original values repeatedly, the system copies the unique value once into the compressed column and uses the dictionary to restore original values during query operations.
2Productivity
If large memory spaces are allocated to store all relevant records, then data processing can be performed, but hardware resource consumption increases significantly
Solution Approach 1:
The patent divides the data structure into essential components (compressed column with unique values, dictionary for value mapping, and block offsets) and discards redundant information. This segmentation enables processing of large datasets in memory with minimal resource consumption by only loading necessary unique values rather than all repeated records.
Solution Approach 2:
The patent discards redundant repeated values during compression and recovers them during query processing by referencing the dictionary. This allows the system to use minimal memory for storage while maintaining the ability to process and retrieve complete data when needed.
3Quantity of substance
If data is compressed using traditional methods, then storage space is reduced, but processing speed and search efficiency may deteriorate
Solution Approach 1:
The patent performs preliminary sorting of the data before compression to group identical values together in contiguous blocks. This preliminary action enables efficient compression by identifying unique values and their positions, and also facilitates fast search operations by maintaining the sorted structure that allows binary search and range queries.
Solution Approach 2:
The patent creates a dictionary that serves multiple functions: it maps compressed values back to original values for data restoration, enables fast lookups during search operations, and supports various query types including equality searches and range queries. This multi-functional structure maintains processing speed while achieving compression.
Data Source
AI summary
Methods and apparatus, including computer program products, for block compression of tables with repeated values. In general, value identifiers representing a compressed column of data may be sorted to render repeated values contiguous, and block dictionaries may be generated. A block dictionary may be generated for each block of value identifiers. Each block dictionary may include a list of block identifiers, where each block identifier is associated with a value identifier and there is a block identifier for each unique value in a block. Blocks may have standard sizes and block dictionaries may be reused for multiple blocks.