Block-Level Table Compression Using Reusable Value Dictionaries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Processing large amounts of data in database tables requires significant hardware resources due to the need for extensive memory and storage, leading to high demands on processor cache, network bandwidth, and overall system performance.
Innovation Solution
Implementing dictionary-based compression techniques to generate value identifiers, block dictionaries, and block vectors, which allow for efficient data compression by representing repeated values and maintaining data dependencies, thereby reducing memory and storage requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is stored in uncompressed format in database tables, then data can be accessed and processed directly without compression overhead, but significant hardware resources including memory space, storage capacity, and processor cache are required
Solution Approach 1:
The patent segments the data storage structure by dividing tables into blocks and columns into block dictionaries. Each block dictionary stores compressed data for a specific column block, allowing selective loading and processing of only relevant data segments into memory, thereby reducing overall memory requirements while maintaining processing efficiency
Solution Approach 2:
The patent applies compression parameters to transform data from uncompressed to compressed format. By changing the data representation parameters (using value identifiers instead of full values, implementing dictionary-based compression), the system reduces storage capacity requirements and memory usage while enabling efficient processing of compressed data
2Quantity of substance
If compression techniques are applied to reduce data volumes, then memory and storage requirements are reduced, but additional processing steps including compression and decompression are required
Solution Approach 1:
The patent performs preliminary compression actions during data loading and storage. Data is compressed into block dictionaries before being stored, and value identifiers are pre-computed and stored alongside compressed data. This preliminary processing reduces the complexity of runtime operations by having compression artifacts ready for efficient decompression and querying
Solution Approach 2:
The patent introduces block dictionaries as intermediary structures between the compressed data and the query processing system. These block dictionaries serve as mediators that store both compressed data and metadata (value identifiers, offsets) necessary for efficient decompression and searching, thereby managing processing complexity through structured intermediate representations
3Speed
If in-memory processing is used to improve response times, then faster data access is achieved, but large amounts of random access memory space are required to store relevant records
Solution Approach 1:
The patent segments data into block-level units organized in block dictionaries, allowing selective loading of only relevant blocks into memory based on query requirements. This segmentation enables in-memory processing of frequently accessed data blocks while keeping less frequently accessed blocks on disk, thereby reducing total memory space requirements while maintaining fast response times for active data
Solution Approach 2:
The patent changes the data representation parameters by storing compressed data with value identifiers in block dictionaries. This parameter transformation enables more compact in-memory storage of relevant records, allowing larger datasets to fit within available memory space while maintaining the speed benefits of in-memory processing through efficient compression ratios
Data Source
AI summary
Methods and apparatus, including computer program products, for block compression of tables with repeated values. In general, value identifiers representing a compressed column of data may be sorted to render repeated values contiguous, and block dictionaries may be generated. A block dictionary may be generated for each block of value identifiers. Each block dictionary may include a list of block identifiers, where each block identifier is associated with a value identifier and there is a block identifier for each unique value in a block. Blocks may have standard sizes and block dictionaries may be reused for multiple blocks.