Table Compression Using Bit Vectors for In-Memory Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Processing large amounts of data in database tables requires significant hardware resources, leading to high memory and storage demands, and existing compression methods are inefficient for handling massive datasets.
Innovation Solution
Implementing dictionary-based compression and bit vector compression techniques to identify and remove most-frequently occurring values, sorting columns to prioritize these values, and generating bit vectors to represent occurrences, thereby reducing data volume and improving search efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If large amounts of data are stored in database tables, then complete data availability is achieved, but memory and storage requirements increase significantly
Solution Approach 1:
The patent extracts frequently occurring values from the original data and stores them separately in a compressed format. By identifying and removing redundant duplicate values, the system maintains data availability while significantly reducing storage requirements. The extracted values are stored in a compact representation that can be efficiently reconstructed when needed.
Solution Approach 2:
The patent changes the parameter representation of data by encoding frequently occurring values using bit vectors and occurrence counts instead of storing full values. This parameter transformation allows the same information to be represented in a more compact form, reducing memory and storage requirements while preserving data integrity.
2Productivity
If all records are loaded into random access memory for processing, then processing speed is improved, but hardware resource requirements increase
Solution Approach 1:
The patent creates a compressed copy of the original data that can be loaded into memory for processing. This compressed representation contains all necessary information in a reduced form, allowing faster in-memory processing while requiring fewer hardware resources. The compressed data structure enables efficient access and manipulation without needing to load the complete uncompressed dataset.
3Quantity of substance
If compression techniques are applied to reduce data volume, then storage requirements are reduced, but compression efficiency may be insufficient for massive datasets
Solution Approach 1:
The patent segments the data by identifying and separating frequently occurring values from unique or rare values. This segmentation allows the application of different compression strategies optimized for each type of data. Frequently occurring values are compressed using bit vectors and occurrence counts, while other values are handled differently, achieving high overall compression efficiency for massive datasets.
Solution Approach 2:
The patent applies parameter changes by representing frequently occurring values using compact bit vectors and occurrence counts instead of full value representations. This parameter transformation achieves high compression ratios by exploiting the statistical properties of the data, specifically the frequency distribution of values, to reduce data volume efficiently.
Data Source
AI summary
Methods and apparatus, including computer program products, for compression of tables based on occurrence of values. In general, a number representing an amount of occurrences of a frequently occurring value in a group of adjacent rows of a column is generated, a vector representing whether the frequently occurring value exists in a row of the column is generated, and the number and the vector are stored to enable searches of the data represented by the number and the vector. The vector may omit a portion representing the group of adjacent rows. The values may be dictionary-based compression values representing business data such as business objects. The compression may be performed in-memory, in parallel, to improve memory utilization, network bandwidth consumption, and processing performance.