Column Value Compression with Bit Vectors for In-Memory Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Processing large amounts of data in database tables requires significant hardware resources, leading to high memory and storage demands, and existing compression methods are inefficient for handling massive datasets.
Innovation Solution
Implementing dictionary-based compression and bit vector compression techniques to identify and remove most-frequently occurring values, sorting columns to prioritize these values, and generating bit vectors to represent occurrences, thereby reducing data volume and improving search efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is stored in uncompressed form in database tables, then search operations can access all data directly, but memory and storage requirements become excessively large for massive datasets
Solution Approach 1:
The patent segments data by identifying and separating frequently occurring values from less frequent ones. Bit vectors are created to represent the presence/absence of frequently occurring values, while a separate structure stores the actual value data. This segmentation allows efficient searching of common values without storing complete uncompressed data, resolving the contradiction between data accessibility and storage requirements.
Solution Approach 2:
The patent changes the representation parameter of data from storing complete value sequences to using bit vectors and occurrence counts. Instead of storing actual value data for frequently occurring entries, the system uses binary representations (0 or 1) indicating presence, fundamentally changing how data is parameterized to reduce storage while maintaining search capability.
2Quantity of substance
If compression techniques are applied to reduce data volume, then memory and storage requirements decrease, but processing and search efficiency may be degraded
Solution Approach 1:
The patent performs preliminary organization of data during the compression phase by sorting columns and identifying frequently occurring values before compression. Bit vectors are pre-computed to mark positions of these values. This preliminary action ensures that search operations can directly utilize the pre-organized structure without additional processing overhead, maintaining high productivity despite compression.
3Quantity of substance
If all columns are compressed using bit vectors, then compression ratio increases, but the complexity of managing and processing multiple bit vectors increases
Solution Approach 1:
The patent applies different compression qualities to different columns based on their characteristics. Columns with high frequency of repeated values receive full bit vector compression, while columns with lower repetition rates may use lighter compression or remain uncompressed. This local differentiation optimizes overall compression ratio while avoiding unnecessary complexity in managing bit vectors for columns where they provide minimal benefit.
Data Source
AI summary
Methods and apparatus, including computer program products, for compression of tables based on occurrence of values. In general, a number representing an amount of occurrences of a frequently occurring value in a group of adjacent rows of a column is generated, a vector representing whether the frequently occurring value exists in a row of the column is generated, and the number and the vector are stored to enable searches of the data represented by the number and the vector. The vector may omit a portion representing the group of adjacent rows. The values may be dictionary-based compression values representing business data such as business objects. The compression may be performed in-memory, in parallel, to improve memory utilization, network bandwidth consumption, and processing performance.