Table Compression by Value Occurrence for In-Memory Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data processing methods for large database tables require significant hardware resources due to the need for large amounts of memory and storage space, especially when dealing with massive amounts of data, which can lead to increased demands on processor cache, network bandwidth, and overall system performance.
Innovation Solution
The implementation of dictionary-based compression and bit vector compression techniques, where columns are sorted based on the occurrence of values, with most-frequently occurring values removed and represented using bit vectors, allows for efficient in-memory searches and reduced data storage, enabling asynchronous updates and improved system performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored in traditional uncompressed format, then data accessibility and search capability are maintained, but memory and storage requirements become excessively large for massive datasets
Solution Approach 1:
The patent segments data into distinct columns and applies different compression techniques to each column based on its characteristics. Columns are divided into frequent values and infrequent values, with bit vector compression applied to frequent values and dictionary-based compression to infrequent values, optimizing both space and accessibility
Solution Approach 2:
The patent transforms data representation by changing parameters such as using bit vectors to represent frequent values (changing from storing actual values to storing binary indicators), and using dictionary codes for infrequent values (changing from full value storage to compressed code storage), achieving significant space reduction while maintaining search capability
2Quantity of substance
If compression techniques are applied to reduce data volume, then memory and storage requirements are reduced, but processing overhead and system complexity increase
Solution Approach 1:
The patent performs preliminary actions by pre-computing and storing bit vectors for frequent values and dictionary codes for infrequent values during data loading. This preliminary compression setup eliminates the need for complex real-time compression operations during querying, reducing processing overhead during actual data access
Solution Approach 2:
The patent introduces intermediary structures such as bit vectors and dictionary tables that act as mediators between the compressed data and the query processing system. These intermediaries enable efficient search operations on compressed data without requiring full decompression, balancing space reduction with processing efficiency
3Speed
If all data is loaded into memory for processing, then search speed is improved, but hardware resources and cost are significantly increased
Solution Approach 1:
The patent changes the parameter of data representation to enable more efficient memory utilization. By encoding frequent values as bit vectors (1 bit per value) and infrequent values as compact dictionary codes, the same memory space can hold much larger datasets, enabling in-memory processing of massive data volumes that would otherwise require expensive hardware
Data Source
AI summary
Methods and apparatus, including computer program products, for compression of tables based on occurrence of values. In general, a number representing an amount of occurrences of a frequently occurring value in a group of adjacent rows of a column is generated, a vector representing whether the frequently occurring value exists in a row of the column is generated, and the number and the vector are stored to enable searches of the data represented by the number and the vector. The vector may omit a portion representing the group of adjacent rows. The values may be dictionary-based compression values representing business data such as business objects. The compression may be performed in-memory, in parallel, to improve memory utilization, network bandwidth consumption, and processing performance.