Table Compression Using Value Occurrence Vectors for In-Memory Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data processing methods for large database tables require significant hardware resources due to the need for large amounts of memory and storage space, which can lead to high demands on processor cache, network bandwidth, and overall system performance.
Innovation Solution
The implementation of dictionary-based compression and bit vector compression techniques, which generate columns of dictionary-based compression values, sort columns based on most-frequently occurring values, and use bit vectors to represent these values, allowing for efficient in-memory searches and reduced data storage requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If large amounts of data are stored in database tables, then search functionality is provided, but memory and storage space requirements increase significantly
Solution Approach 1:
The patent segments the data storage into multiple compressed columns, where each column stores a specific attribute type (e.g., customer_name, product_name). This segmentation allows for targeted compression techniques to be applied to each column based on its characteristics, reducing overall storage requirements while maintaining search capability across all attributes.
Solution Approach 2:
The patent creates compressed representations (copies) of the original data in multiple formats: dictionary-based compressed columns, bit vector columns, and hybrid columns. These compressed copies enable search operations to be performed on reduced-size data structures that retain the essential information needed for querying.
2Quantity of substance
If data is compressed using dictionary-based and bit vector techniques, then storage requirements are reduced, but data processing complexity increases
Solution Approach 1:
The patent performs preliminary compression actions during data loading, organizing data into dictionary-based columns, bit vector columns, and hybrid columns before search operations. This preliminary structuring eliminates the need for complex compression algorithms to be executed during query processing, as the data is already in an optimized format for both storage efficiency and retrieval.
Solution Approach 2:
The patent introduces intermediary data structures (dictionary mappings, bit vector representations, and hybrid column formats) that mediate between the original data and the compressed storage format. These intermediaries simplify the processing complexity by providing straightforward lookup and comparison mechanisms during search operations.
3Speed
If columns are sorted by most-frequently occurring values, then search efficiency is improved, but sorting time and computational resources increase
Solution Approach 1:
The patent applies sorting locally to individual columns based on their specific characteristics, particularly sorting each column by the frequency of its values. This local sorting approach optimizes search efficiency for each attribute type independently, allowing the most frequently queried values to be positioned in a manner that accelerates retrieval operations.
Solution Approach 2:
The patent performs sorting as a preliminary action during the data loading and compression phase, before any search operations occur. By pre-sorting the compressed columns by value frequency, the system eliminates the need for repeated sorting during queries, amortizing the sorting cost over multiple search operations and improving overall search efficiency.
Data Source
AI summary
Methods and apparatus, including computer program products, for compression of tables based on occurrence of values. In general, a number representing an amount of occurrences of a frequently occurring value in a group of adjacent rows of a column is generated, a vector representing whether the frequently occurring value exists in a row of the column is generated, and the number and the vector are stored to enable searches of the data represented by the number and the vector. The vector may omit a portion representing the group of adjacent rows. The values may be dictionary-based compression values representing business data such as business objects. The compression may be performed in-memory, in parallel, to improve memory utilization, network bandwidth consumption, and processing performance.