Column Dictionary Compression for Faster Cross-Column Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database systems face inefficiencies in data access due to the limitations of row-major and column-major storage formats, which are not optimized for all types of queries and data manipulation operations.
Innovation Solution
The implementation of column domain dictionary compression techniques, where column values are represented by tokens and mapped through a dictionary, allowing for efficient data encoding and query optimization by rewriting queries to access a dictionary table instead of the base table.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If row-major format is used for storing database data, then entire row access is efficient, but data density and compressibility are reduced
Solution Approach 1:
The patent segments data storage into two distinct formats: row-major format for efficient row access operations and column-major compressed format for storage efficiency. This segmentation allows the system to optimize for both access patterns simultaneously by storing data in the most appropriate format for each operational context.
Solution Approach 2:
The system dynamically selects between row-major and column-major compressed formats based on the type of operation being performed. Query rewrite rules dynamically transform queries to access data in the optimal format, making the storage system adaptive rather than static.
2Quantity of substance
If column-major format is used for storing database data, then data density and compressibility are improved, but queries requiring multiple columns or row modifications become less efficient
Solution Approach 1:
The patent introduces query rewrite rules as an intermediary layer between the user's query and the physical data storage. These rules translate high-level queries into optimized access patterns that leverage column-major compressed storage while maintaining ease of operation, effectively mediating between the compressed storage format and user-friendly query interfaces.
3Ease of operation
If data is stored in uncompressed format, then data access is straightforward, but storage space consumption increases
Solution Approach 1:
The patent changes the parameter of data representation by introducing tokenization. Instead of storing raw data values, the system stores compact tokens that reference a dictionary of actual values. This parameter change dramatically reduces storage space while maintaining data access capability through the dictionary mapping mechanism.
4Volume of stationary object
If data is compressed using traditional methods, then storage space is reduced, but query processing complexity increases
Solution Approach 1:
The patent performs preliminary actions by pre-computing and storing query rewrite rules that capture common query patterns. These pre-computed rules are stored alongside the compressed data, eliminating the need for complex real-time decompression and re-compression operations during query processing. The complexity is shifted from runtime to setup time.
Data Source
AI summary
In column domain dictionary compression, column values in one or more columns are tokenized by a single dictionary. The domain of the dictionary is the entire set of columns. A dictionary may not only map a token to a tokenized value, but also to a count (“token count”) of the number of occurrences of the token and corresponding tokenized value in the dictionary's domain. Such information may be used to compute queries on the base table.