Dense Dictionary Encoding for Faster Sparse-Code Database Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database systems face inefficiencies in data compression and retrieval due to the use of sparse and composite codes, which require significant processing resources and storage, especially in multi-node database systems.
Innovation Solution
The implementation of relational approaches using specialized SQL operators like ENCODE_FIRST and ENCODE_SPARSE_CODE to generate dense and sorted codes, optimizing data encoding and decoding processes by reducing storage needs and improving query efficiency through specialized database queries and operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If sparse and composite codes are used for data compression, then data can be encoded with variable length codes, but storage requirements and processing resources increase significantly
Solution Approach 1:
The patent transforms the code representation parameters from sparse/composite format to dense/sorted format. This parameter change reorganizes the encoding structure to use contiguous integer values (1, 2, 3, ...) instead of sparse values with gaps, and sorts codes consistently with token order, thereby reducing storage requirements while improving processing efficiency through more compact data structures
Solution Approach 2:
The patent discards the sparse and composite code structures that consume excessive storage and processing resources. By eliminating the need to manage variable-length codes, gaps in code sequences, and complex code mappings, the system recovers performance through simpler dense integer codes that can be processed more efficiently while maintaining the same data compression capability
2Speed
If dense and sorted codes are generated using relational approaches, then query efficiency improves, but implementation complexity increases
Solution Approach 1:
The patent uses SQL views to create virtual copies of the encoded data that present dense and sorted code sequences to queries. Instead of physically reorganizing the stored data, the views copy and transform the data representation on-demand, providing efficient query performance through standardized code sequences while keeping the underlying storage implementation simple and unchanged
3Quantity of substance
If specialized SQL operators are used for encoding, then data compression efficiency improves, but processing overhead increases
Solution Approach 1:
The patent performs code densification and sorting as preliminary actions during the initial encoding phase using SQL operators. By establishing the dense and sorted code structure upfront during data loading or ETL processes, the system eliminates the need for complex processing during subsequent queries, thereby reducing processing overhead during high-frequency operations while maintaining improved compression ratios
Data Source
AI summary
Unsorted sparse dictionary encodings are transformed into unsorted-dense or sorted-dense dictionary encodings. Sparse domain codes have large gaps between codes that are adjacent in order. Unlike spare codes, dense codes have smaller gaps between adjacent codes; consecutive codes are dense codes that have no gaps between adjacent codes. The techniques described herein are relational approaches that may be used to generate sparse composite codes and sorted codes.