Quasi-dense grouping keys for database aggregation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face inefficiencies in aggregation processing due to the high cost of grouping and deduplication, especially with variable-width natural grouping keys, leading to issues like collisions and wasteful pre-allocation of bins, which exacerbate processing time and memory consumption.
Innovation Solution
The implementation of quasi-dense grouping keys, which decompose the aggregation process into two phases: DGK generation and aggregation, using a multilevel hash table and vector instructions to accelerate the generation and aggregation of results, allowing for concurrent grouping in memory and reducing the need for pre-allocation of bins.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If pre-allocation of bins is used for natural grouping keys, then direct mapping to aggregation bins is achieved, but memory waste occurs when grouping key values are sparse
Solution Approach 1:
The patent transforms natural grouping keys (variable width, sparse values) into dense grouping keys (fixed width, contiguous values) through a mapping function. This parameter transformation enables efficient array indexing while eliminating memory waste from sparse pre-allocation, as the dense keys directly index into compact aggregation bins without requiring bins for all possible natural key values.
Solution Approach 2:
The patent introduces dense grouping keys as an intermediary between natural grouping keys and aggregation bins. This intermediate representation layer decouples the sparse, variable-width natural keys from the dense, fixed-width bin indices, enabling both direct mapping efficiency and memory optimization by only allocating bins for actually occurring groups.
2Speed
If hash tables are used for grouping, then constant time access is achieved, but cache thrashing occurs due to collision resolution
Solution Approach 1:
The patent extracts the collision resolution step entirely by using dense grouping keys that directly index into aggregation bins without requiring hash table lookup. This eliminates the multi-level memory access pattern inherent in hash tables, preventing cache thrashing while maintaining constant time access through direct array indexing with the dense key as offset.
3Reliability
If variable-width natural grouping keys are used, then data fidelity is maintained, but processing efficiency decreases
Solution Approach 1:
The patent applies a transformation function that converts variable-width natural grouping keys into fixed-width dense grouping keys. This parameter change standardizes the key format for efficient processing (enabling vector operations and direct indexing) while preserving the one-to-one mapping relationship that maintains data fidelity and enables accurate aggregation results.
4Loss of substance
If dictionary encoding is used for variable-width keys, then memory efficiency improves, but dictionary maintenance becomes expensive
Solution Approach 1:
The patent enables each processing thread to independently generate its own dense grouping keys and local aggregation bins without requiring a centralized dictionary. This self-service approach eliminates dictionary maintenance overhead while achieving memory efficiency through local dense encoding, as threads autonomously map their encountered natural keys to contiguous dense keys without external reference structures.
Data Source
AI summary
Herein are techniques for dynamic aggregation of results of a database request, including concurrent grouping of result items in memory based on quasi-dense keys. Each of many computational threads concurrently performs as follows. A hash code is calculated that represents a particular natural grouping key (NGK) for an aggregate result of a database request. Based on the hash code, the thread detects that a set of distinct NGKs that are already stored in the aggregate result does not contain the particular NGK. A distinct dense grouping key for the particular NGK is statefully generated. The dense grouping key is bound to the particular NGK. Based on said binding, the particular NGK is added to the set of distinct NGKs in the aggregate result.