Lock-Free Column Generation with Minimal Dictionaries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed database management systems, existing approaches for plan operations in OLAP engines lead to inefficient handling of 'group-by' results, particularly due to the generation of unnecessary dictionaries and lack of parallelization, resulting in high memory usage and slow performance during query processing.
Innovation Solution
A lock-free column generation process is implemented, where parallel worker jobs create minimal dictionaries by mapping and renaming value identifiers, reducing memory requirements and improving performance by storing only used values and allowing for dynamic load balancing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If materialized columns are generated saving each value directly within the column, then data integrity is maintained, but memory usage increases and processing time increases
Solution Approach 1:
The patent extracts duplicate values from the column data and stores them separately in a dictionary structure. Instead of storing every value directly in the column (materialized columns), only unique values are stored in the dictionary, and integer identifiers are stored in the column. This separation reduces memory usage while maintaining data integrity through the mapping between identifiers and values.
Solution Approach 2:
The patent creates a copy of the dictionary structure that maps integer identifiers to values. This dictionary copy is shared across multiple columns that reference the same values, allowing efficient storage of referenced values without duplicating the actual value data in each column. This copying approach reduces overall memory consumption while preserving data integrity.
2Productivity
If traditional plan operations generate result columns, then query processing is completed, but processing time increases significantly
Solution Approach 1:
The patent segments the column generation process into parallel worker threads that operate independently on different portions of the data. Each worker thread processes a segment of the input data, creates local dictionaries, and generates result columns in parallel. This segmentation enables concurrent processing and significantly reduces overall processing time while maintaining correctness through coordination of the parallel workers.
Solution Approach 2:
The patent performs preliminary actions by pre-computing and caching dictionaries of unique values and their integer identifiers before the main query processing. This preliminary dictionary creation allows subsequent column generation to use fast integer-to-value mapping instead of repeated value lookups or materialization, significantly accelerating query processing time.
3Loss of information
If dictionaries are generated for all values in group-by columns, then complete data mapping is achieved, but memory consumption increases
Solution Approach 1:
The patent applies local quality by creating dictionaries only for the specific set of unique values that actually appear in the group-by columns during query execution, rather than pre-creating dictionaries for all possible values. Each worker thread maintains a local dictionary for its segment of data, and these local dictionaries are merged to form the complete mapping. This approach ensures complete data mapping for actual values while minimizing memory consumption by excluding unused values.
Data Source
AI summary
A new dictionary can be created for a result column in a query plan operation executed on a database. The result column can be generated by multiple worker jobs running in parallel to read tasks from a shared queue as part of a query plan operation that includes a group-by column within an input set of input columns. The group-by column can include an original dictionary for all values contained within the group-by column If the new dictionary has fewer entries than the original dictionary for the group-by column such that mapping is required between old value identifiers within the group-by column and new value identifiers within the result column, the old value identifiers are renamed to the new value identifiers using a mapping vector.