Parallel Compressed Column Loading in Column-Store Databases
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Column-store databases face challenges in loading data efficiently due to input data typically being in row-major format, requiring conversion to column-major format, which hampers performance and increases load time.
Innovation Solution
The method involves using multiple page-formatter threads to add compressed columns in parallel to database pages, each thread handling a single compressed column, thereby exploiting parallel and vector processing capabilities to reduce load times.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If data is loaded using traditional sequential methods converting from row-major to column-major format, then data conversion accuracy is maintained, but data load time is excessive
Solution Approach 1:
The patent divides the data loading process into multiple independent segments by creating separate page-formatter threads for different compressed columns. Each thread independently loads and converts data for its assigned column, enabling parallel processing that reduces overall load time while maintaining conversion accuracy through dedicated processing for each column segment.
Solution Approach 2:
The patent implements dynamic parallel processing by launching multiple page-formatter threads that can simultaneously execute the conversion process. This dynamic approach allows the system to utilize multiple CPU cores and processors, transforming the static sequential conversion into a dynamic concurrent process that significantly reduces data load time without sacrificing data integrity.
2Productivity
If multiple threads are used to load data in parallel, then data loading speed is improved, but thread coordination complexity increases
Solution Approach 1:
The patent reduces thread coordination complexity by segmenting the work so that each page-formatter thread handles a distinct compressed column. This segmentation minimizes the need for inter-thread communication and synchronization, as threads operate independently on separate data streams, thereby maintaining high loading speed while reducing coordination overhead.
Solution Approach 2:
Each page-formatter thread is designed to be self-sufficient, independently reading input data, converting it to column-major format, and writing to its designated output buffer without requiring complex coordination with other threads. This self-service approach simplifies thread management while achieving parallel loading speed improvements.
3Quantity of substance
If data is converted from row-major to column-major format using single-threaded processing, then memory usage is optimized, but conversion time becomes excessive
Solution Approach 1:
The patent segments the conversion process across multiple threads, each handling a portion of the data. By dividing the input data and output buffers into column-specific segments that can be processed independently, the system achieves parallel conversion that reduces time while maintaining optimized memory usage through efficient buffer management for each segment.
Solution Approach 2:
The patent transitions from single-threaded sequential processing to multi-threaded parallel processing, adding a temporal dimension to the conversion process. This dimensional change allows simultaneous execution of conversion operations across multiple threads, reducing overall conversion time while maintaining memory efficiency through structured buffer allocation for each thread.
Data Source
AI summary
In one embodiment, a method includes adding, by a computer processor, two or more compressed columns to one or more pages of a database. The adding is performed in parallel by a plurality of page-formatter threads. Each page-formatter thread adds data to the database from no more than a single compressed column.