Parallel Column Task Execution in Row-Partitioned Databases
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In row-partitioned database systems, processing column-oriented tasks serially leads to reduced response times due to multiple reads and serial processing of entire rows, which can be inefficient compared to processing entire rows in a columnar format.
Innovation Solution
A database system processes multiple column-oriented tasks in parallel by generating processing tasks for each unique column-oriented task, retrieving relevant rows, and performing column data processing in parallel, allowing for efficient generation of results sets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If column-oriented tasks are processed serially in a row-partitioned database system, then the system maintains simple row-based storage structure, but the response time increases due to multiple reads and serial processing
Solution Approach 1:
The patent segments the serial processing of column-oriented tasks into multiple parallel processing threads. Each thread handles a specific column-oriented task independently, allowing simultaneous execution of multiple tasks that were previously processed sequentially. This segmentation directly addresses the bottleneck of serial processing while maintaining the underlying row-partitioned storage structure.
Solution Approach 2:
The patent introduces dynamic parallel processing capabilities to the row-partitioned database system. The system dynamically creates and manages multiple processing threads based on the query requirements, enabling flexible adaptation between serial and parallel processing modes. This dynamic approach allows the system to optimize response time by activating parallel processing when column-oriented tasks are detected, while maintaining simple row-based storage structure.
2Loss of time
If multiple column-oriented tasks are processed in parallel, then response time is reduced, but the system complexity increases due to multiple processing threads and task management
Solution Approach 1:
The patent implements a universal processing framework that handles both row-oriented and column-oriented tasks through the same parallel processing infrastructure. The processing threads are designed to be multi-functional, capable of executing different types of column-oriented tasks (aggregations, projections, filters) on different columns simultaneously. This universality reduces system complexity by avoiding separate specialized processors for each task type.
Solution Approach 2:
The patent introduces an intermediary query processing layer that sits between the query parser and the execution engine. This intermediary layer analyzes incoming queries, identifies column-oriented tasks, and generates appropriate parallel processing threads. By centralizing the complexity management in this intermediary layer, the system maintains simple row-based storage structure while enabling parallel processing through a controlled interface that abstracts the complexity of thread management.
3Ease of operation
If entire rows are processed to extract specific columns, then data retrieval is simplified, but unnecessary data is read and processed increasing response time
Solution Approach 1:
The patent extracts only the necessary column data from rows during the parallel processing phase, rather than retrieving and processing entire rows. Each processing thread is configured to extract specific columns relevant to its assigned column-oriented task. This extraction principle eliminates the waste of reading and processing unnecessary data while maintaining the simplicity of row-based storage structure, as the extraction occurs during the parallel processing phase rather than during data retrieval.
Data Source
AI summary
A database system may process multiple column-oriented tasks in parallel for a database being stored according to a row-partitioning protocol. The database system may determine when the query should process the column-oriented task serially or in parallel. For parallel processing, the database system may generate processing tasks for each unique column-oriented task contained in the query used to retrieve column data and to process the column data according to the column-oriented tasks requested. A method of operating the database system may determine that multiple column-oriented tasks included in a query are to be processed in parallel. The method may further include generating a processing task for each unique column-oriented task included in the query. The method may further include performing the column-oriented tasks in parallel based on the processing threads. The method may implement various considerations in determining to process to the column-oriented tasks in parallel.