Parallel Index Creation on Sorted Data Using Worker Processes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional index creation for sorted data in database systems is serial, leading to inefficient processing time and resource consumption, especially when dealing with compressed or encrypted data.
Innovation Solution
The creation of an index for sorted data is parallelized by dividing the data into logical partitions, each processed by a separate worker process, with sub-indexes being built and later merged by a coordinating process, allowing for simultaneous processing and efficient handling of compressed and encrypted data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If serial query plan is used for index creation, then implementation simplicity is maintained, but processing time and resource consumption are excessive
Solution Approach 1:
The patent divides the index creation process into multiple segments by partitioning the table data into N+1 logical partitions and assigning each partition to a separate worker process. This segmentation enables parallel processing of index creation, significantly improving productivity while managing complexity through structured division of labor between coordinator and worker processes
Solution Approach 2:
The patent transitions from a single-dimensional serial processing approach to a multi-dimensional parallel processing architecture by introducing multiple worker processes operating simultaneously on different partitions, with a coordinator process managing the overall workflow. This dimensional expansion resolves the contradiction by adding parallelism without creating unmanageable complexity
2Loss of time
If multiple worker processes are used for parallel index creation, then processing time is reduced, but coordination complexity increases
Solution Approach 1:
The patent segments the index creation workload into N+1 independent partitions, each handled by a dedicated worker process. This segmentation reduces the time loss by enabling simultaneous processing while managing coordination complexity through clear partition boundaries and a centralized coordinator process that orchestrates the worker processes
Solution Approach 2:
The patent introduces a coordinator process as an intermediary that manages communication and coordination between multiple worker processes. This mediator handles the complexity of parallel process management, allowing worker processes to focus on their specific partition tasks while the coordinator manages merging and overall coordination, thus reducing time loss without proportionally increasing coordination complexity
3Productivity
If compressed data rows are processed serially, then memory usage is low, but decompression time consumes excessive processing time
Solution Approach 1:
The patent segments the decompression workload by assigning different data partitions to different worker processes. Each worker process decompresses its assigned partition in parallel, significantly improving decompression processing speed while distributing computational resource consumption across multiple processes, thus resolving the contradiction between productivity and energy use
Data Source
AI summary
Creation of an index for a table of sorted data for use by a data storage application is initiated. Thereafter, N+1 logical partition of rows of the table are defined so that each logical partition has a corresponding worker process. Each worker process then builds a sub-index based on the corresponding logical partition which are later merged to form the index. Related apparatus, systems, techniques and articles are also described.