Multi-Fragment Index Scans to Reduce Query Merge Overhead
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional fragment-local index scan operations on database tables impose significant computational overhead due to the need for merging separate result streams from each fragment, which can be inefficient and costly in terms of processing resources.
Innovation Solution
A multi-fragment index scan operation is performed on chunks of database table fragments, generating partial result sets through merging index scan outputs before an ad-hoc merge operation, thereby reducing computational overhead and leveraging parallel processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional fragment-local index scan operations are performed on database tables, then complete result sets can be obtained from each fragment, but significant computational overhead is incurred due to the need for merging separate result streams from each fragment
Solution Approach 1:
The patent divides the database table into multiple fragments and performs index scans on each fragment independently in parallel. Each fragment processing thread generates partial result sets that are then merged into a final result set. This segmentation allows parallel processing while maintaining complete results, resolving the contradiction between reliability and computational overhead.
Solution Approach 2:
The patent merges separate result streams from multiple fragment scans into a single unified result set. By using efficient merge operations that combine partial results from parallel fragment scans, the system achieves complete results with reduced computational overhead compared to sequential processing or full buffering approaches.
2Reliability
If separate result streams from each fragment are merged after independent index scans, then complete query results are achieved, but processing resources are consumed inefficiently
Solution Approach 1:
The patent performs preliminary index scans on individual fragments in parallel before merging results. By preparing partial result sets from each fragment independently and simultaneously, the system reduces total processing time and resource consumption while ensuring accurate complete results through the subsequent merge operation.
Solution Approach 2:
The patent maintains continuous useful action by performing index scans on multiple fragments simultaneously without idle periods. Parallel processing threads continuously scan different fragments, maximizing resource utilization and reducing total query processing time while maintaining result accuracy through coordinated merging.
3Reliability
If full result sets are buffered and merged from each fragment, then accurate query results are produced, but memory usage and processing time increase significantly
Solution Approach 1:
The patent performs partial index scans on fragments, generating partial result sets rather than complete result sets for each fragment. By processing and merging partial results incrementally, the system reduces memory buffering requirements and processing time while maintaining complete and accurate final results through the merge operation.
Data Source
AI summary
A method for performing a multi-fragment index scan operation included in a query pipeline of a query accessing a database table may include performing an index scan on a first chunk of a first fragment of the database table and a second chunk of a second fragment of the database table. A first output of the index scan on the first chunk of the first fragment of the database table and a second output of the index scan on the second chunk of the second fragment of the database table may be merged to generate a partial result set for ingestion by a next operation in the query pipeline. The multi-fragment index scan operation may continue with index scans on subsequent chunks of the first fragment and second fragment of the database table. Related systems and computer program products are also provided.