Chunked Query Execution with Ordered Index Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Long-running queries in multi-version concurrency control (MVCC) databases consume significant resources and can take extensive time to complete, risking data version expiration or system crashes, leading to inefficient resource usage and restarts.
Innovation Solution
A query processing system (QPS) divides long-running queries into shorter-running queries executed in chunks, using an ordered index to track progress and resume from intermediate results, allowing for resource sharing and simultaneous data updates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If long-running queries are executed against large datasets in MVCC databases, then complete data processing is achieved, but processing resources are monopolized for extended periods and data versions may expire causing query restarts
Solution Approach 1:
The patent divides a long-running query into multiple shorter query executions by using an ordered index to process data in sequential chunks. Each query execution processes a portion of the dataset and stores intermediate results, allowing the system to resume from the last successful chunk rather than restarting the entire query if a data version expires or system failure occurs.
2Productivity
If long-running queries are executed against large datasets in MVCC databases, then complete data processing is achieved, but processing resources are monopolized for extended periods
Solution Approach 1:
The patent segments the long-running query into multiple shorter query executions that process data in chunks. Each chunk is processed independently and intermediate results are stored, allowing processing resources to be released between chunks rather than being monopolized for the entire duration of processing a large dataset.
3Ease of operation
If long-running queries are divided into shorter queries executed in chunks, then resource monopolization is reduced and intermediate results can be shared, but query processing complexity increases
Solution Approach 1:
The patent uses an ordered index structure that is prepared in advance to enable sequential access to data chunks. This preliminary organization of data allows the query processing system to efficiently divide and resume queries without requiring complex dynamic decision-making during execution, thus managing complexity while enabling resource sharing.
Data Source
AI summary
Disclosed herein are system, method, and computer program product embodiments for a query processing system. An embodiment operates by receiving a query associated with a table. An ordered index with entries that include key values that uniquely identifies a corresponding record in the table is determined. A last key value is initialized to store a minimum value for a data type of the key value. The query is executed as a union-all of a plurality of queries. While a count of selected records is less than or equal to a threshold quantity and at least one entry remains in the ordered index, an ordered next entry with a key value subsequent to the last key value in the index is identified, the corresponding record is selected from the table, the last key value is updated, the query is executed, and the count is incremented.