Rowid Range-Based Database Splits for Distributed Query Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In massively parallel or distributed database environments, existing systems face high latency and inconsistent reads due to batch processing and the division of queries into multiple pieces, which can violate read-consistent rules, especially when dealing with large datasets in Hadoop frameworks like Apache Hadoop and Apache Hive.
Innovation Solution
A system and method for dynamically generating database splits based on table properties and user preferences, using a database table accessor to select appropriate splits generators and generate query splits, which can include partition-based, size-based, or ROWID-based splits, and associating System Change Number (SCN) data with query splits to ensure consistent reads across tasks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If queries are divided into multiple pieces and executed separately in batch processing, then processing capacity is improved, but read consistency is violated and latency increases
Solution Approach 1:
The patent applies preliminary action by obtaining the current System Change Number (SCN) before dividing the query into multiple pieces. This SCN is then associated with each query split, establishing a time reference point that ensures all subsequent reads maintain consistency relative to this point in time, preventing read consistency violations that would otherwise occur with parallel batch processing
2Productivity
If batch processing is used to handle large datasets, then processing throughput is improved, but job submission and scheduling overhead increases latency
Solution Approach 1:
The patent extracts the SCN acquisition step from the main query execution flow and performs it preliminarily before query splitting. This separation allows the system to maintain batch processing throughput while reducing latency by avoiding repeated SCN acquisitions during query execution and eliminating scheduling overhead for sequential consistency checks
3Productivity
If dynamic split generation based on table properties and user preferences is implemented, then query processing optimization is improved, but system complexity increases
Solution Approach 1:
The patent applies parameter changes by dynamically adjusting query split parameters based on table properties (such as partition information, data distribution) and user preferences. The system modifies split granularity, split count, and distribution strategies according to these parameters, enabling optimized query processing without requiring fundamental changes to the underlying system architecture
Data Source
AI summary
A system and method is described for database split generation in a massively parallel or distributed database environment including a plurality of databases and a data warehouse layer providing data summarization and querying functionality. A database table accessor of the system obtains, from an associated client application, a query for data in a table of the data warehouse layer, wherein the query includes a user preference. The system obtains table data representative of properties of the table, and determines a splits generator in accordance with one or more of the user preference or the properties of the table. The system generates, by the selected splits generator, table splits dividing the user query into a plurality of query splits, and outputs the plurality of query splits to an associated plurality of mappers for execution by the associated plurality of mappers of each of the plurality of query splits against the table.