Partition-Based Query Split Generation for Distributed Database Consistency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In massively parallel or distributed database environments, existing systems face high latency and inconsistent reads due to batch processing and the division of queries into multiple pieces, which can violate read-consistent rules, especially when dealing with large datasets in systems like Apache Hadoop and Apache Hive.
Innovation Solution
A system and method for dynamically generating query splits based on table properties and user preferences, using a database table accessor to select appropriate splits generators and generate query splits, which can include associating System Change Numbers (SCNs) with each split to ensure consistent reads across tasks, and optimizing query processing with predicate pushdown and efficient connection management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If queries are divided into multiple pieces for parallel processing, then processing throughput is improved, but read consistency is violated
Solution Approach 1:
The patent segments queries into multiple query splits based on partition information, allowing parallel processing while maintaining consistency through systematic division of the query workload across multiple processing units
Solution Approach 2:
The patent implements feedback mechanisms where query execution results are collected and aggregated systematically, ensuring that the final combined result maintains read consistency by verifying that all splits read from consistent data states
2Use of energy by moving object
If batch processing is used for large datasets, then resource utilization is improved, but query latency increases
Solution Approach 1:
The patent dynamically adjusts processing strategies based on query characteristics and data distribution, allowing the system to switch between batch processing and more responsive approaches, thereby reducing latency while maintaining efficient resource utilization
Solution Approach 2:
The patent performs preliminary actions by pre-processing and caching partition metadata and statistics, allowing the system to quickly generate query splits without full batch processing overhead, thus reducing latency while maintaining resource efficiency
3Productivity
If partition-based splits are generated dynamically, then query optimization is improved, but system complexity increases
Solution Approach 1:
The patent implements self-service mechanisms where the system automatically generates query splits based on partition information without requiring complex manual configuration, reducing the perceived complexity for users while maintaining optimization benefits
Solution Approach 2:
The patent creates a universal query split generation framework that handles different partition types and query patterns through a unified approach, reducing system complexity by eliminating the need for separate handling logic for different scenarios
Data Source
AI summary
A system and method is described for database split generation in a massively parallel or distributed database environment including a plurality of databases and a data warehouse layer providing data summarization and querying functionality. A database table accessor of the system obtains, from an associated client application, a query for data in a table of the data warehouse layer, wherein the query includes a user preference. The system obtains table data representative of properties of the table, and determines a splits generator in accordance with one or more of the user preference or the properties of the table. The system generates, by the selected splits generator, table splits dividing the user query into a plurality of query splits, and outputs the plurality of query splits to an associated plurality of mappers for execution by the associated plurality of mappers of each of the plurality of query splits against the table.