Database Split Generation for Distributed Query Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed database environments, existing systems like Apache Hive face high latency due to batch processing overhead and inconsistent reads across multiple query executions, which can violate read-consistent rules, especially when handling large datasets.
Innovation Solution
A system and method for dynamically generating database splits based on table properties and user preferences, using a database table accessor to divide queries into query splits and associate them with system change numbers (SCNs) for consistent execution across multiple mappers, optimizing query processing and reducing latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Power
If batch processing is used to handle large datasets in Apache Hive, then data processing capability is improved, but latency increases due to substantial overhead in job submission and scheduling
Solution Approach 1:
The patent segments the batch processing job into multiple smaller sub-tasks that can be executed in parallel. Instead of submitting one large batch job that incurs substantial overhead, the system divides the data processing workload into finer granularities, allowing concurrent execution and reducing overall latency while maintaining processing power.
Solution Approach 2:
The system performs preliminary actions by pre-processing and preparing data in advance before the main batch processing occurs. This includes pre-computing intermediate results, pre-loading necessary data into memory, and pre-establishing execution plans, thereby reducing the actual processing latency when batch jobs are submitted.
2Productivity
If queries are divided into multiple pieces and executed separately against a database, then processing large datasets is improved, but read consistency is violated as queries are executed at different points in time
Solution Approach 1:
The system implements feedback mechanisms where each query execution reports its results and timing information back to a central coordinator. This allows the system to detect and correct consistency violations by comparing results across multiple query executions and ensuring they reflect a consistent snapshot of the database state.
Solution Approach 2:
An intermediary component is introduced that acts as a mediator between the divided queries and the database. This intermediary manages the execution timing, coordinates snapshots, and ensures that all query pieces operate on consistent data versions, thereby maintaining read consistency while allowing parallel execution.
3Ease of operation
If data is transferred from JDBC to data warehouse layer in distributed database environment, then data accessibility is improved, but latency is high due to existing batch processing overhead
Solution Approach 1:
The system maintains continuous data transfer operations from JDBC to the data warehouse layer by implementing streaming mechanisms and avoiding complete batch processing interruptions. Data is transferred continuously in smaller chunks, keeping the data pipeline active and reducing overall transfer latency while maintaining accessibility.
Solution Approach 2:
The data transfer mechanism is made dynamic by adjusting transfer parameters such as batch size, parallelism degree, and resource allocation based on real-time system conditions. This allows the system to optimize transfer latency dynamically while ensuring data remains accessible, adapting to changing workloads and resource availability.
Data Source
AI summary
A system and method is described for database split generation in a massively parallel or distributed database environment including a plurality of databases and a data warehouse layer providing data summarization and querying functionality. A database table accessor of the system obtains, from an associated client application, a query for data in a table of the data warehouse layer, wherein the query includes a user preference. The system obtains table data representative of properties of the table, and determines a splits generator in accordance with one or more of the user preference or the properties of the table. The system generates, by the selected splits generator, table splits dividing the user query into a plurality of query splits, and outputs the plurality of query splits to an associated plurality of mappers for execution by the associated plurality of mappers of each of the plurality of query splits against the table.