Database Clustering Execution Manager for Skewed Batches
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database table organization and maintenance techniques result in increased processing times, inefficient use of computing resources, and higher costs due to manual reclustering operations, which are resource-intensive and can lead to processing delays caused by data skew.
Innovation Solution
A clustering execution manager (CEM) is employed to manage and scale clustering execution by identifying and isolating batches with similar sizes, using an iterative execution model to reduce skew and optimize the use of execution nodes, allowing for efficient and flexible management of clustering processes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If manual reclustering operations are performed to organize database tables, then data organization quality is improved, but processing time increases and computing resources are used inefficiently
Solution Approach 1:
The patent divides the database table into multiple partitions and processes them in batches. The clustering execution manager segments the reclustering operation into manageable units that can be processed independently and concurrently, improving both efficiency and resource utilization while maintaining data organization quality.
Solution Approach 2:
The patent implements dynamic batch size adjustment based on data skew detection. The system automatically adapts batch sizes during execution to optimize processing efficiency, reducing processing time while maintaining effective data organization through clustering.
2Manufacturing precision
If manual reclustering operations are performed to organize database tables, then data organization quality is improved, but computing resource utilization becomes inefficient
Solution Approach 1:
By segmenting the reclustering operation into partition-level batches, the system enables parallel execution across multiple computing nodes. This segmentation allows efficient utilization of computing resources while achieving the same data organization quality as manual full-table reclustering.
Solution Approach 2:
The system dynamically changes batch size parameters based on detected data skew. This adaptive parameter adjustment optimizes computing resource utilization by processing larger batches when data is evenly distributed and smaller batches when skew is present, reducing wasted computational energy.
3Productivity
If clustering execution is scaled to process more partitions concurrently, then productivity is improved, but data skew causes processing delays
Solution Approach 1:
The system dynamically adjusts batch sizes based on real-time detection of data skew. When skew is detected, batch sizes are reduced to minimize processing delays; when data is evenly distributed, larger batches are processed concurrently to maximize productivity. This dynamic adaptation resolves the contradiction between throughput and delay.
Solution Approach 2:
The clustering execution manager implements feedback mechanisms that monitor processing progress and detect data skew conditions. Based on this feedback, the system automatically adjusts subsequent batch processing strategies, enabling productive concurrent execution while mitigating skew-induced delays through adaptive control.
Data Source
AI summary
A method includes retrieving, by at least one hardware processor in a database system, a database table. The database table includes a plurality of partitions. A plurality of batches is generated for the database table based on a file selection task of the database system. Each batch of the plurality of batches includes a partition subset of the plurality of partitions. A plurality of execution jobs is configured based on an execution management task of the database system. Each execution job of the plurality of execution jobs includes a batch subset of the plurality of batches, and the skew of batch sizes for the batch subset is below a threshold skew. Concurrent execution of the plurality of execution jobs is performed to cluster the partition subset associated with each of the plurality of execution jobs.