Source Range Partition Discovery for Balanced Data Extract Jobs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data migration services face challenges in efficiently partitioning large data sets without adequate partitioning information, leading to issues such as unwanted data reorganization, excessive task scheduling, memory and processing problems, and uneven workloads.
Innovation Solution
A method for automated discovery of source range partitioning using a data extractor that employs a partitioning discoverer, sampler, profiler, and recommender to determine optimal data ranges for partitioning, even in the absence of predefined partitioning information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is extracted without predefined partitioning information, then data extraction can be performed, but data reorganization and processing inefficiencies occur
Solution Approach 1:
The system performs preliminary sampling of data from the source system before actual extraction. This sample is then used to automatically discover partitioning information and generate partitioning recommendations. By performing this analysis in advance on a sample rather than the complete dataset, the system avoids reorganization inefficiencies during the main extraction process while still achieving efficient partitioned data transfer.
Solution Approach 2:
The system enables self-service partitioning discovery where the data extractor automatically analyzes the source data structure, samples values, determines appropriate partition boundaries, and generates partitioning recommendations without requiring manual intervention or predefined partitioning information. This self-service approach eliminates the need for external partitioning information while avoiding processing inefficiencies through automated optimization.
2Adaptability or versatility
If automated partitioning discovery is implemented, then partitioning information can be determined without predefined data, but system complexity increases
Solution Approach 1:
The automated partitioning discovery system is segmented into distinct functional modules: a sampler that extracts sample data, a profiler that analyzes the sample and identifies partitioning columns, a partitioning discoverer that determines partition boundaries, and a recommender that generates final partitioning recommendations. This segmentation allows each component to perform its specific function independently, managing overall system complexity while maintaining high adaptability to various data structures.
Solution Approach 2:
The system introduces an intermediary sampling and analysis layer between the data extractor and the actual data extraction process. This intermediary component (comprising the sampler, profiler, and partitioning discoverer) processes a representative sample to infer partitioning characteristics, acting as a mediator that translates raw data into structured partitioning recommendations without requiring direct complex interaction with the complete dataset.
3Productivity
If data is partitioned without proper boundary determination, then data can be distributed, but workload distribution becomes uneven
Solution Approach 1:
The system incorporates feedback mechanisms where the profiler continuously analyzes sampled data values and adjusts partition boundary recommendations accordingly. The partitioning discoverer evaluates the distribution of values and refines partition boundaries to ensure balanced workload distribution. This feedback loop ensures that partitioning recommendations are iteratively improved to achieve both efficient data transmission and balanced processing across all partitions.
Solution Approach 2:
The system dynamically determines partition boundary values by analyzing statistical parameters of the sampled data, such as mean, median, and standard deviation. It adjusts partition boundaries based on the actual data distribution characteristics, changing parameters like the number of partitions and boundary thresholds to optimize both transmission speed and workload balance. This parameter adaptation ensures even distribution while maintaining high productivity.
Data Source
AI summary
Techniques are described for the discovery of source range partitioning information. An example method includes a device determining a partition boundary value for the data based at least in part on the following steps. The device can determine a first plurality of bounded value sets and a second plurality of bounded value sets. The device can calculate a first average value of a first value and a second average value. The device can determine a first deviation value of the first average value from the first value and a second deviation value of the second average value from a third value. The device can determine the first partition boundary value based at least in part on the first deviation value and the second deviation value, the first partition boundary value being the first candidate partition boundary value or the second candidate partition boundary value.