Iterative Primary Key Broadcast for Shuffle-Free CDC Materialization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Batch materialization of large datasets in distributed computing architectures is resource-intensive and time-consuming due to the need for shuffle operations, which are costly and lack control over completion times, risking missed service-level agreements.
Innovation Solution
Iterative broadcasting of primary keys from an incremental change data capture (CDC) changeset to executors, filtering the baseline data table into match and unmatched dataframes, and applying changes without requiring a shuffle operation, thereby reducing resource usage and completion time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If shuffle operations are used for batch materialization of large datasets, then data merging can be performed across distributed executors, but resource consumption increases and processing time becomes uncontrolled
Solution Approach 1:
The patent segments the large dataset into smaller partitions that can be independently processed and broadcast to executors. By dividing the data processing task into manageable chunks, the system avoids the resource-intensive shuffle operation while maintaining distributed processing capabilities across multiple executors.
Solution Approach 2:
The patent performs preliminary actions by pre-processing and preparing data partitions before the actual materialization process. Data is organized and staged in advance, allowing executors to work with pre-prepared datasets rather than performing complex shuffling during the critical materialization phase.
2Productivity
If shuffle operations are used for batch materialization, then data can be redistributed across executors, but processing time becomes uncontrolled and SLAs may be missed
Solution Approach 1:
By segmenting the dataset into fixed partitions, the patent enables predictable processing times for each partition. This segmentation allows the system to control and monitor processing time across multiple executors, ensuring SLA compliance while maintaining high data merging speed.
Solution Approach 2:
The preliminary organization and staging of data partitions before materialization enables the system to estimate and control processing times. By having data ready in advance in appropriate formats and locations, the actual materialization process can proceed with predictable timing, preventing SLA violations.
3Use of energy by moving object
If iterative broadcast is used instead of shuffle, then resource consumption is reduced, but handling large primary key dataframes becomes challenging
Solution Approach 1:
The patent applies segmentation by dividing large primary key dataframes into smaller manageable partitions for iterative broadcast. This segmentation enables the system to work within broadcast limitations while maintaining resource efficiency, as each partition can be broadcast independently without requiring the full dataset to be transmitted at once.
Solution Approach 2:
The iterative broadcast mechanism employs periodic action by transmitting data partitions in successive waves rather than all at once. This periodic transmission approach allows the system to manage large primary key dataframes within resource constraints, broadcasting partitions in controlled intervals that prevent memory overflow while maintaining overall processing efficiency.
Data Source
AI summary
Systems and methods are described for batch materialization of an incremental change data capture (CDC) changeset. The primary keys are extracted from the incremental CDC changeset and are broadcast to a plurality of executors. By iteratively broadcasting the extracted primary keys in groups, each being a size that is less than a broadcast limitation, the full dataframe of extracted primary keys is broadcast to the executors. Each executor filters a baseline data table based on the extracted primary keys to generate a baseline match dataframe with all primary keys matching the extracted primary keys, and a baseline unmatched dataframe with all primary keys not matching the extracted primary keys. Each executor receives a partitioned incremental CDC changeset and applies the changes to the baseline match dataframe to produce a baseline change dataframe, which is merged with the baseline unmatched dataframe to produce a final changed baseline data table.