Bloom Filter Batch Materialization for Shuffle-Free CDC Merges
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Batch materialization of large datasets in distributed computing architectures is hindered by the need for resource-intensive shuffle operations, which are time-consuming and risk missing service-level agreements (SLAs).
Innovation Solution
The method involves extracting primary keys from an incremental change data capture (CDC) changeset, adding them to Bloom filters, and broadcasting these filters to executors. Each executor filters a baseline data table to generate match and unmatched dataframes, allowing changes to be applied efficiently without requiring a shuffle operation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If shuffle operation is used for batch materialization of large datasets, then complete merging of changes can be achieved, but processing time and resource consumption increase significantly
Solution Approach 1:
The patent extracts only the primary keys from the changeset data rather than processing the entire dataset. By taking out just the essential identification elements (primary keys), the system can perform filtering operations without the computational overhead of shuffling and merging complete records, thus reducing processing time while maintaining data completeness.
Solution Approach 2:
The patent segments the materialization process into distinct phases: extracting primary keys, broadcasting filters, filtering baseline data, and applying changes. This segmentation allows each phase to be optimized independently, avoiding the need for a single resource-intensive shuffle operation and enabling more efficient parallel processing.
2Reliability
If shuffle operation is used for batch materialization, then all data changes can be merged accurately, but resource consumption and cost increase
Solution Approach 1:
The patent performs preliminary actions by extracting primary keys and creating filter structures before the actual materialization process. By preparing the filtering mechanism in advance, the system avoids the need for resource-intensive shuffle operations during the main processing phase, reducing overall computational resource consumption while ensuring accurate data merging.
Solution Approach 2:
The patent introduces an intermediary filtering mechanism that uses primary keys as a mediator between the changeset and baseline data. This intermediary layer enables accurate matching and merging without requiring the expensive shuffle operation, thus reducing resource consumption while maintaining merging accuracy.
3Reliability
If traditional merge logic is used with large datasets, then comprehensive data integration can be achieved, but the process takes hours and lacks control over completion time
Solution Approach 1:
The patent replaces the traditional mechanical shuffle and merge operations with a filtering-based approach. Instead of physically shuffling and comparing entire datasets, the system uses filter structures built from primary keys to efficiently identify and integrate only the necessary data changes, dramatically improving processing speed while maintaining integration completeness.
Solution Approach 2:
The patent changes the fundamental parameter of processing from operating on complete records to operating on primary keys and filter structures. This parameter change reduces the data volume requiring processing from terabytes to much smaller key sets, enabling faster materialization while ensuring all necessary data integrations are captured.
Data Source
AI summary
Systems and methods are described for batch materialization of an incremental change data capture (CDC) changeset. The primary keys are extracted from the incremental CDC changeset and an indication of the extracted primary keys are broadcast to a plurality of executors. The primary keys may be added to Bloom filter or a plurality of Bloom filters that are broadcast to the executors. Each executor filters a baseline data table based on the extracted primary keys to generate a baseline match dataframe with all primary keys matching the extracted primary keys, and a baseline unmatched dataframe with all primary keys not matching the extracted primary keys. Each executor receives a partitioned incremental CDC changeset and combines the changes with the baseline match dataframe to produce a final changed baseline data table.