Bloom Filter Batch Materialization for CDC Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Batch materialization of large datasets in distributed computing architectures often requires resource-intensive shuffle operations, which are time-consuming and risk missing service-level agreements (SLAs).
Innovation Solution
The method involves extracting primary keys from an incremental change data capture (CDC) changeset, adding them to Bloom filters, and broadcasting these filters to executors. Each executor filters a baseline data table to generate match and unmatched dataframes, allowing the incremental changes to be applied directly to the unmatched dataframe without the need for shuffle operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If shuffle operations are used for batch materialization of large datasets, then data merging can be completed, but resource consumption increases and processing time extends
Solution Approach 1:
The patent segments the baseline data table into two distinct dataframes: matched dataframe (containing rows with primary keys present in the CDC changeset) and unmatched dataframe (containing rows with primary keys not present in the CDC changeset). This segmentation eliminates the need for shuffle operations by allowing direct application of changes to the matched dataframe while preserving the unmatched dataframe, thereby reducing resource consumption and processing time.
2Reliability
If traditional merge logic is applied to large datasets, then data changes can be integrated, but the process takes hours and risks missing SLAs
Solution Approach 1:
The patent performs preliminary filtering of the baseline data table before the merge operation by identifying and separating matched and unmatched rows based on primary keys from the CDC changeset. This preliminary action prepares the data in advance, allowing the actual merge to be completed rapidly by simply applying changes to the pre-identified matched dataframe, thereby ensuring SLA compliance and reducing materialization time.
3Productivity
If full row changes are applied without filtering, then all data can be updated, but unnecessary data processing increases resource usage
Solution Approach 1:
The patent applies local quality by treating matched and unmatched data differently: the matched dataframe receives full row changes from the CDC changeset, while the unmatched dataframe is preserved as-is without unnecessary processing. This selective application of updates ensures that computational resources and energy are consumed only where changes are actually needed, improving processing efficiency and reducing energy consumption.
Data Source
AI summary
Systems and methods are described for batch materialization of an incremental change data capture (CDC) changeset with full row changes. The primary keys are extracted from the incremental CDC changeset and an indication of the extracted primary keys are broadcast to a plurality of executors. The primary keys may be added to Bloom filter or a plurality of Bloom filters that are broadcast to the executors. Each executor filters a baseline data table based on the extracted primary keys to generate a baseline match dataframe with all primary keys matching the extracted primary keys, and a baseline unmatched dataframe with all primary keys not matching the extracted primary keys. Each executor receives full row changes from a partitioned incremental CDC changeset and combines the changes with the baseline unmatched dataframe to produce a final changed baseline data table.