Tabular Data Merge with Matched-Row Filtering and Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Updating datasets at scale is computationally expensive and time-consuming due to inefficient data merging processes that fail to preserve data clustering and result in excessive network transfer and computationally expensive join operations.
Innovation Solution
A system that selectively transfers and processes matched rows of data, performing targeted join operations and reconfiguring file sizes to optimize data merging, while preserving data locality and reducing unnecessary data transfer.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional data merging processes are used to update datasets at scale, then comprehensive data integration is achieved, but computational cost and processing time increase significantly
Solution Approach 1:
The patent segments the data merging process into distinct phases: identifying matched rows between source and target tables, performing join operations only on matched rows, and reconfiguring file sizes. This segmentation avoids processing entire datasets, significantly reducing computational overhead and processing time while maintaining data integration completeness.
Solution Approach 2:
The patent performs preliminary identification of matched rows between source and target tables before executing join operations. By pre-filtering and organizing data that requires merging, the system avoids unnecessary computational work on non-matching records, thereby reducing overall processing time and improving merging efficiency.
2Productivity
If traditional data merging processes are used, then complete data integration is achieved, but network transfer volume increases excessively
Solution Approach 1:
The patent extracts and processes only the matched rows from source and target tables that require merging, rather than transferring and processing entire datasets. This extraction approach minimizes network transfer volume by focusing computational resources only on the necessary data subsets, reducing energy consumption and improving merging efficiency.
3Productivity
If join operations are performed on all data rows, then complete data integration is achieved, but computational cost increases excessively
Solution Approach 1:
The patent applies partial action by performing join operations only on matched rows identified in the preliminary phase, rather than executing joins on all data rows. This selective approach reduces computational cost and energy consumption while still achieving complete data integration for the relevant subsets, improving overall merging efficiency.
4Adaptability or versatility
If data is transferred and processed without preserving clustering, then merging flexibility is improved, but read performance deteriorates
Solution Approach 1:
The patent preserves data clustering and locality characteristics within the processed data subsets during the merging process. By maintaining the organizational structure and spatial relationships of data rows that are actually being merged, the system ensures that read performance is not deteriorated while still achieving the flexibility needed for comprehensive data integration.
Data Source
AI summary
A method, system, and computer system for performing an operation with respect to a target table are disclosed. The method includes performing first, second and a third jobs, and obtaining a resulting table based at least in part on the second job resulting file(s) and third job resulting file(s). Performing the first job includes determining a set of matching target table files and storing target table information indicating for each of the set of matching target table files, a particular set of rows having matching rows. Performing the second job includes performing a matching action based on matched rows and obtaining the second job resulting file(s). Performing the third job includes determining unmatched rows for target table files and storing the unmatched rows in third job resulting file(s).