Tabular Data Merge with Deletion Vectors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current big data processing systems face inefficiencies in updating datasets at scale, as they are computationally expensive and time-consuming, often redistributing unmodified data and failing to preserve data clustering, leading to high costs and latency in merging target and source data.
Innovation Solution
The system dynamically associates unique identifiers with records to track matching data across tables, selectively transfers data across the network based on matched rows, and performs optimized join operations, reconfiguring file sizes and processing data in batches to reduce computational costs and maintain data clustering.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional merge operations are performed on large datasets, then data updates can be completed, but computational costs and processing time increase significantly
Solution Approach 1:
The patent divides the merge operation into distinct phases: identifying matched rows between target and source tables, transferring only those matched rows, performing join operations on the filtered data, and finally merging results. This segmentation allows the system to process only necessary data portions rather than entire datasets, significantly reducing computational overhead and processing time while maintaining complete data update functionality.
2Productivity
If all target table data is transferred for merging, then complete data integration is achieved, but network data transfer volume and costs increase
Solution Approach 1:
The patent extracts and identifies matched rows between the target table and source table before initiating data transfer. By using unique identifiers and matching logic, the system determines exactly which rows from the target table have corresponding entries in the source table. Only these identified matched rows are then transferred across the network, eliminating unnecessary data movement while ensuring complete integration of all relevant data.
3Productivity
If data is redistributed during merging operations, then integration is achieved, but data clustering and locality are lost
Solution Approach 1:
The patent preserves data clustering by maintaining the original physical location and organization of data rows within each table. The matching and merge operations are performed based on logical identifiers rather than physical redistribution. This approach ensures that data remains localized in its original storage positions, preserving clustering properties and data locality while still achieving complete data integration through the join operations on matched rows.
Data Source
AI summary
A method, system, and computer system for performing an operation with respect to a target table are disclosed. The method includes performing first and second jobs, and persist, in one or more deletion vector files, one or more deletion vectors for corresponding rows of the one or more target table files, and obtaining a resulting table based at least in part on the second job resulting file(s). Performing the first job includes determining a set of matching target table files and storing target table information indicating for each of the set of matching target table files, a particular set of rows having matching rows. Performing the second job includes performing a matching action based on matched rows and one or more deletion of vectors associated with previously removed rows of the matching target table files and obtaining the second job resulting file(s).