Big Data Table Merge with Post-Processing Compaction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing big data processing systems face inefficiencies in updating datasets at scale, as they often redistribute unmodified data, leading to high computational costs and inefficiencies in data clustering and processing.
Innovation Solution
The system dynamically associates unique identifiers with records to track matching data across target and source datasets, selectively transferring only necessary data across the network, and performs optimized file reconfiguration to reduce computational costs and maintain data clustering.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If existing big data processing systems update datasets at scale, then data is updated, but computational costs are very high and significant time is consumed
Solution Approach 1:
The patent extracts and identifies only the modified records from the source dataset that need to be applied to the target dataset, rather than processing the entire dataset. This is achieved through record matching using unique identifiers, which isolates the essential changes and eliminates unnecessary data processing, thereby reducing computational costs while maintaining update effectiveness.
Solution Approach 2:
The patent segments the data update process into distinct phases: identifying modified records through unique identifiers, matching these records between source and target datasets, and applying changes only where needed. This segmentation allows the system to process only relevant portions of the data, improving productivity without proportionally increasing computational energy consumption.
2Productivity
If existing systems redistribute unmodified data, then data is transferred, but this leads to high computational costs and inefficiencies
Solution Approach 1:
The patent extracts only the modified records from the source dataset using unique identifiers for matching, rather than redistributing the entire dataset including unmodified records. This extraction approach eliminates unnecessary data transfer, reducing both computational costs and time loss while ensuring that only essential changes are propagated to the target dataset.
3Loss of energy
If the system transfers only necessary data, then computational costs are reduced, but data clustering and processing optimization are challenged
Solution Approach 1:
The patent performs preliminary actions by dynamically associating unique identifiers with records before the actual data transfer and processing. This pre-establishment of identification relationships enables efficient matching and filtering of modified records, ensuring that only necessary data is transferred. The preliminary organization of data through unique identifiers maintains processing efficiency while reducing computational costs associated with transferring unnecessary data.
Data Source
AI summary
A method, system, and computer system for performing an operation with respect to a target table are disclosed. The method includes performing first and second jobs, obtaining one or more other resulting files based at least in part on unmatched rows, and obtaining a set of processed files based at least in part on performing a post-processing operation with respect to the set of resulting files. The set of processed files has less files than the set of resulting files. Performing the first job includes determining a set of matching target table files and storing target table information indicating for each of the set of matching target table files, a particular set of rows having matching rows. Performing the second job includes performing a matching action based on matched rows and obtaining the second job resulting file(s).