Database Batch Processing Segmentation for Data Integrity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current batch processing methods are inefficient due to the need to move and process entire record sets, including irrelevant data, leading to increased resource usage and processing time, especially with large datasets.
Innovation Solution
The Detached Field Batch Process (DFBP) and Horizontal Processing of Sequential Data (HPSD) methods, which detach irrelevant fields during processing and process data horizontally instead of vertically, respectively, to reduce resource usage and processing time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional batch processing methods are used to process entire record sets, then data integrity is maintained, but processing time and resource usage increase significantly
Solution Approach 1:
The patent segments the record processing into two distinct phases: first processing only the key fields (segment 1) to identify matching records, then processing the complete records (segment 2) only for matched pairs. This segmentation allows the system to reduce I/O operations by avoiding movement of entire records during the initial matching phase, thereby reducing processing time while maintaining data integrity through the two-phase approach.
Solution Approach 2:
The patent extracts and processes only the essential key fields from the complete records during the first phase. By taking out just the necessary key components for matching purposes and leaving the full record processing for the second phase, the system minimizes unnecessary I/O operations on large datasets while ensuring complete data integrity is maintained when the full records are processed for matched pairs.
2Adaptability or versatility
If traditional batch processing methods move entire record sets, then all data is available for processing, but memory and disk usage increase
Solution Approach 1:
The patent segments data handling into key field processing and complete record processing. During the first phase, only key fields are loaded into memory and processed, significantly reducing memory and disk usage. The complete records are only loaded and processed in the second phase for matched pairs, ensuring that full data availability is maintained when needed while minimizing resource consumption during the matching phase.
Solution Approach 2:
The patent applies partial action by processing only the necessary key portions of records during the first phase rather than complete records. This partial processing approach reduces memory and disk usage significantly while still achieving the matching objective. The complete record processing is performed only for matched pairs in the second phase, avoiding excessive resource usage on non-matching records.
3Stability of the object's composition
If sequential row-by-row processing is used, then data is processed in order, but processing speed decreases for large datasets
Solution Approach 1:
The patent merges multiple row processing operations into a single database scan by utilizing the database management system's native ability to join tables based on key fields. Instead of sequentially processing rows one by one with multiple scans, the system combines the processing of key fields and matching logic into a single efficient database operation, maintaining proper processing order while dramatically improving processing speed for large datasets.
Data Source
AI summary
A method is provided for processing data in a database, wherein the database includes a first table and a second table. Each of the tables have a plurality of rows of data, wherein a key identifies one or more rows of data in the tables. There is a plurality of matching key pairs among the tables. Each key pair includes a key of a first table, and a matching key and a unique row of a second table. In operation, the method involves simultaneously processing row data of key pairs associated with a first row of the second table, and then simultaneously processing row data of key pairs associated with any remaining rows of the second table in sequential row order.