Batch Processing Method Using Sequence Keys for Parallel Row Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current batch processing methods in relational database management systems are extremely slow due to sequential data reading, making it impractical to process large datasets with billions of rows within a reasonable timeframe.
Innovation Solution
A new batch processing method that uses relational database tables without sequential reading, employing additional keys to identify row sequences and processing multiple rows simultaneously, significantly improving processing speed by up to 300 times compared to existing cursor processing methods.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If cursor processing method is used to fetch data row by row sequentially, then data processing can be performed in batch mode, but processing speed becomes extremely slow (only 10,000 to 20,000 rows per minute)
Solution Approach 1:
The patent segments the batch processing task by introducing a sequence number column that divides the sequential row-by-row processing into parallelizable segments. By adding a sequence number dimension to the data structure, the system can process multiple rows simultaneously while maintaining the logical sequence required for batch processing, thereby resolving the contradiction between sequential processing requirements and speed requirements.
Solution Approach 2:
The patent adds a new dimension (sequence number column) to the existing data structure, transforming the one-dimensional sequential processing approach into a multi-dimensional processing approach. This allows the system to maintain batch processing logic while enabling parallel execution across multiple rows, achieving both correctness and performance.
2Productivity
If sequential reading of data is used in batch processing, then data can be processed in the traditional batch mode, but processing large datasets with billions of rows becomes impossible within a reasonable timeframe
Solution Approach 1:
The patent segments the large dataset by introducing sequence numbers that allow dividing the processing into independent parallel tasks. This segmentation enables the system to handle billions of rows by processing them in parallel across multiple sequence groups, thereby increasing both throughput and the quantity of processable data.
Solution Approach 2:
The patent introduces dynamic parallel processing by allowing the system to adaptively process different sequence numbers concurrently. The processing logic dynamically adjusts to handle multiple sequence groups in parallel, transforming the static sequential processing model into a dynamic parallel model that can scale with data size.
3Ease of manufacture
If legacy mainframe batch processing systems are used, then existing business logic can be maintained, but maintenance costs are expensive and human resources are diminishing
Solution Approach 1:
The patent replaces the mechanical mainframe batch processing system with a database-centric SQL-based system. By substituting the legacy mechanical processing approach with modern database operations, the system maintains business logic while reducing maintenance complexity and cost, and enabling greater automation through standard SQL batch processing capabilities.
Data Source
AI summary
A computer program product is provided wherein a batch process operates on structured data in multiple tables, including a first and a second source table, and a first and a second processing table. An additional key is added within each natural key to at least the second source and processing table for identifying a sequence of rows for updating.