Batch Record Lock Management via Segmented Commit Points
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Long-running batch jobs in data systems often hold record locks for extended periods, leading to timeouts for other transactions waiting for these resources, particularly in transactional environments like financial record management.
Innovation Solution
Implementing a method where a batch application processes a group of records, locking the first record only and committing all records upon completion of the last record in the group, with the option to commit earlier if a specified commit count is reached or if pending record lock requests exceed a threshold.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If a batch application processes many records without commit points, then the batch job can be simpler to implement, but record locks are held for extended periods causing timeouts to other transactions
Solution Approach 1:
The batch application's record processing is segmented into groups of N records. Instead of processing all records in a single transaction, the system divides the work into multiple segments, each followed by a commit point. This segmentation allows record locks to be released periodically, reducing the duration locks are held while maintaining batch processing functionality.
2Reliability
If a batch application holds record locks for the entire job duration, then data consistency is maintained, but other transactions experience timeouts and reduced productivity
Solution Approach 1:
The transaction is divided into multiple smaller transactions separated by commit points. Each segment processes N records and commits, maintaining data consistency within each segment while allowing other transactions to access released records. This segmentation resolves the conflict between maintaining consistency and enabling system throughput.
Solution Approach 2:
Commit points are introduced periodically during batch processing, creating a rhythm of lock acquisition, processing, and lock release. This periodic action ensures data consistency is maintained within each commit interval while periodically freeing resources for other transactions, thereby improving overall system productivity.
3Loss of time
If record locks are released early during batch processing, then other transactions can access records faster, but the batch application may lose data consistency
Solution Approach 1:
The system performs preliminary actions by establishing commit points at predetermined intervals before processing completes. These pre-planned commit points ensure that locks are released at controlled intervals, maintaining data consistency within each commit boundary while preventing excessive lock duration. The preliminary structuring of commit points balances consistency and accessibility.
Data Source
AI summary
In one embodiment, a method includes receiving, at a job control manager, a command specifying execution of a batch application. The method also includes receiving, at the job control manager, a commit count associated with the batch application, and initiating the batch application, with the batch application processing a group of records. The method also includes locking a first record of the group of records in response to the first record being processed by the batch application with the proviso that the batch application does not cause the first record to be unlocked unless the batch application has finished processing a last record in the group of records. Moreover, the method includes committing, in response to the batch application having completed processing of an nth record of the group of records, all records of the group of records that are locked resulting from execution of the batch application.