Opportunistic Wait-Triggered Elastic Commit for Micro-Batch Transactions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge of reducing contention between batch and online transaction processing (OLTP) workloads in shared datasets, particularly exacerbated by competing workloads in 24/7 business models, where micro-batches retain locks leading to significant wait times and cascading performance issues.
Innovation Solution
A method that determines whether to trigger an early commit of a micro-batch transaction by using a record retrieval get record call, processing the response to decide if the record is available or a flag is received, and based on that, deciding to commit the transaction early to reduce contention.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If micro-batches retain locks on records until commit to ensure data consistency, then transaction reliability is improved, but wait times for competing workloads increase and system throughput deteriorates
Solution Approach 1:
The patent implements dynamic lock retention by allowing micro-batches to hold locks until commit under normal conditions, but enabling early commit when contention is detected. This dynamic adjustment of lock retention policy resolves the contradiction between maintaining reliability through lock retention and reducing wait times through early commit when necessary.
Solution Approach 2:
The system changes the commit timing parameter based on contention detection. When a micro-batch encounters a locked record, the system evaluates whether to proceed with early commit of previous records, effectively changing the commit timing parameter from fixed (end of micro-batch) to variable (early when contention detected), thereby reducing wait times while maintaining data consistency.
2Productivity
If micro-batch size is increased to improve processing efficiency, then productivity is improved, but contention with other workloads increases and system performance deteriorates
Solution Approach 1:
The patent segments batch processing into micro-batches that can be committed independently. When contention is detected on a specific record, only the affected micro-batch is held back while others can commit early, allowing larger overall batch sizes to maintain productivity without proportionally increasing system-wide contention.
Solution Approach 2:
The system allows partial commit of micro-batches when contention is detected. Instead of waiting for entire micro-batches to complete, the system commits records that don't cause contention while holding back only the specific records causing conflicts, enabling larger batch sizes to proceed with reduced overall contention impact.
3Stability of the object's composition
If micro-batches wait for locks held by other workloads to maintain proper processing order, then data consistency is improved, but cascading wait times increase and system throughput deteriorates
Solution Approach 1:
The patent implements feedback-based commit decision-making where micro-batches monitor for locked records and receive feedback about contention status. When a micro-batch encounters a locked record, the system evaluates the situation and determines whether early commit of other records is appropriate, using this feedback to maintain data consistency while minimizing cascading waits.
Solution Approach 2:
The system performs preliminary evaluation of contention before forcing micro-batches to wait. When a locked record is detected, the system proactively evaluates whether early commit can occur for other records in the micro-batch, taking preliminary action to prevent unnecessary cascading waits while maintaining data consistency through controlled lock retention.
Data Source
AI summary
A method is described for determining whether to trigger an early commit of a micro-batch transaction. The method includes, during the micro-batch transaction, requesting access to a record using a record retrieval get record call. The method further includes receiving a response to the record retrieval get record call, and determining whether the response includes the record or a flag. The record is processed in response to a determination that the response includes the record. Additionally, in response to a determination that the response includes the flag, it is determined whether to trigger an early commit of the micro-batch transaction.