Database Batch Request Parallelism via Selective Dequeuing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In database systems, batch requests often lead to reduced parallelism due to resource collisions when multiple processing managers attempt to lock the same resources, even when hardware resources are available, resulting in decreased efficiency and performance.
Innovation Solution
Implementing a job queue manager with a dequeue agent that selectively dequeues requests from different batches, allowing processing managers to access different resources concurrently, thereby reducing collisions and enhancing parallelism.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If multiple processing managers operate in parallel to process batch requests, then processing speed increases, but resource collisions occur when managers attempt to lock the same resources
Solution Approach 1:
The patent segments batch requests into individual job items and distributes them to multiple processing managers. Each manager processes independent job items from different batches, dividing the workload to prevent resource collisions while maintaining parallel processing capability.
Solution Approach 2:
The patent introduces a job queue manager as an intermediary component that coordinates between batch requests and processing managers. This mediator selectively dequeues job items from different batches and assigns them to available managers, preventing direct resource conflicts.
2Reliability
If batch requests are processed sequentially one at a time, then resource collisions are avoided, but processing efficiency decreases
Solution Approach 1:
The patent segments batch requests into individual processable job items and distributes them across multiple parallel managers. This segmentation allows concurrent processing of different batches without resource conflicts, significantly improving throughput while maintaining data integrity through proper locking mechanisms.
Solution Approach 2:
The job queue manager performs preliminary actions by pre-organizing batch requests into a structured queue format with metadata about resource requirements. This preliminary organization enables efficient distribution to managers and helps predict resource needs before conflicts occur.
3Device complexity
If all processing managers access jobs from the same batch, then workload distribution is simple, but parallelism is reduced due to resource contention
Solution Approach 1:
The patent implements dynamic workload distribution where the job queue manager continuously monitors batch progress and resource availability. Managers dynamically receive job items from different batches based on real-time conditions, optimizing parallelism without requiring complex manual coordination.
Solution Approach 2:
The system incorporates feedback mechanisms where processing managers report job completion status and resource usage to the job queue manager. This feedback enables intelligent redistribution of remaining jobs to maximize parallel processing efficiency while avoiding resource conflicts.
Data Source
AI summary
A system selectively dequeues journal requests instead of dequeuing on a first come, first served basis. A database system detects that a request for access from a client is a journal batch, which includes multiple journal requests. The system queues the journal batch in a queue with other journal batches for processing. The system selectively dequeues journal requests from the journal batch queue, including selecting at least one journal request from multiple different journal batches. Thus, the journal requests do not conflict for shared resources, and the system can improve parallelism in processing the journal batches.