Asynchronous Task Queue Leasing for Scalable Database Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing task processing techniques in database systems are not scalable and suffer from inefficient node distribution for executing asynchronous tasks, leading to challenges in managing and executing long-running tasks effectively.
Innovation Solution
A task queue manager (TQM) is implemented to manage asynchronous tasks, utilizing a combined in-memory and persistence state configuration that allows for scalable task distribution and execution, with features like task enqueuing, leasing, and dequeuing, and a serverless API for application developers to specify task execution details.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If existing task processing techniques are used, then task execution can be performed, but scalability is limited and node distribution is inefficient
Solution Approach 1:
The task queue is divided into multiple shards distributed across different nodes. Each shard independently manages a portion of the tasks, allowing the system to scale horizontally by adding more shards and nodes. This segmentation resolves the contradiction by enabling both efficient task execution within each shard and system-wide scalability through distributed architecture.
Solution Approach 2:
The system introduces a new dimension of task management by implementing a distributed task queue architecture with multiple shards across multiple nodes. This dimensional expansion from a single queue to a distributed shard-based system enables simultaneous improvement in execution efficiency (through parallel processing) and scalability (through horizontal expansion).
2Loss of information
If task tracking is implemented by producers, then task generation can be monitored, but the process becomes challenging and time-consuming
Solution Approach 1:
The task queue manager acts as an intermediary between task producers and the execution system. It centralizes task tracking and status management, providing producers with simplified APIs to monitor task progress without requiring complex tracking logic. This intermediary role resolves the contradiction by maintaining full task progress visibility while abstracting away the management complexity.
Solution Approach 2:
The task queue system implements self-service mechanisms where the queue manager automatically tracks, monitors, and manages task states without requiring manual intervention from producers. The system provides built-in introspection capabilities that automatically report task progress, resolving the contradiction by maintaining information visibility while eliminating the time-consuming manual tracking burden.
3Productivity
If uniform scale-out of consumption resources is enabled, then resource distribution is optimized, but task queue management complexity increases
Solution Approach 1:
The task queue is segmented into multiple shards that can be independently distributed across consumption resources. This segmentation enables uniform scale-out by allowing each node to manage a consistent portion of the total workload, optimizing resource utilization while keeping individual node management simple and manageable.
Solution Approach 2:
The system changes the management parameter from centralized queue control to distributed shard-based control. Each shard is managed independently with consistent parameters, enabling uniform scale-out across resources. This parameter change resolves the contradiction by optimizing resource utilization through distributed management while avoiding the complexity of centralized control at scale.
Data Source
AI summary
Techniques for configuring an asynchronous task queue are disclosed. An example method includes enqueuing, by at least one hardware processor of a first computing node, a task in a task queue. The at least one hardware processor detects a lease request for the task. The lease request is received from a second computing node. The at least one hardware processor initiates a lease of the task to the second computing node. The at least one hardware processor updates a state of the task based on the lease. The at least one hardware processor determines whether to dequeue the task based on a task processing update received from the second computing node.