Asynchronous Task Queue Configuration in Database Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing task processing techniques in database systems are not scalable and face inefficiencies in node distribution for executing asynchronous tasks, leading to challenges in managing and reliably executing long-running tasks.
Innovation Solution
A task queue manager (TQM) is implemented to configure and manage task queues using a combined in-memory and persistence state, allowing for scalable task execution, decoupling task production from consumption in time and space, and providing a serverless API for application developers to specify task execution functions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If existing task processing techniques are used, then task execution can be performed, but scalability is insufficient and node distribution efficiency deteriorates
Solution Approach 1:
The task queue is divided into multiple shards distributed across different nodes. Each shard is independently managed, allowing tasks to be distributed and processed in parallel across multiple nodes. This segmentation enables horizontal scaling by simply adding more shards and nodes without increasing overall system complexity.
Solution Approach 2:
A task queue manager is introduced as an intermediary component that handles task distribution, node management, and coordination. The TQM abstracts the complexity of node distribution and task scheduling, providing a simplified interface for task submission while managing the underlying complexity of distributed execution across multiple nodes.
2Loss of time
If task production and consumption are coupled in time and space, then simpler coordination is possible, but latency in task completion increases
Solution Approach 1:
The system decouples task production and consumption by introducing a temporal dimension through the task queue. Tasks are submitted to the queue and can be processed at a different time by consumer nodes. This time-based decoupling allows producers and consumers to operate independently without direct coordination, reducing latency while maintaining operational simplicity.
3Speed
If in-memory task queue is used, then fast task access is achieved, but task persistence and recovery capability deteriorates
Solution Approach 1:
The system merges in-memory and persistent storage mechanisms into a unified task queue implementation. The task queue manager maintains task metadata and state information in memory for fast access while simultaneously persisting task definitions and critical state to durable storage. This combination provides both high-speed task access and reliable persistence, with automatic recovery capabilities from persistent storage if memory is lost.
Data Source
AI summary
A method includes decoding, by at least one hardware processor, an enqueue request received from a data process of a database system. The enqueue request includes a task. The task is enqueued in an in-memory task queue. An enqueue acknowledgment is encoded for transmission to the data process responsive to the enqueue request. The task is persisted in a storage location associated with the in-memory task queue. Initiate a lease of the task to a worker node in response to a lease request received from the worker node. A dequeue request is received from the worker node where the dequeue request indicates completion of the task by the worker node. The task is dequeued from the in-memory task queue based on the dequeue request.