In-Database Sharded Queue for Shared-Disk Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional message queues do not scale well, leading to increased contention for 'hot' messages at the head of the queue and excessive communication between systems when enqueue and dequeue sessions are distributed across multiple systems.
Innovation Solution
Implementing a sharded queue system where messages are distributed across multiple shards, with each enqueue session assigned to a specific shard to reduce contention, and using partitioned tables and message caches to manage message distribution and retrieval efficiently across multiple database instances.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If a conventional message queue is used, then process-to-process communication is facilitated, but contention for hot messages at the head of the queue increases as the number of dequeue sessions increases
Solution Approach 1:
The queue is divided into multiple shards (first shard, second shard, third shard, etc.), each maintaining its own head messages. Dequeue sessions are assigned to specific shards, distributing the contention for hot messages across multiple independent segments rather than having all sessions compete for a single queue head.
2Adaptability or versatility
If enqueue and dequeue sessions are spread across several systems, then system distribution is achieved, but communication on the interconnect between systems becomes excessive
Solution Approach 1:
Each database instance maintains local shards and can perform enqueue and dequeue operations locally without requiring interinstance communication. Only when a dequeue session needs messages from a shard not locally available does interinstance communication occur, minimizing unnecessary network traffic while maintaining system distribution.
3Productivity
If messages are distributed across multiple shards, then contention is reduced and scalability improves, but system complexity increases
Solution Approach 1:
The sharded queue system is implemented within the existing database management system, reusing database tables, storage structures, and query mechanisms. This allows the queue to leverage the database's existing capabilities for persistence, transaction management, and concurrency control, reducing the need for separate complex queue management infrastructure.
Data Source
AI summary
An in-database sharded queue for a shared-disk database is provided. First messages from at least one first enqueuer are stored on a first queue shard. The first enqueuer/s execute in a first instance of a shared-disk database system. The first shard includes a first plurality of subshards. Second messages from at least one second enqueuer are stored on a second queue shard. The second enqueuer/s execute in a second instance of the shared-disk database system. The second shard includes a second plurality of subshards. In volatile memory of the respective instances, a first message cache and a second message cache is maintained for the first shard and the second shard. The respective message caches include at least one buffer associated with the plurality of subshards.