In-Database Sharded Queue Memory Pressure Handling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional message queues do not scale well, leading to performance degradation due to increased contention for 'hot' messages at the head of the queue and excessive communication between systems, especially when enqueue sessions and dequeue sessions are spread across multiple systems, and are affected by low memory situations.
Innovation Solution
A sharded queue system is implemented, where a single queue is divided into multiple shards, with messages ordered by enqueue-time within subshards, and a message cache is used to store messages in volatile memory, dynamically adjusting caching based on memory availability to handle low memory situations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional message queues are used, then message communication between processes is enabled, but scalability deteriorates due to increased contention for hot messages at the head of the queue
Solution Approach 1:
The queue is divided into multiple shards (e.g., shard 0, shard 1, shard 2) that operate independently. Each shard maintains its own message queue and handles dequeuing operations separately, eliminating the single-point contention problem. Dequeue sessions are distributed across different shards, allowing parallel message consumption without interfering with each other.
2Adaptability or versatility
If enqueue sessions and dequeue sessions are spread across several systems, then system distribution is achieved, but communication overhead on the interconnect increases excessively
Solution Approach 1:
Each shard maintains local message queues and local dequeue operations, allowing systems to process messages locally without excessive interconnect communication. The sharding strategy enables local caching and processing, reducing the need for frequent inter-system message transfers and lowering interconnect overhead.
3Productivity
If messages are enqueued faster than they are dequeued, then message accumulation occurs, but memory consumption increases and low memory situations arise
Solution Approach 1:
The queue is segmented into multiple shards that can be independently managed and scaled. This segmentation allows the system to distribute message accumulation across multiple storage locations, preventing any single memory location from becoming a bottleneck and enabling better control over overall memory consumption patterns.
Data Source
AI summary
Handling memory pressure in an in-database sharded queue is described. Messages from a plurality of enqueuers are stored in a plurality of shards of a sharded queue. Messages from a first enqueuer are stored in a first shard. A queue table corresponding to the sharded queue is maintained. In volatile memory, a plurality of message caches is maintained, each message cache corresponding to a shard of the plurality of shards. Memory pressure is detected based on memory usage of the volatile memory. To store a specific message from the enqueuer, the specific message is stored in rows of the queue table that are assigned to the first shard. When memory pressure is not detected, the specific message is stored in a first message cache corresponding to the first shard. Subscribers of the sharded queue are caused to dequeue messages from the plurality of shards.