Distributed In-Memory Timer Queues for Low-Latency Scale-Out Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing techniques for managing user-specific timers at scale face challenges with high latency and network requests, leading to complex and error-prone solutions, especially when dealing with large numbers of timers expiring simultaneously.
Innovation Solution
A distributed in-memory timer service that supports multi-tenancy, using a durable in-memory database and append-only logs to efficiently manage timers, ensuring fault tolerance and minimizing duplicate sends, with features like sharded time buckets and consumer groups for even distribution and recovery from failures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If existing techniques for storing and processing unique per user timers are used, then timers can be managed, but latency increases and network requests increase
Solution Approach 1:
The patent segments the timer queue by dividing time into discrete buckets (e.g., 1-second intervals) and further sharding each bucket across multiple nodes in a distributed system. Each node maintains only the portion of the timer queue relevant to its shard, eliminating the need for centralized storage and reducing latency for timer processing while distributing system complexity across multiple independent units.
2Productivity
If current techniques for timers at scale are implemented, then large numbers of timers can be handled, but storage IOPs and network requests increase significantly
Solution Approach 1:
The patent merges multiple timer operations into batched processing within each time bucket. Instead of processing individual timers separately (which would generate numerous network requests and storage IOPs), the system accumulates timers into buckets and processes them in batches, significantly reducing the total number of IOPs and network requests while maintaining high throughput for large numbers of timers.
Solution Approach 2:
The patent performs preliminary organization of timers into time-based buckets and sharded partitions before processing occurs. This pre-organization allows the system to efficiently locate and process only the relevant timer buckets for the current time window, avoiding full-scans of entire timer queues and reducing both storage IOPs and processing overhead while maintaining scalability.
3Reliability
If distributed queues with multiple consumers are used, then fault tolerance improves, but duplicate processing may occur during consumer outages
Solution Approach 1:
The patent implements feedback mechanisms where consumer groups report their processing status and acknowledgment of processed timer buckets back to the system coordinator. This feedback allows the system to detect which consumers are active and which have failed, enabling dynamic reassignment of timer bucket ownership to prevent both duplicate processing (by ensuring only active consumers process their assigned buckets) and processing gaps (by reassigning buckets from failed consumers to surviving members of the consumer group).
Data Source
AI summary
Disclosed are systems and methods for implementing distributed in-memory timer queues. A plurality of queues is distributed across a plurality of nodes. The plurality of queues includes a first queue. The first queue corresponds to current timer expirations for a tenant, time bucket, and shard. A sorted set includes minute references to unacknowledged timers for a tenant and shard. A plurality of processes is created. Each of the plurality of processes is configured to be a unique consumer of the distributed plurality of queues. A first consumer group and a second consumer group are created within each of the plurality of processes. The first consumer group corresponds to the first queue. The second consumer group corresponds to the sorted set. The plurality of processes is used to proportionally consume the first queue with the first consumer group and the sorted set with the second consumer group.