Kafka Producer Retry Cache for Timeout-Resilient Message Delivery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed event streaming environments like Apache Kafka, producer failures during upgrades, migrations, or disaster recoveries can lead to message rejections due to exceeding delivery timeout values, resulting in potential message loss and inefficiencies in handling such failures.
Innovation Solution
Implement a centralized cache to store messages that cannot be sent to a particular topic, using a key-partitioner algorithm to pre-compute partition IDs for re-sending, and a recurrent watchdog per microservice group to manage retries, ensuring only one pod attempts re-sending, thereby reducing contention and message loss.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a delivery timeout property is set for the producer, then the system can report success or failure within a defined time, but messages may be rejected during broker upgrades or migrations due to the timeout
Solution Approach 1:
The system pre-computes partition IDs using a key-partitioner algorithm before broker upgrades occur. This preliminary computation stores the necessary routing information in advance, allowing messages to be resent to the correct partitions after broker recovery without requiring real-time partition discovery, thus preventing message rejection during timeout periods.
Solution Approach 2:
A recurrent watchdog process acts as an intermediary between the producer and the centralized cache. The watchdog monitors message delivery status and automatically retries sending failed messages to the correct partitions, mediating between the timeout constraint and the need for reliable delivery during broker upgrades.
2Reliability
If messages are resent to the same partition during broker recovery, then message delivery can be attempted, but contention increases when multiple pods attempt re-sending simultaneously
Solution Approach 1:
The system segments the retry responsibility by assigning specific partitions to specific pods based on pre-computed partition IDs. Each pod is responsible for retrying messages only for its assigned partitions, eliminating the need for complex inter-pod coordination and reducing contention while maintaining reliable message delivery.
3Reliability
If the delivery timeout value is increased to allow broker upgrades, then message rejection is reduced, but the time to report success or failure increases significantly
Solution Approach 1:
Partition IDs are pre-computed and stored before broker upgrades occur, eliminating the need for extended timeout periods. When brokers recover, messages can be immediately resent to the correct partitions using the pre-stored routing information, achieving high reliability without increasing timeout duration.
Solution Approach 2:
The recurrent watchdog continuously monitors and retries message delivery after broker recovery, ensuring that the useful action of message delivery continues without interruption rather than waiting for an extended timeout period to expire.
Data Source
AI summary
In accordance with an embodiment, described herein are a system and method for use with a distributed event streaming environment (e.g., a Kafka environment), for making services resilient of producer failures. When a determination is made that one or more messages could not be sent to a particular topic after a timeout error, those messages are stored in a centralized cache (e.g., as provided by a database service). A key-partitioner algorithm or process is used to pre-compute a partition ID into which the message will be re-sent. The pre-computed partition ID is used to compute the key of the cache entry for the message as stored within the centralized cache. A recurrent watchdog per group of microservice resources (e.g., per pod) operates to query the centralized cache for the messages to be re-sent into the partitions pertaining to those resources (i.e., that pod).