Kafka Broker Rolling Update Order via Controller Identity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Rolling updates in Kafka clusters managed by container orchestration tools like Kubernetes can lead to performance degradation due to random selection of a new controller during updates, causing cascaded reselections and prolonged downtime.
Innovation Solution
An algorithm is implemented to determine the rolling update order based on the StatefulSet and identity of the controller, ensuring the controller is updated last, thereby minimizing disruption and guaranteeing a single reselection of the controlling broker.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If rolling update is applied to Kafka brokers in random order, then all brokers are updated, but controller reselection occurs multiple times causing performance degradation
Solution Approach 1:
The system performs preliminary identification of the current controller broker before the rolling update begins. The update order is pre-calculated to ensure non-controller brokers are updated first, and the controller is updated last. This preliminary planning prevents multiple controller reselections during the update process.
Solution Approach 2:
Instead of updating brokers in random order or updating the controller first, the system inverts the conventional approach by updating non-controller brokers first and deliberately postponing the controller update until last. This reverse ordering strategy ensures controller stability throughout the update process.
2Loss of time
If controller is updated early in rolling update, then update progress is made, but frequent controller reselection occurs causing downtime
Solution Approach 1:
The system calculates the optimal update order in advance, identifying which brokers are non-controllers and should be updated first. This preliminary ordering prevents the need for frequent controller reselections and minimizes downtime without requiring complex real-time coordination during the update process.
3Productivity
If standard rolling update is used without considering controller identity, then implementation is simple, but cascaded reselections occur degrading performance
Solution Approach 1:
The system performs a preliminary identification of the current controller and calculates the optimal update order before the rolling update begins. This upfront planning ensures high system performance during the update by avoiding cascaded reselections, while keeping the implementation relatively simple through a straightforward ordering algorithm.
Data Source
AI summary
A method for automatically updating a plurality of brokers in a cluster of a distributed streaming platform within a container orchestration tool, the method including querying each of a plurality of brokers to obtain metadata from a distributed streaming platform; using the obtained metadata to identify a broker within the plurality of brokers that acts as a controlling broker; computing an order in which the plurality of brokers will be updated based on the obtained metadata, wherein the order specifies that the controlling broker will be updated last; and performing, via an update manager executed by at least one processor, a rolling update of each of the plurality of brokers according to the computed order.