Broker Cluster Workload Balancing via ML Health Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In a KAFKA distributed event streaming platform, workload balancing among brokers is uneven due to the lack of consideration for current workload and health of lead brokers, leading to unbalanced load distribution.
Innovation Solution
A processor monitors brokers to identify workload attributes, determines a health value for the lead broker using a machine learning model, and reallocates tasks based on an optimization algorithm to achieve a balanced workload distribution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a single lead broker is assigned to manage a partition in KAFKA, then the partition has a clear responsible broker for read/write operations, but the workload becomes unevenly distributed among brokers in the cluster
Solution Approach 1:
The patent implements dynamic leader broker selection by continuously monitoring workload attributes (CPU usage, memory usage, network I/O, disk I/O) of brokers and using machine learning models to predict health values. The system dynamically reassigns leader roles based on real-time conditions rather than static assignment, allowing the cluster to adapt to changing workload patterns and prevent any single broker from becoming overloaded
Solution Approach 2:
The patent establishes a feedback mechanism where the system continuously monitors broker workload attributes, evaluates health values using machine learning models, and uses this information to make informed decisions about leader broker reassignment. The optimization algorithm receives feedback about current workload distribution and adjusts leader assignments to achieve more balanced load distribution across the cluster
2Productivity
If traditional load balancing methods are used without considering broker health, then task redistribution can be performed, but the system cannot identify unhealthy brokers leading to suboptimal performance
Solution Approach 1:
The patent performs preliminary health assessment of brokers before executing task redistribution. The machine learning model predicts broker health values by analyzing workload attributes in advance, allowing the system to identify potentially unhealthy brokers before they fail. This preliminary action enables proactive load balancing decisions that prevent performance degradation
Solution Approach 2:
The patent replaces traditional mechanical load balancing methods with an intelligent system using machine learning models and optimization algorithms. Instead of using simple rules or random distribution, the system uses ML-based health prediction and optimization algorithms to make sophisticated decisions about task redistribution, achieving more balanced and reliable load distribution
Data Source
AI summary
An example operation may include one or more of monitoring a plurality of brokers within a cluster to identify current workload attributes of the plurality of brokers, determining a health value of a lead broker within the cluster via execution of a machine learning model on current workload attributes of the lead broker, determining to modify resources assigned to the lead broker based on the determined health value of the lead broker, executing an optimization algorithm on the current workload attributes of the plurality of brokers within the cluster to determine an optimum task distribution, and reallocating tasks amongst the lead broker and the one or more other brokers within the cluster based on the optimum task distribution.