Message Broker Consumer Group Versioning for Microservices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In message broker systems, especially in microservices architectures, individual microservices can become overwhelmed with incoming messages, leading to delays and message dropping as the architecture scales. Additionally, upgrades to microservices cause delays in message processing due to the need for existing consumers to disconnect before new consumers can be assigned to partitions.
Innovation Solution
Implementing consumer group versioning, where each application runtime provides a version for its consumer group, allowing for immediate disconnection of outdated consumer groups and reassignment of partitions to newer versions, thus avoiding delays and improving message processing efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If microservices are scaled to handle more messages, then message processing capacity increases, but individual microservices become overwhelmed leading to delays and message dropping
Solution Approach 1:
The patent segments the consumer group into multiple versions (v1, v2, etc.), allowing different microservice versions to coexist and process messages in parallel. This segmentation enables the system to distribute message processing across multiple consumer instances, increasing capacity while maintaining reliability through version diversity.
Solution Approach 2:
The patent introduces a version dimension to the consumer group concept, transforming it from a single-version structure to a multi-version hierarchical structure. This dimensional change allows simultaneous existence of multiple consumer versions under one consumer group, enabling scalable message processing without overwhelming individual services.
2Adaptability or versatility
If existing consumers disconnect before new consumers are assigned during upgrades, then partition reassignment can occur, but message processing experiences delays
Solution Approach 1:
The patent performs preliminary actions by pre-assigning new consumer versions to partitions before old consumers fully disconnect. The message broker maintains partition assignments during the transition, allowing new consumers to be ready and immediately take over message processing roles, eliminating idle time during upgrades.
Solution Approach 2:
The patent ensures continuity of useful action by maintaining message processing throughout the upgrade process. Multiple consumer versions process messages simultaneously during transition, and the message broker continuously assigns partitions to available consumers, ensuring no interruption or delay in message processing during upgrades.
Data Source
AI summary
In an example embodiment a mechanism for consumer group versioning is introduced. Here, each application runtime provides a version for any consumer group during its deployment and keeps increasing the version whenever there is an enhancement or bug fix. Thus, both the application and the consumer group will have a version. Once it is recognized that a consumer group assigned to partitions in a topic has an outdated consumer group version number (i.e., a consumer group with the same name/application but a later consumer group version number has been registered with the message broker), the old consumer group is disconnected immediately. This allows the message broker to immediately assign partitions to the consumers in the newer consumer group, thus avoiding the aforementioned delays and associated technical problems.