Event Server Partition Management for Microservice Data Consistency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In microservice architectures, consumers face challenges in adapting to changes in data store configurations, particularly in mapping processed partitions and offsets to new configurations, especially when they have limited knowledge about the data store's configuration.
Innovation Solution
An event server manages partition assignments for consumers, using identifiers and a cache to track which data each consumer has received and has yet to receive, allowing it to dynamically reassign partitions when necessary and maintain data consistency across different instances and geographic locations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If consumers directly access data store partitions and offsets, then they can retrieve event data, but they face difficulties when data store configuration changes and require knowledge of data store configuration
Solution Approach 1:
The patent introduces an event server as an intermediary between consumers and the data store. The event server manages partition assignments, tracks consumer progress, and handles configuration changes, while consumers only interact with the event server through simplified registration and subscription APIs. This mediator absorbs the complexity of data store configuration management, allowing consumers to adapt easily to configuration changes without needing to understand or map partitions and offsets directly.
2Adaptability or versatility
If a single publisher publishes data for multiple consumers, then data can be distributed to multiple consumers, but portability of publishers and consumers across different environments and topologies is limited
Solution Approach 1:
The event server provides universal functionality that works across different environments and topologies. It implements a standardized interface for consumers to subscribe to events regardless of the underlying data store configuration, publisher location, or network topology. The event server's partition assignment mechanism and configuration management enable consumers to be deployed portably across different data centers, cloud environments, and geographic locations without modification.
3Reliability
If consumers need to map processed partitions and offsets to new data store configuration, then they can maintain processing state, but this becomes difficult when configuration changes occur
Solution Approach 1:
The event server implements self-service by automatically managing partition assignments and tracking consumer processing state. When configuration changes occur, the event server dynamically reassigns partitions to consumers and updates the mapping information without requiring consumer intervention. Consumers simply register their interest in event types, and the event server handles all the complexity of maintaining consistent state mapping across configuration changes, making the system easy to operate while ensuring reliability.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Some embodiments provide a novel policy-driven method for providing event data to several event consumers. An event server stores in a set of one or more data storages event data published by a set of one or more event data publishers. The event server receives first and second different event-distribution policies from first and second event consumers for first and second streams of event data tuples for which the first and second event consumers register with the event server to receive. Each event consumer has multiple consumer instances. The event server uses the first and second event-distribution policies to differently distribute the first and second streams of event data tuples to the consumer instances of the first and second event consumers.