Topic-Based Data Routing in Publish-Subscribe Messaging
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In publish-subscribe messaging systems, message brokers experience increased load and memory consumption when subscribers do not consume data promptly, leading to potential crashes due to data backlog, and dynamic topic publishing can cause subscribers to be unaware of topic names.
Innovation Solution
Implementing a system with a message broker that receives topic metadata at a fixed interval, a topic list subscriber that obtains and assigns topics to subscribers based on predefined criteria, and a subscription manager that dynamically adjusts topic assignments to balance load, using a caching mechanism to manage topic updates and subscriber additions/removals.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If message brokers deliver data to subscribers in a traditional publish-subscribe system, then data routing functionality is provided, but memory consumption increases and broker crashes occur when subscribers do not consume data promptly
Solution Approach 1:
The patent segments the data delivery process by introducing an event store that separates data persistence from subscriber consumption. Data is written to the event store independently of subscriber processing speed, allowing the broker to avoid holding data in memory while subscribers consume at their own pace. This segmentation resolves the contradiction by maintaining reliability through persistent storage while reducing memory consumption through asynchronous processing.
Solution Approach 2:
The event store acts as an intermediary between the message broker and subscribers. Instead of the broker directly managing data in memory for all subscribers, the event store serves as a buffer that decouples data production from consumption. This intermediary mechanism allows the broker to maintain stability while reducing its memory burden by offloading data persistence to the event store.
2Adaptability or versatility
If subscribers dynamically subscribe to topics, then system flexibility is improved, but subscribers become unaware of topic names causing routing issues
Solution Approach 1:
The subscription manager implements a feedback mechanism that provides subscribers with information about the topics they are subscribed to. When subscribers dynamically subscribe, the system maintains topic name awareness by querying the event store for topic information and providing this feedback to the subscription manager, which then routes data appropriately. This feedback loop resolves the contradiction by maintaining adaptability while preventing information loss about topic names.
Solution Approach 2:
The system performs preliminary actions by pre-fetching and caching topic metadata before data routing occurs. When a subscriber expresses interest in a topic, the subscription manager proactively queries the event store for related topic names and information in advance. This preliminary action ensures that topic name awareness is established before data flow begins, resolving the information loss issue while maintaining dynamic subscription capability.
Data Source
AI summary
Techniques for providing topic-based data routing in a publish-subscribe messaging environment are disclosed. In one example, a system supported by hardware in a publish-subscribe messaging environment may include a message broker to receive topic metadata at a first instance according to a first interval into a predefined fixed topic. Further, the system may include a topic list subscriber that subscribes to the predefined fixed topic and obtains a list of topics from the message broker. Furthermore, the system may include a subscription manager to assign topics in the list of topics to a set of subscribers according to a criteria. In one example, the message broker may route data corresponding to the topics to the set of subscribers in accordance with the topics assigned to the set of subscribers.