Subscription Groups in Publish-Subscribe Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current publish-subscribe systems face inefficiencies due to varied consumption rates among subscribers, leading to excessive database calls, scalability issues, and incompatibility with different producers, resulting in resource consumption and delays.
Innovation Solution
The system groups subscribers with similar consumption characteristics into subscription groups, dynamically adjusting group membership based on consumption rates and transmitting messages efficiently, while also optimizing routing and reducing database connections.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If publishers publish messages to all subscribers regardless of consumption rates, then all subscribers receive messages, but slower subscribers cause back pressure and excessive database calls
Solution Approach 1:
The patent segments subscribers into different groups based on their consumption characteristics (e.g., fast consumers, slow consumers, intermittent consumers). Each segment receives messages at appropriate rates, preventing back pressure from slow consumers while ensuring all subscribers eventually receive messages. This segmentation resolves the contradiction by allowing the publisher to maintain high throughput for fast consumers while still delivering to slow consumers without excessive database calls.
Solution Approach 2:
The patent implements dynamic message delivery where the publishing rate to each subscriber is adjusted based on their consumption characteristics. Fast consumers receive messages at higher rates, while slow consumers receive messages at lower rates. This dynamic adjustment allows the system to maintain overall high productivity while ensuring reliable delivery to all subscribers without causing back pressure.
2Ease of operation
If publishers establish separate connections for each subscriber set, then routing is precise, but computing and network resources are consumed excessively
Solution Approach 1:
The patent merges multiple subscriber connections into subscription groups, where a single connection can serve multiple subscribers within the same group. Messages are published once to the group and distributed to individual subscribers. This merging reduces the number of connections from potentially thousands (one per subscriber) to a manageable number (one per group), significantly reducing computing and network resource consumption while maintaining routing accuracy through group-based addressing.
3Speed
If the system is tightly coupled for efficient message publishing, then message transmission is fast, but scalability to multiple locations is limited
Solution Approach 1:
The patent implements a universal subscription group mechanism that can operate efficiently in both tightly-coupled single-location scenarios and distributed multi-location scenarios. The same group-based publishing approach works whether subscribers are on the same network or distributed across multiple locations, providing scalability without sacrificing transmission speed. Messages can be published locally with fast delivery to co-located subscribers while also being routed to remote subscribers through the group mechanism.
4Reliability
If publishers refetch messages for lagging subscribers from storage, then message delivery is ensured, but excessive database calls consume significant time
Solution Approach 1:
The patent applies preliminary action by pre-fetching messages only for fast-consuming subscribers who are likely to need them immediately, while slow-consuming subscribers receive messages at a paced rate that prevents lagging. This preliminary selective publishing eliminates the need for subsequent refetching operations from storage, ensuring reliable delivery to all subscribers while minimizing database access time by avoiding unnecessary refetches for slow consumers.
Data Source
AI summary
Disclosed is a technology for publishing subscriptions in a publish-subscribe system in accordance with subscription groups. The technology may include (i) determining a consumption characteristic by which each of multiple subscribers in a publish-subscribe system consumes a subscription published by a publisher; (ii) identifying the subscribers whose consumption characteristics satisfy a specified similarity criterion; (iii) defining multiple subscription groups, each of which includes subscriptions of those of the subscribers whose consumption characteristics satisfy the specified similarity criterion; (iv) storing the subscriptions in multiple logical partitions of a storage system where each of the logical partitions contains a separate non-overlapping subset of the subscriptions; and (v) transmitting the subscriptions to the subscribers in accordance with the subscription groups.