Dynamic Bucketing for Real-Time Event Subscriber Distribution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In real-time events with a large number of subscribers, the overwhelming amount of data flowing through the system consumes significant bandwidth and processing power, leading to inefficiencies and potential saturation of networks and devices, especially when a small number of publications per second are distributed to thousands or more subscribers.
Innovation Solution
A two-level bucketing system is implemented, where subscribers are allocated to slots within buckets, and the number of buckets is dynamically adjusted based on publication rates to manage the distribution of publications efficiently, ensuring that only subscribers within the same bucket receive publications, thereby reducing the load on each subscriber and maintaining context.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If publications are distributed to all subscribers in a real-time event, then all subscribers receive complete information, but network bandwidth and processing power are overwhelmed
Solution Approach 1:
The patent segments subscribers into different buckets based on their information needs and priorities. Instead of distributing all publications to all subscribers, the system divides the subscriber base into groups (buckets) and selectively distributes publications only to relevant buckets, thereby reducing overall network bandwidth consumption while ensuring each subscriber receives the information they need.
Solution Approach 2:
The patent applies local quality by allowing different buckets to receive different sets of publications based on their specific requirements. Each bucket is assigned a tailored subset of publications rather than a uniform distribution to all subscribers, optimizing the balance between information completeness and resource consumption for each group.
2Productivity
If the number of buckets is increased to reduce publication rate per bucket, then processing load per subscriber decreases, but system complexity increases
Solution Approach 1:
The patent implements dynamic bucket management where the number and configuration of buckets can be adjusted based on current system conditions, publication rates, and subscriber patterns. This dynamic approach allows the system to optimize processing efficiency by creating or merging buckets as needed, rather than using a fixed complex structure, thereby adapting to changing loads and reducing unnecessary complexity.
Solution Approach 2:
The system incorporates feedback mechanisms that monitor publication rates and subscriber behavior to dynamically adjust the number of buckets. When publication rates exceed thresholds, the system responds by adjusting bucket configurations, creating a self-regulating mechanism that balances processing efficiency with manageable system complexity through continuous adaptation rather than static complexity.
3Loss of energy
If publications are filtered to reduce bandwidth consumption, then network efficiency improves, but information completeness for subscribers may be compromised
Solution Approach 1:
The patent ensures information completeness is maintained at the local level for each subscriber bucket. Each bucket is configured with the specific publications relevant to its subscribers, so while overall network traffic is reduced through selective distribution, each subscriber receives complete information for their specific needs without receiving unnecessary publications from other categories.
Data Source
Figure 1
Figure 2A~2C
Figure 3A~3B
AI summary
Techniques to manage large numbers of subscribers to real-time events are described. Embodiments may filter subscribers dynamically into buckets where publications are distributed only to subscribers in the same bucket as the publisher. Buckets may be dynamically created and removed to lower or raise the publication rate for subscribers to an acceptable level. Embodiments may use a two-level bucketing algorithm for filtering that is memory and CPU efficient, and scales with large numbers of subscriptions. The techniques may also reduce the context loss for a subscriber when a new bucket has to be created and subscribers distributed between buckets. Embodiments may be scaled out to multiple servers, to accommodate very large numbers of subscribers and publications. Other embodiments are described and claimed.