Publisher-Assisted Broker Caching for Pub-Sub Bandwidth Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In publish-subscription (pub/sub) environments, the current approach results in excessive network traffic due to repetitive data being sent to all users, leading to high bandwidth consumption, as all content is sent regardless of relevance, and the same data is often resent multiple times to the same subscribers.
Innovation Solution
Implementing a publisher-assisted, broker-based cache system that identifies repetitive data and assigns it a unique identifier, allowing for abbreviated messages to be sent, which are then expanded by the broker system using cached data, reducing network traffic and bandwidth usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If all message content is sent to all subscribers in a pub/sub environment, then complete information is delivered to all users, but network traffic and bandwidth consumption increase significantly
Solution Approach 1:
The message is segmented into two parts: a unique identifier (cache key) and the actual data content. The identifier is sent to subscribers, while the full content is retrieved from the broker's cache, reducing network traffic while maintaining information completeness.
Solution Approach 2:
The broker pre-caches message content when it first receives it from the publisher. This preliminary action allows subsequent subscribers to retrieve content from the cache without requiring the publisher to resend the full message, reducing redundant network traffic.
2Reliability
If the same data is re-sent multiple times to the same subscriber, then all messages are delivered, but network bandwidth is wasted on repetitive content
Solution Approach 1:
The broker creates a copy of the message content and stores it in its cache. When the same data needs to be sent to subscribers, the broker retrieves the copy from cache and sends only the unique identifier, avoiding redundant transmission of the same content while ensuring reliable delivery.
3Loss of energy
If a cache system is implemented to reduce network traffic, then bandwidth consumption decreases, but system complexity increases due to cache management
Solution Approach 1:
The broker acts as an intermediary between the publisher and subscribers, managing the cache on its own side. This centralizes cache management in one component rather than requiring distributed cache management across multiple systems, reducing overall system complexity while achieving bandwidth reduction.
4Quantity of substance
If unique identifiers are used to represent cached data, then message size is reduced, but the system requires additional processing to manage and retrieve cached content
Solution Approach 1:
The broker's cache management system automatically handles the mapping between unique identifiers and cached content without requiring manual intervention or complex external coordination. The system serves itself by maintaining the cache and performing lookups autonomously, reducing the perceived processing overhead despite the added functionality.
Data Source
AI summary
An approach for a publisher-assisted, broker-based cache that can be utilized to reduce a volume of data delivered between a publisher and broker is provided. Specifically, when a message is being generated on a publisher system, the publisher system will determine if the message includes a set of data that has a potential to be repeated in subsequent messages. An identified set of data will be associated/marked/tagged with a unique identifier corresponding thereto. The modified message will be sent to a broker system, which will detect/locate the unique identifier, cache the corresponding data, and send the message along to any applicable subscriber systems. When a subsequent message that is supposed to contain the cached set of data is generated, the publisher system will substitute the unique identifier for the set of data to yield an abbreviated message and send the abbreviated message to the broker system.