Publisher-Side Cache for Pub-Sub Traffic Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In publish-subscription environments, the current messaging paradigm results in excessive network traffic due to repetitive data being sent to multiple subscribers, leading to high bandwidth consumption.
Innovation Solution
Implementing a broker-assisted, publisher-side cache that identifies potentially repetitive data, assigns a unique identifier, and stores it for future messages, allowing for abbreviated messages to be sent, which are then completed by the broker system before being sent to subscribers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all message content is sent to all subscribers in a pub/sub model, then every subscriber receives complete information, but network traffic volume increases proportionally with the number of subscribers
Solution Approach 1:
The message is segmented into two parts: a compact unique identifier (sent to all subscribers) and the full message content (stored in broker cache). This segmentation allows subscribers to receive minimal data while the broker maintains the ability to provide complete information on demand.
Solution Approach 2:
The broker acts as an intermediary that maintains a cache of full message content. Instead of all-to-all communication, the broker receives the identifier, retrieves the full content from its cache, and distributes it to subscribers, thereby mediating between the publisher and multiple subscribers.
2Loss of information
If the same data is re-sent multiple times in different messages, then subscribers receive updated information, but bandwidth consumption increases significantly
Solution Approach 1:
Instead of sending the full message content repeatedly, the system sends a compact copy (unique identifier) that references the full content stored in the broker's cache. This copying approach minimizes network traffic while ensuring subscribers receive the necessary information.
Solution Approach 2:
The broker pre-loads and caches the full message content before it needs to be distributed. This preliminary action allows the broker to quickly retrieve and distribute complete information without requiring repeated full message transmissions, thereby reducing bandwidth consumption.
3Quantity of substance
If a cache system is implemented to store repetitive data, then network traffic is reduced, but system complexity increases due to cache management
Solution Approach 1:
The system automatically identifies repetitive data patterns and manages cache operations without requiring manual configuration or complex external management. The broker autonomously handles cache population, retrieval, and distribution, reducing the perceived complexity for users while maintaining traffic reduction benefits.
Data Source
AI summary
Embodiments of the present invention provide an approach for a broker-assisted, publisher side cache that can be utilized to reduce a volume of data (e.g., network traffic) delivered between a publisher and broker in a publication/subscription (pub/sub) environment. Specifically, in a typical embodiment, when a message is received on a broker system from a publisher system, the broker system analyzes the message for potentially repetitive data. Such data can be determined from historical messages (e.g., determining that a certain set of data appeared in multiple messages and is thus flagged in a database or the like), a set of rules (e.g., if data set “Y” appears, flag it as potentially repetitive), etc. Regardless, once such a set of data has been identified, a unique identifier/key corresponding thereto will be sent back to the publisher system and the set of data will be stored in a (data) cache. Upon receipt, the publisher system will cache/store the unique identifier in a (key) cache. Then, when the publisher system generates a future message that would contain the same set of data, the publisher system will substitute the unique identifier (as retrieved from the key cache) for the set of data to yield an abbreviated/truncated message. The abbreviated message would be sent to the broker system, which would: analyze the message; locate the unique identifier; retrieve the corresponding set of data from the (data) cache; replace the unique identifier with the set of data to yield a completed message; and then send the completed message to relevant subscriber system(s).