Probabilistic Event Filtering at Source
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
As the number of event sources and targets in an event bus system grows, many events ingested by the system are not of interest to the targets, leading to resource waste and inefficiency, as the system currently lacks effective mechanisms to filter out irrelevant events before they are routed.
Innovation Solution
Implementing probabilistic data structures, such as bloom filters, at the event sources to determine whether events should be sent to the event bus, allowing for efficient filtering and reducing the number of events sent, thereby conserving resources and improving processing speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If events are sent to the event bus without filtering, then all event targets can potentially receive all events, but computational resources are wasted processing irrelevant events
Solution Approach 1:
The patent applies preliminary action by implementing filtering mechanisms at the event source before events are published to the event bus. The system pre-processes events to determine which ones are relevant to any event targets, and only publishes those filtered events. This prevents irrelevant events from consuming computational resources in the first place, resolving the contradiction between resource conservation and processing efficiency.
2Loss of energy
If filtering mechanisms are implemented at the event bus, then irrelevant events can be blocked, but the system complexity increases
Solution Approach 1:
The patent extracts the filtering function from the central event bus and relocates it to the event sources. Each event source independently determines which of its events are relevant by checking against known event target subscriptions. This extraction eliminates the need for complex centralized filtering logic at the event bus, reducing system complexity while still achieving resource conservation.
3Adaptability or versatility
If the number of event sources and targets grows, then the system becomes more versatile, but the number of irrelevant events increases leading to resource waste
Solution Approach 1:
The patent implements feedback mechanisms where event sources receive information about which events are subscribed to by event targets. Event sources use this feedback to make informed decisions about which events to publish, dynamically adjusting their publishing behavior based on current system state. This allows the system to scale while maintaining efficiency, as each event source adapts its behavior based on real-time subscription information.
Data Source
AI summary
A probabilistic data structure such as a bloom filter may be used to provide a space efficient representation of rules that define events of interest to sources and may be used to reduce the number of events ingested by an event bus system. An event bus system may generate a probabilistic data structure based on the rules that may be used at the event source as a filter. The probabilistic data structure predetermines whether the events to be sent to the event bus system will be of interest to any event targets and if an event is of no interest to any of the event targets, the event will not be sent.