Collaborative Triggers for Distributed Data Discovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed and dynamic computing systems face challenges in providing scalable and efficient notification mechanisms, as existing methods like triggers and polling overwhelm nodes with notifications, leading to bandwidth and processing issues.
Innovation Solution
Implementing collaborative triggers that distribute notifications among nodes, suppressing further triggers of the same type until an actor responds, allowing for efficient global notification without flooding the system.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional triggers or polling mechanisms are used for notifications in distributed computing systems, then nodes can be notified of data source updates, but the system is overwhelmed with redundant notifications leading to bandwidth and processing issues
Solution Approach 1:
The patent merges multiple notification triggers of the same type into a single collaborative trigger. When the first node detects an update event, it generates a collaborative trigger that is propagated to other nodes. Subsequent nodes that detect the same type of event suppress their own triggers and wait for the collaborative trigger response, effectively combining multiple redundant notifications into one system-wide notification event.
Solution Approach 2:
The system implements feedback through the collaborative trigger mechanism where nodes monitor for responses to triggers they have propagated. When a node propagates a collaborative trigger, it enters a suppressed state and waits for feedback in the form of a response from other nodes. This feedback loop prevents redundant notifications while ensuring all nodes are eventually notified of the update event.
2Loss of information
If nodes propagate all update notifications to all other nodes, then comprehensive notification coverage is achieved, but network bandwidth and processing resources are wasted on redundant messages
Solution Approach 1:
The notification system dynamically adjusts its behavior based on the current state of each node. Nodes transition between active and suppressed states depending on whether they have received or propagated a collaborative trigger of a given type. This dynamic state management allows the system to adaptively reduce redundant transmissions while maintaining comprehensive notification coverage across all nodes.
Solution Approach 2:
The system changes the parameter of notification propagation by introducing a suppression mechanism that temporarily disables redundant trigger propagation. When a node is in a suppressed state for a particular trigger type, it stops propagating that trigger type until it receives feedback. This parameter change from always-propagate to conditionally-propagate reduces bandwidth consumption while maintaining notification effectiveness.
3Speed
If each node independently processes all update events, then local responsiveness is maintained, but system-wide coordination and resource efficiency deteriorate
Solution Approach 1:
The notification processing function is segmented between two roles: trigger propagation (performed by the first node that detects the event) and trigger suppression (performed by subsequent nodes). This segmentation allows the system to maintain local responsiveness when a node first detects an event, while coordinating system-wide resource efficiency through the collaborative trigger mechanism that other nodes participate in by suppressing redundant triggers.
Data Source
AI summary
Table discovery in a distributed and dynamic computing system can include detecting an event of a type indicating an update to a data source of a selected node of a plurality of nodes of a computing system, in response to the event, generating, within the selected node, a first collaborative trigger having a query identifier correlated with the type of the event, and distributing the first collaborative trigger through other ones of the plurality of the nodes of the computing system so that the first collaborative trigger reaches an actor of the computing system. Table discovery further can include, in response to receiving the first collaborative trigger within the actor, generating, within the actor, a query for table definition information. The query may be implemented as an aggregation query and may be distributed through the plurality of the nodes.