Alerting System Using Stateful Transformation Nodes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional alerting systems are inefficient due to the need to recompute full formulas on every input change or schedule, lack reusability of computed results, and run checks in isolation, leading to resource-intensive and unnecessary computation cycles, especially as the number of event streams and alerts grows.
Innovation Solution
An alerting system utilizing a network of transformation nodes that store state information, subscribe to other nodes, and react only to significant changes, allowing checks to be represented as directed acyclic graphs (DAGs) and minimizing recomputation by storing and reusing intermediate results, thus reducing unnecessary computation and improving scalability and performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional alerting systems recompute full formulas on every input change or schedule, then checks can be performed periodically, but computational resource usage increases significantly and unnecessary computation cycles occur
Solution Approach 1:
The system pre-computes and stores intermediate results in a computation graph before they are needed for final check evaluation. Transformation nodes maintain cached results of their computations, so when checks need to be evaluated, the system can reuse these pre-computed values instead of recalculating everything from scratch, significantly reducing computational resource usage while maintaining check accuracy.
Solution Approach 2:
The computation graph structure allows intermediate transformation results to serve multiple purposes - they are used both for their original computational purpose and as reusable inputs for multiple different checks. This multi-functionality enables the system to share computation across multiple checks, reducing redundant calculations and improving overall system efficiency.
2Productivity
If traditional alerting systems process each check independently, then checks can be executed in isolation, but the system lacks reusability of computed results and performs redundant calculations
Solution Approach 1:
The system merges multiple independent check processing paths into a unified computation graph where shared transformation nodes are consolidated. Instead of each check having its own separate processing pipeline, multiple checks share common transformation nodes and their results, allowing the system to execute checks more quickly by reusing previously computed intermediate values rather than recalculating them.
Solution Approach 2:
The system creates a computational model (computation graph) that copies and represents the relationships between checks and transformations. This graph structure allows the system to efficiently traverse and evaluate checks by copying relevant intermediate results from the graph rather than re-executing transformations, significantly reducing recomputation time while maintaining check execution speed.
3Adaptability or versatility
If the number of event streams and alerts grows, then the system can monitor more data, but scalability of storage and computational layers is challenged
Solution Approach 1:
The system segments the monolithic check evaluation process into a distributed computation graph of transformation nodes. Each node handles a specific transformation independently, and the graph structure naturally divides the computational workload. This segmentation allows the system to scale to more event streams and alerts by adding more transformation nodes without proportionally increasing overall system complexity, as each node operates independently with localized state.
Data Source
AI summary
An alerting system is provided that includes a network of transformation nodes, and a state change processors. The transformation nodes include input transformation nodes, output transformation nodes, and intermediate nodes that connect the input and output transformation nodes. Each input transformation node can receive an events stream, and is coupled to one of the output transformation nodes by one or more intermediate transformation nodes. Each transformation node (except the input transformation nodes) can receive state updates from those transformation nodes that it subscribes to. Each output transformation node can generate a check result when stored state information for each of the transformation nodes that the output transformation node subscribes to collectively indicates that the check result should be generated. Each output transformation node is coupled to one of the state change processors that can determine whether the check results should trigger an action, and if so, can then perform an action.