Distributed Rule Matching With Centralized Compilation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing monitoring and observability systems face challenges in efficiently evaluating large numbers of fluctuating rules on high-volume data streams, leading to quadratic computational complexity and bottlenecks in routing and filtering components.
Innovation Solution
The implementation of a distributed matching system with a rule manager node that decouples rule compilation from rule evaluation, using multi-rule optimization techniques to generate optimized evaluation instructions, and employing strategies like time-controlled or priority-based compilation to balance fast rule application with resource efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If rule compilation is performed on each matching node to achieve locally optimized evaluation, then matching performance is improved, but device complexity and processing overhead increase
Solution Approach 1:
The patent extracts the rule compilation function from individual matching nodes and centralizes it in a dedicated rule manager node. This separation allows matching nodes to focus solely on evaluation while the rule manager handles the complex compilation task, reducing device complexity while maintaining matching throughput through optimized pre-compiled rules.
Solution Approach 2:
The rule manager performs rule compilation in advance before rules are distributed to matching nodes. This preliminary action ensures that rules are pre-optimized and ready for immediate use, improving matching throughput without requiring complex real-time compilation capabilities at each node.
2Productivity
If complete rule set analysis is performed to generate globally optimized evaluation instructions, then matching efficiency is improved, but loss of time and processing resources increase
Solution Approach 1:
The system performs complete rule set analysis and generates optimized evaluation instructions in advance through the rule manager, before distribution to matching nodes. This preliminary optimization achieves globally efficient rule evaluation while the actual matching operations use these pre-generated instructions, minimizing time loss during runtime.
Solution Approach 2:
The rule manager creates optimized copies of rule evaluation instructions that are distributed to multiple matching nodes. These copied instructions are pre-optimized versions that can be executed efficiently across the distributed system without requiring each node to perform complete analysis.
3Adaptability or versatility
If rule updates are applied immediately to maintain fast rule application, then adaptability is improved, but matching throughput decreases due to compilation overhead
Solution Approach 1:
The rule manager acts as an intermediary that receives rule updates, performs compilation in the background, and distributes updated rules to matching nodes without interrupting their operation. This mediator approach enables fast rule updates while maintaining matching throughput by decoupling the update application from the compilation process.
Solution Approach 2:
The rule manager performs rule compilation in advance of when updated rules are needed by matching nodes. This preliminary compilation allows rules to be updated and prepared without interrupting the matching throughput, as the compilation occurs in the background while matching operations continue with existing rules.
4Productivity
If distributed matching nodes each evaluate all rules independently, then system scalability is improved, but computational complexity increases quadratically
Solution Approach 1:
The patent merges the rule compilation function across all distributed matching nodes into a centralized rule manager. This consolidation eliminates redundant compilation efforts at each node, reducing overall computational complexity from quadratic to linear while maintaining the scalability benefits of distributed matching evaluation.
Solution Approach 2:
The complex rule compilation and optimization function is extracted from individual matching nodes and placed in a dedicated rule manager. This extraction allows matching nodes to remain simple evaluators that can be easily scaled, while the complex analytical work is performed once by the rule manager, reducing overall system computational complexity.
Data Source
AI summary
A technology is presented for the efficient matching of received data elements with medium to large sets of fluctuating matching rules. To cope with high volumes of data elements, a distributed architecture is used, which leverages optimized multi-rule evaluation approaches, like Intel's Hyperscan, to achieve sublinear computational complexity for the rule matching process. Processing of rule updates, and generation of corresponding optimized evaluation instructions is performed on a central management node, which distributes generated optimized matching code to multiple worker nodes for the actual matching process. Further, optimized multi-rule evaluation is combined with application of individual match rules, to support the fast application of matching rule changes if required. Compilation strategies are applied to eventually transform individually applied rules into a corresponding optimized multi-rule evaluation form.