Adaptive Message Filtering via Flow Network Graphs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing message filtering systems using regular expressions are challenging due to their rigidity, leading to errors and inefficiencies in data processing, as users often struggle to configure them correctly, resulting in incorrect output and reduced accuracy in anomaly detection.
Innovation Solution
A system that generates a flow network graph based on sample messages matching a regular expression, allowing for the analysis of non-matching messages to determine their similarity and suggest modifications or new patterns, thereby improving filter accuracy and reducing the need for 100% conformity to the original expression.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If rigid regular expression filtering is used, then filtering precision is improved, but adaptability deteriorates
Solution Approach 1:
The system dynamically adjusts filter patterns based on learned message characteristics. Instead of using static regular expressions, the system continuously learns from message samples and adapts the filtering patterns to match evolving message formats, resolving the contradiction between rigid filtering precision and adaptability to new formats.
Solution Approach 2:
The system changes the parameters of filter patterns by learning from message samples. It extracts common patterns from samples and uses these learned patterns to create flexible filters that can adapt to variations in message formats while maintaining filtering precision through pattern matching.
2Measurement precision
If complex regular expressions are configured, then filtering precision is improved, but ease of operation deteriorates
Solution Approach 1:
The system performs self-learning by automatically analyzing message samples and generating filter patterns without user intervention. This eliminates the need for users to manually configure complex regular expressions while maintaining high filtering precision through the system's learned patterns.
Solution Approach 2:
The system performs preliminary learning by analyzing message samples before actual filtering operations. This pre-learning phase automatically generates the filter patterns that will be used for filtering, eliminating the need for users to manually configure complex expressions and making the system easier to operate.
3Measurement precision
If 100% conformity to original expression is required, then filtering precision is improved, but productivity deteriorates
Solution Approach 1:
The system uses partial matching by learning common patterns from message samples rather than requiring 100% conformity to rigid expressions. This allows messages that partially match the learned patterns to be correctly filtered, improving productivity while maintaining sufficient precision for practical purposes.
Data Source
AI summary
A computer system receives a set of messages, and processes the messages using a filter. In some examples, the filter is defined using a pattern matching language such as a regular expression. The system collects a set of representative messages that match the filter. Using the set of representative messages, the system generates a corresponding flow network graph. Using the flow network graph, the system determines a similarity measure that indicates whether a new message resembles other matching messages of the filter. Based on the similarity measure, in various embodiments, the system identifies potential errors in the filter definitions, omissions in the terms of the filter, and message outliers that indicate system anomalies or events of particular interest to the user.