Signature Detection Sublinear Processing Streaming Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for processing large volumes of streaming data records with multiple signature rules are inefficient, leading to high processing costs and inability to handle frequent changes in signature sets without disruptive recompilation, especially in real-time analytics scenarios.
Innovation Solution
The method involves generating atomic recognizers and context lookup tables from signature rules, allowing for sublinear processing of streaming data records by using deterministic finite state automata and memory representations, enabling early quit in signature match processing and non-disruptive updates to the signature rule set.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional linear processing methods are used to check each signature rule against streaming data records, then complete signature detection accuracy is achieved, but processing time and computational cost increase linearly with the number of signature rules
Solution Approach 1:
The patent segments the signature rule set into multiple groups or buckets, and processes these groups in parallel using multiple threads. Each thread handles a subset of signature rules simultaneously, reducing the sequential processing time while maintaining complete detection accuracy through coordinated results aggregation.
Solution Approach 2:
The patent performs preliminary compilation of signature rules into optimized data structures (such as decision trees or lookup tables) before actual data processing. This pre-processing step creates efficient access patterns that enable faster matching during streaming data analysis, reducing real-time processing time without sacrificing detection accuracy.
2Adaptability or versatility
If the signature rule set is frequently updated to reflect changing analytics needs, then adaptability to new patterns is improved, but system disruption and recompilation overhead increase
Solution Approach 1:
The patent implements dynamic signature rule sets that can be modified, added, or removed without requiring complete system recompilation. The architecture allows individual rules to be updated independently, with the compilation system adapting to changes incrementally, thereby maintaining high adaptability while minimizing disruption to ongoing data processing operations.
3Adaptability or versatility
If more signature rules are added to detect additional classification categories, then classification comprehensiveness is improved, but processing complexity and resource requirements increase linearly
Solution Approach 1:
The patent divides the expanded signature rule set into manageable segments or groups that can be processed in parallel. By organizing rules into multiple independent threads or processing units, the system handles increased classification comprehensiveness without proportionally increasing per-thread complexity, as each thread manages a subset of rules with reduced individual complexity.
Solution Approach 2:
The patent creates a universal processing framework that handles diverse signature rules through common compilation and execution mechanisms. This multi-functional architecture allows the same core engine to process different types of classification rules efficiently, reducing overall processing complexity despite the increased number and variety of signature rules.
Data Source
AI summary
A method for multicategory classification of streaming data records includes receiving a list of signature rules, each comprising a Boolean expression of a set of atomic recognizers (ARs) of one or more input fields of an input record and an assignment of a signature if the Boolean expression evaluates to TRUE, generating a list of all signature contexts from the list of signature rules, generating a context lookup table for each context, and processing a stream of input records on which signature detection is performed by using said ARs, said list of signature contexts, and said context lookup table for each context, wherein each input record in the stream of input records is classified into one of a plurality of categories based on the signature detection result, wherein an amount of processing grows sublinearly with a number of signature rules being processed.