DFA Tokenizer Profiles for Fast AI Malicious Traffic Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current AI-based Web Application Firewalls (WAFs) face performance limitations due to inefficient tokenizers, lack of unified APIs, and difficulty in handling emerging threats, making them inflexible and costly to maintain.
Innovation Solution
Implementing a Deterministic Finite Automata (DFA)-based tokenizer with a generator and tokenizer components that support user-defined token profiles, enabling high-performance tokenization and flexible tokenization capabilities for AI-based WAFs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional tokenizers are used in AI-based WAFs, then the system can process traffic with basic functionality, but the classification speed is slow and performance is limited
Solution Approach 1:
The patent replaces conventional tokenizer mechanisms with a DFA-based tokenizer that uses deterministic finite automata theory to achieve high-speed tokenization. The DFA tokenizer uses pre-compiled transition tables and state machines to process traffic, substituting the mechanical/conventional tokenization approach with a theoretically optimized automata-based system that delivers both high speed and high accuracy simultaneously
Solution Approach 2:
The patent changes the fundamental parameters of the tokenizer by adopting DFA theory with pre-compiled transition tables, changing the time complexity from linear scanning to constant-time state transitions. This parameter change enables the system to achieve 100 times faster classification speed while maintaining detection accuracy through the mathematical properties of deterministic finite automata
2Adaptability or versatility
If multiple different tokenizers are used to handle various tokenization needs, then the system can support diverse requirements, but the device complexity increases and maintenance becomes difficult
Solution Approach 1:
The patent creates a universal DFA-based tokenizer that can handle multiple tokenization requirements through a single unified system. The DFA tokenizer uses configurable transition tables and state definitions that can be adapted to different tokenization needs without requiring separate tokenizer implementations, making one system perform multiple functions
Solution Approach 2:
The patent segments the tokenization logic into modular DFA components with separate transition tables for different token types (SQL keywords, operators, identifiers, etc.). This segmentation allows the complex tokenization task to be divided into manageable state machines that can be independently configured and maintained, reducing overall system complexity
3Productivity
If conventional tokenization methods are used, then the implementation is straightforward with existing tools, but the processing speed is insufficient for high-performance WAF requirements
Solution Approach 1:
The patent applies preliminary action by pre-compiling the DFA transition tables and state machines during system initialization or offline processing. The transition tables are pre-calculated and stored in memory, allowing the runtime tokenization to proceed at maximum speed without performing complex computations during traffic processing, thus achieving high productivity
Data Source
AI summary
Methods and apparatus for a flexible Deterministic Finite Automata (DFA) tokenizer for AI-based malicious traffic detection. A DFA compiler is used to process profiles, such as SQLi, HTML5 and XSS profiles, as well as user-defined profiles, to generate corresponding DFA transition tables. The DFA tokenizer includes a DFA engine that employs the DFA transition table(s) to generate token sequences derived from input strings. The token sequences are converted into feature vectors using a feature extraction engine, and the feature vectors are used for training a machine learning/Artificial Intelligence (AI) model configured to perform binary classification (benign or malicious). During run-time, strings are extracted from input received via a network and tokenized with the DFA tokenizer to generate token sequences that are converted into feature vectors. The feature vectors are then classified using the AI model to determine whether the input is benign or malicious.