Regex Matching System Using Grouped DFAs for Reduced Data Loading
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing regex matching methods require large Deterministic Finite Automata (DFA) that exceed the capacity of ordinary logical chips, leading to inefficient data loading and high resource consumption, especially when dealing with tens of thousands of check rules.
Innovation Solution
Sorting regexes into groups with common strings, compiling each group into smaller DFAs, and matching data streams against these generic strings to reduce the data needed for loading and improve performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple regex rules (tens of thousands) are compiled into a single large DFA, then all check rules can be applied without omission, but the DFA size becomes several hundreds of megabytes or even 1 G, which cannot be integrated into ordinary logical chips
Solution Approach 1:
The patent segments the large DFA into multiple smaller DFAs by grouping regex rules that share common strings. Instead of compiling all tens of thousands of rules into one monolithic DFA, the system divides them into manageable groups where each group shares a common substring prefix, creating multiple smaller DFAs that can be individually stored and loaded into the logical chip.
Solution Approach 2:
The patent extracts the common string portion from each regex rule and uses it as a key to organize rules into groups. By taking out the common string element, the system can identify which rules share structural similarities and group them together, enabling the extraction of smaller, manageable DFA fragments that can be stored in chip memory.
2Adaptability or versatility
If a large DFA is loaded into external SRAM or SDRAM for matching, then all regex rules can be accessed, but the data table entries are loaded repeatedly during state transitions, consuming too much time and storage resources
Solution Approach 1:
The patent segments the regex rules into groups based on common strings, creating multiple smaller DFAs. This segmentation allows the system to load only the relevant DFA fragment into the logical chip's cache memory when a particular common string is matched, rather than loading the entire large DFA repeatedly. Each smaller DFA can be independently managed and loaded only when needed.
Solution Approach 2:
The patent performs preliminary grouping of regex rules by common strings before the actual matching process. This pre-organization allows the system to quickly identify which smaller DFA fragment needs to be loaded into cache during matching, avoiding the need to scan through or reload the entire large DFA. The preliminary grouping creates an index structure that enables fast retrieval of relevant DFA fragments.
3Adaptability or versatility
If the DFA is made more complex to accommodate more rules, then coverage increases, but the number of data table entries to be loaded increases, consuming more storage resources and reducing matching performance
Solution Approach 1:
The patent applies segmentation by dividing the complex DFA into multiple simpler DFAs based on common string groups. Instead of creating one extremely complex DFA to handle tens of thousands of rules, the system creates multiple smaller, simpler DFAs that each handle a subset of rules sharing a common string prefix. This reduces the complexity of individual DFA structures while maintaining overall coverage.
Solution Approach 2:
The patent applies local quality by creating different DFA structures tailored to specific local needs. Each small DFA is optimized for its specific group of rules sharing a common string, rather than creating a uniform complex structure for all rules. This allows each local DFA to be simpler and more efficient, with fewer data table entries to manage, while collectively providing comprehensive rule coverage.
Data Source
AI summary
The present invention discloses a regex matching method and system, and relates to the field of computer technologies. The method includes: sorting multiple regexes into several regex groups, where all regexes in one regex group include a common string, which is known as a generic string; compiling each regex group into a DFA, and setting up a correlation between the generic string of each regex group and the DFA; matching to-be-matched data streams with the generic string respectively, and using the matched generic string as a matched string; obtaining a DFA corresponding to the matched string; and performing regex matching for the to-be-matched data streams according to the DFA, and outputting a matching result. The embodiments of the present invention shorten the data loading process, decrease the time consumed by data loading, and improve the matching performance.