Dynamic Dictionary for Continuous Keyword Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing keyword searching systems face challenges in dynamically updating static dictionaries without disrupting ongoing traffic analysis, as recompilation of static dictionaries renders them ineffective for new keywords until completion, and irrelevant keywords may be added during this process.
Innovation Solution
The implementation of dynamic dictionaries, comprising management and runtime automata, allows for on-the-fly updates and searches without recompilation, using a static dictionary in conjunction with two dynamic dictionaries to maintain keyword detection efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a static dictionary is recompiled to add new keywords, then the dictionary includes updated keywords, but traffic analysis cannot be performed on new keywords during recompilation and irrelevant keywords may be added
Solution Approach 1:
The dictionary is divided into two separate components: a static dictionary that maintains a stable, pre-compiled set of keywords for reliable matching, and a dynamic dictionary that handles new keywords separately. This segmentation allows the static dictionary to remain unchanged during updates, preserving detection accuracy, while the dynamic dictionary adapts to new keywords without requiring recompilation of the entire dictionary structure.
Solution Approach 2:
A dynamic dictionary acts as an intermediary layer between the static dictionary and incoming traffic containing new keywords. The dynamic dictionary temporarily holds and processes new keywords before they are potentially integrated into the static dictionary, allowing continuous keyword detection without interruption or risk of adding irrelevant keywords during the transition period.
2Speed
If a static dictionary is recompiled frequently to add new keywords promptly, then new keywords are detected quickly, but system performance degrades due to repeated recompilation
Solution Approach 1:
The system implements a dynamic dictionary that can be updated in real-time without recompilation, allowing new keywords to be added immediately as they are detected in traffic. This dynamic approach eliminates the need for frequent static dictionary recompilations, maintaining high keyword update speed while preserving system throughput since the static dictionary remains stable and optimized.
3Adaptability or versatility
If the static dictionary is kept large to include all possible keywords, then comprehensive keyword detection is achieved, but memory usage and search time increase
Solution Approach 1:
The keyword dictionary is segmented into a compact static dictionary containing frequently occurring and high-priority keywords, and a dynamic dictionary that handles less frequent or newly observed keywords. This segmentation allows the static dictionary to remain small and efficient for fast matching, while still achieving comprehensive coverage through the dynamic component that can accommodate additional keywords without expanding the static structure.
Data Source
AI summary
An apparatus and techniques for constructing and utilizing a “dynamic dictionary” that is not a compiled dictionary, and therefore does not need to be recompiled in order to be updated. The dynamic dictionary includes respective data structures that represent (i) a management automaton that includes a plurality of management nodes, and (ii) a runtime automaton that is derived from the management automaton and includes a plurality of runtime nodes. The runtime automaton may be used to search input data, such as communication traffic over a network, for keywords of interest, while the management automaton manages the addition of keywords to the dynamic dictionary. Typically, at least two (e.g., exactly two) such dynamic dictionaries are used in combination with a static dictionary.