Keyword Spotting Using Representative Dictionary
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing keyword searching techniques face challenges with large dictionaries that are too large to fit in fast memory, leading to slow search times and high false-positive detections, especially in applications like data leakage prevention and intrusion detection systems.
Innovation Solution
A method is introduced to derive a smaller representative dictionary from a full dictionary, using sub-strings that are shorter and exclude common phrases causing false detections, allowing for efficient search by storing the representative dictionary in faster cache memory and accessing the full dictionary only upon detection, thereby reducing search complexity and time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a full dictionary is stored in memory for keyword searching, then search completeness is improved, but memory usage increases and search speed decreases
Solution Approach 1:
The full dictionary is segmented into two parts: a compressed representative dictionary stored in fast cache memory for quick filtering, and the complete dictionary stored in slower main memory for verification. This segmentation allows the system to achieve fast initial filtering while maintaining complete match verification.
Solution Approach 2:
The representative dictionary acts as an intermediary between the input data and the full dictionary. It pre-filters potential matches using compressed patterns, reducing the number of queries to the full dictionary and thereby improving overall search speed while maintaining completeness.
2Speed
If a full dictionary is stored in cache memory, then search speed is improved, but memory capacity requirements increase
Solution Approach 1:
The invention extracts only the essential pattern information from the full dictionary to create a compressed representative dictionary. This extracted version contains sufficient information for initial filtering while occupying minimal cache memory space, allowing fast search without requiring large cache capacity.
Solution Approach 2:
The dictionary patterns are transformed from their original form into a compressed representation using bit manipulation and pattern matching techniques. This parameter transformation reduces the storage requirements while preserving the ability to identify potential keyword matches.
3Reliability
If common phrases are included in the dictionary, then search coverage is improved, but false-positive detections increase
Solution Approach 1:
The system dynamically adjusts the filtering criteria based on the detected pattern. When a potential match is found in the representative dictionary, the system performs a more rigorous verification against the full dictionary, adapting the level of scrutiny based on the initial filter result. This dynamic approach maintains coverage while reducing false positives.
Data Source
AI summary
Methods and systems for keyword spotting, i.e., for identifying textual phrases of interest in input data. In the embodiments described herein, the input data comprises communication packets exchanged in a communication network. The disclosed keyword spotting techniques can be used, for example, in applications such as Data Leakage Prevention (DLP), Intrusion Detection Systems (IDS) or Intrusion Prevention Systems (IPS), and spam e-mail detection. A keyword spotting system holds a dictionary of textual phrases for searching input data. In a communication analytics system, for example, the dictionary defines textual phrases to be located in communication packets—such as e-mail addresses or Uniform Resource Locators (URLs).