Dynamic Lexicon Generation for Suspicious Email Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing email filtering systems lack a user-friendly method to achieve desired accuracy in detecting information leakage, relying on pre-defined keywords and frequency analysis, and do not utilize user feedback to provide multiple options for word schemes.
Innovation Solution
An automatic lexicon generation system that extracts key phrases and assigns weights dynamically, using user-provided example documents and emails, iteratively refining the list to meet specified accuracy levels, and allowing users to simulate modifications for flexible policy creation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If pre-defined keywords and frequency analysis are used for email filtering, then the system can operate automatically, but the detection accuracy is insufficient and cannot adapt to user needs
Solution Approach 1:
The system incorporates feedback mechanisms where users can rate the relevance of detected keywords and provide feedback on false positives and false negatives. This feedback is used to iteratively refine and update the lexicon, improving detection accuracy over time while maintaining automated operation.
Solution Approach 2:
The lexicon is designed to be dynamic rather than static, automatically adapting to changing email patterns and terminology. The system continuously learns from new data and user feedback, allowing it to maintain high detection accuracy across evolving communication patterns without requiring manual reconfiguration.
2Measurement precision
If manual creation of keyword lists is performed, then detection accuracy can be optimized, but the process is time-consuming and not user-friendly
Solution Approach 1:
The system performs self-service by automatically generating and refining keyword lists without requiring manual intervention. It autonomously analyzes email data, identifies patterns, creates lexicons, and optimizes detection parameters, eliminating the need for users to manually create keyword lists while maintaining high accuracy.
Solution Approach 2:
The system performs preliminary actions by pre-processing and analyzing large volumes of email data to automatically generate initial keyword lists and detection policies. This preliminary work is done automatically before user review, significantly reducing the time required for policy creation while maintaining optimized detection accuracy.
3Adaptability or versatility
If a single fixed keyword list is used, then the system is simple to operate, but it cannot adapt to changing email patterns and user needs
Solution Approach 1:
The keyword list is transformed into a dynamic structure that automatically adapts to changing email patterns, terminology, and user needs. The system continuously updates the lexicon based on new data and feedback, maintaining adaptability while preserving ease of operation through automated updates rather than manual reconfiguration.
Solution Approach 2:
The system uses feedback loops where users can rate detected keywords and provide feedback on detection quality. This feedback is automatically processed to refine and update the keyword list, enabling the system to adapt to changing patterns and user needs while maintaining simple operation through automated learning rather than manual adjustment.
Data Source
AI summary
A system for generating a lexicon of words, organized into weighted categories, from a user defined set of example documents for detecting suspicious e-mails from a mail archive is provided. The system uses a set of example documents and e-mails given by the user to probabilistically find possible lists of critical words. The obtained list is now applied on an archive of e-mails. The system generates an inverted index on the mails from the archive to facilitate search for the key phrases. User feedback is taken on the results obtained and corrections in the lexicon made if necessary. Thus, the mails are scanned based on user feedback, user defined words and automatically generated word list. These lists constantly adapt as e-mails in the archive change. The system then combines all these to present the user with several possible sets of keywords and their relative importance that can be used as a policy for a desired level of accuracy. The system also shows the user any change if the set is modified. Finally, the system searches through the entire mail archive to find suspicious e-mails.