Stateful Email Detection Using Schemaless Context Fragments
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Email scanners lack the ability to detect malicious content effectively due to their inability to analyze emails within the context of a conversation and historical relationships between senders and recipients.
Innovation Solution
A method and system that stores schemaless data fragments from previous emails in a datastore, indexed by derived keys, and uses these fragments to provide context for determining the suspiciousness of new emails, including actions based on threat levels.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If emails are scanned individually without context, then the scanning process is simple and fast, but the detection accuracy of malicious content is reduced
Solution Approach 1:
The system performs preliminary actions by extracting and storing data fragments from emails during the scanning process. These fragments (sender, recipient, subject, body, attachments, headers, metadata) are saved in a datastore before new emails arrive, enabling contextual analysis without adding complexity to the core scanning mechanism
Solution Approach 2:
A datastore acts as an intermediary between the email scanner and the detection system. It stores schemaless data fragments from previous emails and provides them as context to the scanner when analyzing new emails, allowing the system to maintain detection accuracy while keeping the scanning process simple
2Reliability
If contextual data from previous emails is stored and analyzed, then the detection accuracy improves, but the data storage and processing complexity increases
Solution Approach 1:
The email data is segmented into discrete, schemaless fragments (sender, recipient, subject, body, attachments, headers, metadata) that are stored independently in the datastore. This segmentation allows flexible retrieval and combination of relevant context without requiring complex structured storage schemes
Solution Approach 2:
The system changes the data storage parameter from structured relational databases to schemaless document storage. This allows the system to store variable-length email contexts with different attributes without predefined schemas, reducing storage complexity while maintaining detection accuracy
3Adaptability or versatility
If schemaless data fragments are used to store email contexts, then the system becomes more flexible and adaptable, but the data retrieval and matching complexity increases
Solution Approach 1:
When a new email arrives, the system creates a copy of its data fragments and attempts to match them against stored fragments in the datastore. This copying approach allows flexible schemaless storage while simplifying retrieval by using direct fragment comparison rather than complex queries
Solution Approach 2:
The system implements feedback by using matching data fragments from previous emails to inform the scanning of new emails. The context from matched fragments is fed back into the scanner, improving detection accuracy while the matching process remains simple through direct fragment comparison
Data Source
AI summary
A computer-implemented method includes sending email scan requests to an email scanner. The method further includes receiving, from the email scanner, a verdict of suspicion and one or more data fragments. The method further includes storing the one or more data fragments for each email of the plurality of emails in a datastore. The method further includes receiving a new email. The method further includes deriving one or more new keys for the new email. The method further includes retrieving one or more matching data fragments from the datastore by matching the one or more new keys with the one or more keys stored in the datastore. The method further includes providing, to the email scanner, the new email and the one or more matching data fragments. The method further includes receiving a new verdict of suspicion and one or more new data fragments.