Multiscale Text Fingerprints for Spam Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for detecting spam and online fraud are vulnerable to obfuscation techniques used by spammers and fraudsters, such as misspelling and embedding content within larger blocks of text, which can lead to false negatives due to changes in text resulting in different hashes.
Innovation Solution
A system that calculates a text fingerprint for electronic documents by processing text blocks into a sequence of characters within a predetermined length, using a combination of hashing and zoom-in/zoom-out methods to efficiently compare and classify documents, employing a fingerprint calculator to determine a text fingerprint and a document classifier to compare it with reference fingerprints for classification.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If conventional hashing methods are used for spam detection, then detection speed is improved, but detection accuracy deteriorates due to vulnerability to obfuscation techniques
Solution Approach 1:
The patent segments the text into multiple tokens and generates multiple local fingerprints for different token sequences (bigrams, trigrams, etc.). Instead of hashing the entire text as one unit, it creates a set of local fingerprints that can individually detect obfuscated spam content, thereby maintaining detection accuracy while preserving speed through efficient local comparisons.
Solution Approach 2:
The patent transitions from single-dimension global hashing to multi-dimensional local fingerprinting by considering multiple token sequences and their relationships. It adds dimensional depth by examining n-grams and their positional relationships, enabling detection of obfuscated content that would be missed by conventional single-hash methods.
2Reliability
If text is segmented into multiple tokens for fingerprinting, then detection accuracy is improved, but computational complexity increases
Solution Approach 1:
The patent applies partial action by selectively generating local fingerprints only for relevant token sequences rather than processing all possible combinations. It uses a threshold-based approach where only token sequences meeting certain criteria are fingerprinted, reducing computational overhead while maintaining sufficient detection accuracy.
Solution Approach 2:
The patent merges multiple local fingerprint results into a unified detection decision. By combining evidence from various token sequences and their local fingerprints, it achieves high detection accuracy through aggregation of partial results, avoiding the need to process every possible token combination independently.
3Productivity
If fingerprint length is constrained to predetermined range, then comparison efficiency is improved, but information loss increases
Solution Approach 1:
The patent dynamically adjusts fingerprint parameters including length constraints and token sequence lengths based on the specific detection context. By changing these parameters adaptively rather than using fixed values, it optimizes the balance between comparison efficiency and information retention for different types of spam detection scenarios.
Solution Approach 2:
The patent applies different fingerprinting strategies to different local regions of the text based on their characteristics. Important token sequences receive more detailed fingerprinting while less critical regions use simplified approaches, thereby maintaining overall information quality while improving comparison efficiency through localized optimization.
Data Source
Figure 1~2-A
Figure 2-B~2-C
Figure 3-A~3-B
AI summary
Described systems and methods allow a classification of electronic documents such as email messages and HTML documents, according to a document-specific text fingerprint. The text fingerprint is calculated for a text block of each target document, and comprises a sequence of characters determined according to a plurality of text tokens of the respective text block. In some embodiments, the length of the text fingerprint is forced within a pre-determined range of lengths (e.g. between 129 and 256 characters) irrespective of the length of the text block, by zooming in for short text blocks, and zooming out for long ones. Classification may include, for instance, determining whether an electronic document represents unsolicited communication (spam) or online fraud such as phishing.