Text Message Filtering via Dynamic Tokenization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current content filtering systems for text messages, such as email, are ineffective in handling short text messages with abbreviations and misspellings, and fail to discern user-generated content lacking discernable patterns, leading to the inability to accurately filter out undesirable content.
Innovation Solution
A system and method that tokenizes text messages, compares generated tokens with databases, and calculates probabilities based on historical data and tokenization strategies to determine suitability, with dynamic adaptation to content changes and user feedback for improved filtering accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If automated software filtering is used with keyword matching, then filtering speed and productivity are improved, but filtering accuracy deteriorates due to inability to handle abbreviations and misspellings
Solution Approach 1:
The system dynamically adapts its tokenization strategy based on the characteristics of the text message. It selects from multiple tokenization approaches (word-based, character-based, n-gram) depending on the message structure, enabling the filter to handle both standardized keywords and creative misspellings effectively while maintaining high processing speed
Solution Approach 2:
The system changes the parameter of token granularity by selecting different tokenization strategies. For messages with abbreviations or misspellings, it uses character-level or n-gram tokenization instead of word-level, allowing the filter to capture patterns in distorted text while maintaining automated processing efficiency
2Measurement precision
If human filtering is used, then filtering accuracy is improved, but time consumption and cost increase
Solution Approach 1:
The system performs self-learning by automatically analyzing filtered messages and updating its tokenization strategies and probability models without human intervention. This enables the automated system to continuously improve its accuracy while maintaining high processing speed, eliminating the need for slow human review
Solution Approach 2:
The system incorporates feedback loops where filtering results are used to refine future filtering decisions. By analyzing patterns in messages that require human review, the system adapts its tokenization and probability calculation methods, progressively improving accuracy while maintaining automated processing
3Device complexity
If traditional keyword filtering is used, then device complexity is reduced, but adaptability to new content patterns deteriorates
Solution Approach 1:
The system segments the filtering task into multiple independent components: tokenization strategy selection, token generation, database comparison, and probability calculation. This modular approach allows each component to be optimized independently while maintaining overall system adaptability to new content patterns
Solution Approach 2:
The system implements a universal tokenization framework that can handle multiple types of text patterns (standard words, abbreviations, misspellings, slang) using a single adaptable architecture. By selecting from different tokenization strategies based on message characteristics, the system achieves high versatility without proportionally increasing complexity
Data Source
AI summary
A system and method is provided for filtering text messages to determine which of the messages, if any, contains undesirable or unsuitable content. The method includes receiving a message containing at least one text character. The text is manipulated to generate a plurality of tokens that are compared with data contained in a plurality of databases. As a result, a plurality of probabilities are generated related to the suitability of the message. The probabilities are weighted to generate an overall probability related to the suitability of the message. The overall probability is then compared to a threshold value to rate the suitability of the message. Further, the rating associated with the message is displayed on a computer display screen or the like.