Language-Agnostic Short Form Detection via Max Pooling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems for detecting short forms and expanding them to their corresponding long forms are limited by language-specific rules, making them ineffective in multi-lingual environments and requiring complex text processing techniques.
Innovation Solution
A computer-implemented method and system that uses deep learning models, distributional semantics, and a novel string match scoring algorithm based on max pooling to detect short forms and their long forms independently of language, without relying on pre-defined stop word lists or complex text processing techniques, and provides confidence scores for long form expansions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If language-specific rules are used for short form detection, then detection accuracy for that language is improved, but the system cannot be effectively implemented in multi-lingual environments
Solution Approach 1:
The patent applies universality by creating a detection system that works across multiple languages using a single unified approach. The system uses character n-gram extraction and cosine similarity calculation that are language-agnostic, allowing the same algorithm to detect short forms in any language without requiring language-specific rule sets, thus achieving both multi-lingual capability and detection accuracy
2Measurement precision
If complex text processing techniques are used, then detection precision is improved, but system complexity and processing time increase
Solution Approach 1:
The patent applies extraction by isolating only the essential features needed for short form detection: character n-grams and cosine similarity. Instead of using complex text processing techniques, the system extracts relevant character sequences and compares them using a simple mathematical operation, achieving detection precision while minimizing system complexity
Solution Approach 2:
The patent replaces complex mechanical text processing systems with a mathematical approach using cosine similarity calculation. This substitution allows the system to achieve high detection precision through efficient vector space operations rather than complex rule-based or machine learning systems
Data Source
AI summary
Embodiments provide a system and method for short form and long form detection. Using a language-independent process, the detection system can ingest a corpus of documents, pre-process those documents by tokenizing the documents and performing a part-of-speech analysis, and can filter one or more candidate short forms using one or more filters that select for semantic criteria. Semantic criteria can include the part of speech of a token, whether the token contains more than a pre-determined amount of symbols or digits, whether the token appears too frequently in the corpus of documents, and whether the token has at least one uppercase letter. The detection system can detect short forms independent of case and punctuation, and independent of language-specific metaphone variants.