Domain-Specificity Prediction for NLP Text Terms
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current natural language processing techniques lack an effective method to determine the domain-specificity of text terms, which is crucial for applications like search, summarization, and term extraction, as they struggle to accurately rank multiword expressions by their domain specificity.
Innovation Solution
A method involving subword-unit tokenizers trained with different vocabulary sizes on domain-specific text corpora is used to generate a domain-specificity fingerprint for text terms, allowing for the determination of their domain specificity by analyzing the number of subword units required to represent them across various domains.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If current natural language processing techniques are used to determine domain-specificity of text terms, then the system can process text data, but it cannot accurately rank multiword expressions by their domain specificity
Solution Approach 1:
The patent segments text terms into subword units using domain-specific tokenizers. Each tokenizer breaks down multiword expressions into constituent subword tokens, allowing the system to analyze the composition and origin of each term. This segmentation enables precise measurement of domain-specificity by tracking which subword units appear in which domains.
Solution Approach 2:
The patent changes the parameter of vocabulary size for different domain-specific tokenizers. Each domain (e.g., biology, chemistry, computer science) has its own tokenizer trained on domain-specific corpora with appropriate vocabulary sizes. This parameter change allows the system to capture domain-specific terminology patterns and accurately determine whether a term belongs to a specific domain or is通用 across multiple domains.
2Measurement precision
If multiple domain-specific tokenizers with different vocabulary sizes are trained and used to generate domain-specificity fingerprints, then domain-specificity prediction accuracy is improved, but computational complexity increases
Solution Approach 1:
The patent creates a universal framework where multiple domain-specific tokenizers work together through a common interface. Each tokenizer is specialized for its domain but they all produce standardized subword unit representations that can be processed by the same domain-specificity fingerprinting algorithm. This multi-functionality approach allows the system to handle multiple domains simultaneously while maintaining a unified analysis pipeline.
Solution Approach 2:
The patent implements a nested structure where domain-specific tokenizers are embedded within a larger domain-specificity analysis system. Each tokenizer nests its domain-specific vocabulary and subword units within the broader framework of the fingerprinting algorithm. This nesting allows the complex tokenizer system to be organized hierarchically, with each layer serving a specific function while contributing to the overall analysis.
3Measurement precision
If domain-specific text corpora are used to train tokenizers for each domain, then the ability to identify domain-specific terms is improved, but data processing time and resource requirements increase
Solution Approach 1:
The patent performs preliminary action by pre-training domain-specific tokenizers on domain corpora before they are needed for actual domain-specificity analysis. The tokenizers are trained offline on large domain-specific corpora (e.g., biology, chemistry, computer science texts) to learn domain-specific subword patterns and vocabulary. This preliminary training allows the system to quickly process queries without retraining, as the tokenizers are already prepared and optimized for their respective domains.
Data Source
AI summary
A method, computer-program product and computer system are provided to determine domain-specificity of a text term. A processor receives a plurality of domain-specific text corpora, wherein each of the plurality of domain-specific text corpora comprises a plurality of text documents of a respective domain. A processor trains a set of subword-unit tokenizers with at least two different vocabulary sizes of the respective domain-specific text corpus. A processor receives the text-term. A processor determines a domain-specificity fingerprint of the text-term, wherein the domain-specificity fingerprint comprises for each subword-unit tokenizer a number of subword-units required to represent the text-term. A processor provides the domain-specificity fingerprint for determining the domain-specificity of the text term.