Language Model Training Apparatus for Technical Term Handling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional language models trained on general documents struggle to handle technical terms effectively, often treating them as unknown words or dividing them into combinations of known short words, leading to inadequate vocabulary and performance in specialized fields like finance and infrastructure.
Innovation Solution
A language model training apparatus that includes an obtainment unit, a generation unit, and a training unit, which obtains target sentences, generates position information and token sequences for both teacher and student models, and updates student model parameters using knowledge distillation based on similarity and masked language model losses to handle new words and technical terms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a language model is trained using general documents with a large-scale corpus, then the model can handle common words well, but it fails to effectively handle technical terms and unknown words in specialized fields
Solution Approach 1:
The patent applies preliminary action by pre-processing the training corpus to identify and extract technical terms before model training. The system creates a specialized vocabulary list from domain-specific documents, then uses this pre-processed vocabulary during training to ensure technical terms are properly recognized and handled, rather than treating them as unknown words during inference
Solution Approach 2:
The patent implements local quality by applying different processing strategies to different types of words within the same model. Technical terms identified from domain-specific corpus are handled with specialized embedding and attention mechanisms, while common words use standard processing. This allows the model to have enhanced capability for technical terms without sacrificing performance on general vocabulary
2Adaptability or versatility
If the vocabulary is increased to handle unknown words and technical terms, then the model can recognize more words, but the training complexity and computational resources increase significantly
Solution Approach 1:
The patent applies segmentation by dividing the training process into distinct phases: first training on general domain corpus to learn common language patterns, then fine-tuning on domain-specific corpus to learn technical terms. The vocabulary is also segmented into common words and technical terms, with different embedding initialization strategies applied to each segment, reducing overall training complexity
Solution Approach 2:
The patent implements universality by designing a multi-functional training framework that can handle both general domain and specialized domain documents using the same base model architecture. The system uses a unified tokenizer and embedding layer that can process both common words and technical terms, eliminating the need for separate models for different vocabulary types
3Ease of manufacture
If conventional training methods are used with completely initialized embedding matrices, then the training process is straightforward, but the model loses the ability to leverage pre-existing word knowledge and learns from scratch
Solution Approach 1:
The patent applies preliminary action by pre-initializing the embedding matrix with pre-trained word embeddings from large-scale general domain corpora before domain-specific training. This pre-initialization preserves general language knowledge and allows the model to start from a knowledgeable state rather than learning basic language patterns from scratch, significantly improving learning efficiency while maintaining training simplicity
Data Source
AI summary
A language model training apparatus obtains a target sentence including a sequence of a word that is a known word or a new word. The apparatus generates position information of a new word included in the target sentence and a token sequence that is a sequence of an identifier of the word included in the target sentence. The token sequence includes a first token sequence for a teacher model and a second token sequence for a student model. The first token sequence and the second token sequence are different in at least some of a processed word. The apparatus updates a parameter of the student model utilizing knowledge distillation from the teacher model to the student model based on the position information, the first token sequence, and the second token sequence.