Semantic Representation Model Training via Lexicon Grammar Semantics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current semantic representation models, such as BERT and XLNet, have limited capabilities due to reliance on co-occurrence information of words or sentences, leading to undesirable accuracy in processing and understanding the complexity of training corpora.
Innovation Solution
A method for processing a semantic representation model by training it using a corpus set based on lexicon, grammar, and semantics, which includes capabilities like masking strategies, analyzing initial letter capitalization, predicting word occurrences, recognizing order and positional relationships, and understanding logical and query correlations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If semantic representation models are trained using only co-occurrence information of words or sentences, then the training process is simple, but the semantic representation accuracy is limited
Solution Approach 1:
The patent segments the training process into three distinct modules: a lexicon training module that trains the model on word-level information, a grammar training module that trains on sentence-level structural information, and a semantics training module that trains on paragraph-level contextual information. This segmentation allows the model to progressively learn at multiple granularities, improving semantic representation accuracy while managing training complexity through modular organization
Solution Approach 2:
The patent extends the training approach from traditional single-dimensional co-occurrence information to multi-dimensional training by incorporating lexicon (word-level), grammar (sentence-level), and semantics (paragraph-level) dimensions. This dimensional expansion enables the model to capture richer linguistic patterns and improve semantic representation accuracy beyond what single-dimensional training can achieve
2Adaptability or versatility
If semantic representation models use simple training approaches, then the training time is short, but the model capability is limited
Solution Approach 1:
The patent implements preliminary action by pre-training the model on large-scale corpora using the three-module approach before fine-tuning on specific tasks. The lexicon, grammar, and semantics modules are trained in advance to build comprehensive linguistic knowledge, enabling the model to adapt quickly to downstream NLP tasks with reduced training time while maintaining high capability
Solution Approach 2:
The patent creates a universal semantic representation model through multi-functionality by designing training modules that can handle diverse linguistic phenomena. The lexicon module handles word-level patterns, the grammar module handles syntactic structures, and the semantics module handles contextual meanings, making the model versatile across different NLP tasks while efficiently utilizing training time
Data Source
AI summary
The present disclosure provides a method, apparatus, electronic device and storage medium for processing a semantic representation model, and relates to the field of artificial intelligence technologies. A specific implementation solution is: collecting a training corpus set including a plurality of training corpuses; training the semantic representation model using the training corpus set based on at least one of lexicon, grammar and semantics. In the present disclosure, by building the unsupervised or weakly-supervised training task at three different levels, namely, lexicon, grammar and semantics, the semantic representation model is enabled to learn knowledge at levels of lexicon, grammar and semantics from massive data, enhance the capability of universal semantic representation and improve the processing effect of the NLP task.