Language Model Training Text Selection via Structural Templates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for training language models using out-of-target-field corpora are inefficient, especially with large vocabularies, as they struggle to select sentences with high generation probabilities due to unknown words, leading to low probabilities for similar sentences outside the target field.
Innovation Solution
A method involving the generation of templates by replacing words in word strings from a target field corpus with special symbols, allowing for the selection of similar sentences from out-of-target-field corpora based on coverage rates, enabling efficient collection of training text without relying on statistical models from the target field.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If statistical models from the target field are used to select sentences from out-of-target-field corpora, then sentences with high generation probabilities can be selected, but unknown words cause low probabilities for similar sentences outside the target field
Solution Approach 1:
The patent extracts only the structural pattern from target field sentences by replacing specific words with special symbols (e.g., <WORD1>, <WORD2>). This extraction creates templates that capture the grammatical and semantic structure without being constrained by domain-specific vocabulary, allowing the same templates to match sentences from out-of-target-field corpora even when those sentences contain unknown words.
Solution Approach 2:
The patent changes the parameter of sentence representation from exact word matching to template-based structural matching. By transforming concrete sentences into abstract templates with special symbols, the system changes how similarity is measured - not by word overlap but by structural conformity to the template, thereby enabling cross-domain applicability.
2Manufacturing precision
If manual dictating operation is used to collect natural language training sentences, then high quality training data can be obtained, but the cost is high and the amount of text sentences is limited
Solution Approach 1:
The patent creates template copies from a small set of high-quality target field sentences. Instead of manually creating many sentences, the system generates multiple template patterns from limited examples and uses these templates to automatically identify and select corresponding sentences from large out-of-target-field corpora, thereby multiplying the effective training data without proportional increases in manual effort.
Solution Approach 2:
The patent performs preliminary action by first creating templates from the target field corpus before selecting sentences from the out-of-target-field corpus. This preliminary template creation establishes the selection criteria in advance, enabling efficient automated filtering of large corpora without requiring manual evaluation of each sentence.
3Adaptability or versatility
If templates with special symbols are generated from target field corpus, then selection of similar sentences from out-of-target-field corpora is enabled, but the process complexity increases
Solution Approach 1:
The patent segments sentences into discrete word units and replaces each unit with a special symbol template. This segmentation transforms continuous text into structured, discrete elements that can be systematically processed and matched, making the template generation and matching process more manageable despite the increased complexity.
Data Source
AI summary
Method of selecting training text for language model, and method of training language model using the training text, and computer and computer program for executing the methods. The present invention provides for selecting training text for a language model that includes: generating a template for selecting training text from a corpus in a first domain according to generation techniques of: (i) replacing one or more words in a word string selected from the corpus in the first domain with a special symbol representing any word or word string, and adopting the word string after replacement as a template for selecting the training text; and/or (ii) adopting the word string selected from the corpus in the first domain as the template for selecting the training text; and selecting text covered by the template as the training text from a corpus in a second domain different from the first domain.