Dialect Language Model Creation Using Transformation Rules
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current language model training devices struggle to create robust language models for dialects due to the lack of comprehensive dialect corpora and the difficulty in assigning appropriate probabilities to dialect-containing n-grams, leading to poor recognition accuracy when input speech contains dialects.
Innovation Solution
A language model creation device and method that utilizes a standard language model by applying transformation rules to create dialect-containing n-grams and adding them to the standard language model, allowing for the creation of a robust language model capable of recognizing dialects effectively.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a language model is trained using standard language corpora, then the language model achieves high recognition accuracy for standard language, but it fails to recognize dialect-containing speech effectively
Solution Approach 1:
The patent segments the language model into two distinct components: a standard language model trained on standard language corpora and a dialect language model trained on dialect-containing corpora. This segmentation allows each model to specialize in its respective language variety, resolving the contradiction by enabling both high standard language accuracy and dialect recognition capability through separate specialized models rather than a single general model
Solution Approach 2:
The patent applies local quality by creating region-specific dialect language models tailored to particular dialects (e.g., Kansai-ben, Tokyo-ben). Each dialect model is trained on corpora specific to that dialect region, allowing the system to maintain high recognition accuracy for standard language while simultaneously adapting to local dialect characteristics through specialized local models
2Adaptability or versatility
If transformation rules are applied to create dialect-containing n-grams from a standard language model, then dialect recognition capability is improved, but the complexity of the language model creation process increases
Solution Approach 1:
The patent applies preliminary action by pre-defining transformation rules that map standard language n-grams to dialect-containing n-grams before training the dialect language model. These transformation rules are established in advance based on linguistic knowledge of dialect characteristics, allowing the system to systematically generate dialect corpora from standard language data without requiring manual annotation of each dialect example, thereby reducing overall process complexity
Solution Approach 2:
The patent uses copying by creating dialect-containing n-grams through transformation of existing standard language n-grams. Instead of manually constructing dialect corpora from scratch, the system copies the structure and patterns from the well-established standard language model and applies transformation rules to generate corresponding dialect variants, significantly reducing the complexity of dialect corpus creation
3Measurement precision
If dialect corpora are collected and used for language model training, then dialect recognition accuracy improves, but the difficulty and cost of corpus collection increase due to lack of comprehensive dialect data
Solution Approach 1:
The patent introduces transformation rules as an intermediary mechanism that bridges standard language corpora and dialect language models. Instead of directly collecting and processing difficult-to-obtain dialect data, the system uses transformation rules as a mediator to systematically generate dialect-containing n-grams from readily available standard language corpora, significantly easing the corpus collection process while maintaining dialect recognition accuracy
Solution Approach 2:
The patent applies self-service by enabling the language model creation system to automatically generate dialect corpora from standard language data through transformation rules. The system serves itself by internally creating the dialect training data it needs without requiring external manual collection of dialect corpora, reducing both the difficulty and cost of corpus acquisition while ensuring comprehensive dialect coverage
Data Source
AI summary
The present invention uses a language model creation device 200 that creates a new language model using a standard language model created from standard language text. The language model creation device 200 includes a transformation rule storage section 201 that stores transformation rules used for transforming dialect-containing word strings into standard language word strings, and a dialect language model creation section 203 that creates dialect-containing n-grams by applying the transformation rules to word n-grams in the standard language model and, furthermore, creates the new language model (dialect language model) by adding the created dialect-containing n-grams to the word n-grams.