Language Model Score Calculation Using Meta-Information Vectors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional multi-party conversation context-aware language models do not explicitly understand meta-information in past speeches, leading to insufficient context and difficulty in language prediction.
Innovation Solution
A device comprising a history speech meta-information understanding unit, history speech embedding unit, speech unit combination vector construction unit, speech sequence embedding vector calculation unit, and language model score calculation unit, which estimates and utilizes meta-information vectors to improve language model scoring.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If only word string and speaker information are used as context information in multi-party conversation language models, then the model structure remains relatively simple, but the context information is insufficient and language prediction accuracy deteriorates
Solution Approach 1:
The patent segments context information into multiple distinct types: word string information, speaker information, and meta-information (speech intentions and topics). Each type is processed separately through dedicated embedding layers and attention mechanisms, allowing the model to handle complex multi-dimensional context without overwhelming structural complexity
Solution Approach 2:
The patent adds meta-information as an additional dimension to the traditional context representation. By incorporating speech intentions and topics as separate embedding vectors that are concatenated with word and speaker embeddings, the model expands the context space from two dimensions (word and speaker) to four dimensions (word, speaker, intention, topic), significantly improving language prediction accuracy
2Loss of information
If meta-information understanding device is added to extract speech intentions and topics, then context information becomes sufficient for accurate language prediction, but the device complexity increases
Solution Approach 1:
The patent applies preliminary action by extracting and embedding meta-information (speech intentions and topics) from historical speeches before the main language modeling process. The meta-information understanding device processes past speeches to generate intention and topic embeddings in advance, which are then integrated into the context representation for current speech prediction, ensuring complete context information is available when needed
Solution Approach 2:
The patent makes the context representation universal by designing a multi-functional embedding structure that simultaneously accommodates word strings, speaker information, speech intentions, and topics. The attention mechanism operates uniformly across all these different types of information, allowing the same core language modeling architecture to handle diverse context sources without requiring separate specialized structures for each information type
Data Source
AI summary
The present invention improves the accuracy of language prediction. A history speech meta-information understanding unit 11 obtains a history speech meta-information vector from a word string of a preceding speech using a meta-information understanding device. A history speech embedding unit 12 converts the word string of the preceding speech and a speaker label into a history speech embedding vector. A speech unit combination vector construction unit 13 obtains a speech unit combination vector by combining the history speech meta-information vector and the history speech embedding vector. A speech sequence embedding vector calculation unit 14 converts a plurality of speech unit combination vectors obtained for the past speech sequences to a speech sequence embedding vector. A language model score calculation unit 15 calculates a language model score of a current speech from a word string of the current speech, a speaker label, and a speech sequence embedding vector.