Neural Network Language Model Semantic Context Integration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current computer-based speech recognition techniques rely on local contexts, which are inadequate for accurately predicting next words, especially when the context is generic or occupied by terms like 'it is', leading to confusion between similar words such as 'surgeon' and 'sturgeon'.
Innovation Solution
Incorporating semantic context information modeled through latent semantic analysis (LSA) into neural network language models, which includes both local and semantic context layers to generate more accurate probability values for predicting the next word in a conversation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If only local context is used for next word prediction, then the system is simple to implement, but prediction accuracy deteriorates when context is generic or ambiguous
Solution Approach 1:
The patent combines local context information (previous words in sequence) with semantic context information (topic modeling from LSA) into a unified prediction framework. The neural network accepts both types of context as input features, merging them to produce more accurate next word predictions, especially in cases where local context alone is ambiguous or generic.
Solution Approach 2:
The patent adds a new dimension of semantic context to the traditional local context approach. By incorporating topic vectors from latent semantic analysis as an additional input dimension to the neural network, the system moves from considering only sequential word patterns to also considering semantic topic information, thereby improving prediction accuracy without simply extending the local context window.
2Reliability
If semantic context information is incorporated into the neural network, then prediction accuracy for ambiguous words improves, but computational complexity increases
Solution Approach 1:
The patent performs semantic context analysis and topic modeling in advance of the prediction task. The LSA model pre-processes the document corpus to create topic vectors and semantic representations that can be efficiently retrieved and fed into the neural network during prediction, rather than computing semantic contexts in real-time during the prediction process.
Solution Approach 2:
The patent introduces topic vectors as an intermediary representation between the raw text input and the neural network prediction. These topic vectors serve as a compressed semantic summary that bridges the gap between extensive document content and the compact neural network input requirements, reducing computational complexity while preserving semantic information.
Data Source
AI summary
In one implementation, a computer-implemented method includes receiving, at a computer system, a request to predict a next word in a dialog being uttered by a speaker; accessing, by the computer system, a neural network comprising i) an input layer, ii) one or more hidden layers, and iii) an output layer; identifying the local context for the dialog of the speaker; selecting, by the computer system and using a semantic model, at least one vector that represents the semantic context for the dialog; applying input to the input layer of the neural network, the input comprising i) the local context of the dialog and ii) the values for the at least one vector; generating probability values for at least a portion of the candidate words; and providing, by the computer system and based on the probability values, information that identifies one or more of the candidate words.