Dialogue System OOD Detection via RNN Output Probability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing dialogue systems face challenges in accurately detecting Out-of-Domain (OOD) user inputs, requiring significant time and cost for rule construction or additional data collection, especially when determining whether user speech corresponds to unsupported requests or meaningless speech.
Innovation Solution
A dialogue system utilizing a natural language generator with a Recurrent Neural Network (RNN) to generate meaning representations and determine the processability of input sentences by calculating parameters based on output probabilities or rankings, allowing for OOD detection without constructing separate OOD data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If rule-based OOD detection is used, then OOD detection accuracy is improved, but time and cost for rule construction increase significantly
Solution Approach 1:
The patent uses the existing language model (trained on domain-specific data) to generate pseudo-OOD data by feeding it with domain sentences. This copying approach allows the system to create OOD detection training data without manually constructing rules or collecting additional OOD samples, thereby maintaining detection accuracy while reducing time and cost
Solution Approach 2:
The language model serves itself by generating OOD detection training data from its own training data. The model processes domain sentences through the language model to create pseudo-OOD samples, eliminating the need for external rule construction or separate OOD data collection processes
2Measurement precision
If machine learning-based OOD detection is used, then OOD detection capability is improved, but additional data construction time and cost increase
Solution Approach 1:
The patent generates pseudo-OOD data by copying and transforming existing domain data through the language model. This approach creates sufficient training samples for machine learning-based OOD detection without requiring separate OOD data collection, thereby improving detection capability while avoiding additional data construction costs
Solution Approach 2:
The system performs preliminary data preparation by generating pseudo-OOD samples from existing domain data before the actual OOD detection task. This preliminary action ensures that the machine learning model has adequate training data ready, eliminating the need for additional data construction during deployment
3Measurement precision
If separate OOD data is collected for learning, then OOD detection accuracy is improved, but data collection cost and time increase
Solution Approach 1:
Instead of collecting separate OOD data, the patent generates pseudo-OOD data by processing domain sentences through the language model. This copying mechanism creates synthetic OOD samples that maintain the statistical properties needed for accurate OOD detection while eliminating the need for expensive and time-consuming data collection
Solution Approach 2:
The language model serves multiple functions: it acts as both the domain language understanding model and the OOD data generation engine. This multi-functionality allows the same model to be used for both domain task performance and OOD detection training data generation, reducing overall system cost and complexity
Data Source
AI summary
A dialogue system includes a processor configured to: generate a meaning representation corresponding to an input sentence by performing Natural Language Understanding on the input sentence, generate an output sentence corresponding to the input meaning representation based on Recurrent Neural network (RNN), and determine whether the input sentence cannot be processed using the natural language generator. The processor calculates a parameter representing a probability of outputting the input sentence when the meaning representation corresponding to the input sentence is input to the natural language generator, and determines whether the input sentence cannot be processed based on the calculated parameter.