Dialogue Response Evaluation Using Semantic Clustering Entropy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current dialogue generation models produce dull and uninteresting responses due to a lack of semantic diversity, and existing evaluation methodologies fail to accurately measure this diversity, leading to biased training datasets that further exacerbate the issue.
Innovation Solution
An apparatus and method that evaluate semantic diversity by clustering responses into latent vector spaces using k-means clustering and calculating semantic entropy values, and adjust training to balance response distributions using weight values to improve semantic diversity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If lexical-level evaluation methodology (distinct-n, entropy-n) is used to measure response diversity, then evaluation can be performed, but it cannot accurately identify semantic diversity because responses with similar words may have different meanings and responses with various words may have similar meanings
Solution Approach 1:
The patent introduces semantic vectors as an intermediary between lexical-level text and semantic diversity evaluation. These vectors capture the semantic meaning of responses, allowing for accurate semantic diversity measurement while maintaining computational efficiency through vector operations rather than complex linguistic analysis
Solution Approach 2:
The patent transforms the evaluation from lexical-level parameters (word frequency, n-grams) to semantic-level parameters (vector similarity, cluster distribution). This parameter transformation enables accurate semantic diversity measurement by operating in the semantic vector space where semantic relationships are preserved
2Productivity
If dialogue datasets with biased semantic distribution are used for training, then training can proceed, but the model generates responses that are not semantically diverse
Solution Approach 1:
The patent implements a feedback mechanism where the semantic diversity of the training dataset is evaluated using cluster-based semantic vectors. This feedback information is then used to identify and resample underrepresented semantic clusters, creating a balanced training dataset that improves both training efficiency and response diversity
Solution Approach 2:
The patent performs preliminary analysis of the training dataset's semantic distribution before training begins. By identifying biased semantic clusters in advance and resampling to create a balanced dataset, the system prevents the diversity problem from occurring during training rather than correcting it afterward
Data Source
AI summary
Provided is an apparatus for evaluating and improving responses, and a method and a computer readable recording medium thereof. The apparatus for evaluating responses according to the present disclosure obtains cluster classifying information for training responses, and based on distribution of clusters to which test responses output from the dialogue generation model are classified, evaluate semantic diversity of the responses output from the dialogue generation model.