Dialogue Response Evaluation Using Semantic Clustering Entropy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current dialogue generation models produce dull and uninteresting responses due to a lack of semantic diversity, and existing evaluation methodologies fail to accurately measure this diversity, leading to biased training datasets that exacerbate the issue.
Innovation Solution
An apparatus and method that evaluate semantic diversity by clustering responses into latent vector spaces using k-means clustering and calculating semantic entropy values, and adjust training through weighted loss functions to balance response distributions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If lexical-level evaluation methodology (distinct-n, entropy-n) is used to measure response diversity, then evaluation can be performed, but it cannot accurately identify semantic diversity because responses with similar words may have very different meanings and responses with various words may have similar meanings
Solution Approach 1:
The patent introduces semantic vectors as an intermediary between lexical expressions and semantic meaning. These vectors are obtained through trained language models and serve as a bridge that captures the actual semantic content of responses, allowing for accurate semantic diversity evaluation without directly comparing complex semantic structures
Solution Approach 2:
The patent replaces the mechanical lexical-level comparison system (distinct-n, entropy-n) with a semantic vector-based evaluation system. Instead of counting word overlaps or calculating lexical entropy, the system uses semantic vectors in a latent space to represent and compare response meanings, substituting a simpler mathematical operation for a more accurate semantic measurement
2Productivity
If dialogue datasets with biased semantic distribution are used for training, then training can be performed, but the model generates responses that are not semantically diverse
Solution Approach 1:
The patent performs preliminary action by pre-calculating semantic vectors for all training responses and organizing them into clusters before training the dialogue generation model. This preprocessing step allows the training process to directly use cluster assignments and semantic diversity targets, eliminating the need for complex real-time semantic analysis during training and enabling the model to learn diverse response patterns more efficiently
Solution Approach 2:
The patent changes the training objective from simple lexical matching to semantic vector-based diversity optimization. By modifying the loss function to incorporate semantic entropy calculations based on cluster distributions, the training process directly optimizes for semantic diversity rather than just response accuracy, allowing the model to generate more varied responses while maintaining productivity
3Measurement precision
If cluster classifying information is obtained and semantic vectors are calculated for all training responses, then accurate semantic diversity evaluation is enabled, but computational resources and time are significantly increased
Solution Approach 1:
The patent calculates semantic vectors for training responses and determines cluster assignments in advance, before the actual evaluation process. This preliminary computation stores the heavy lifting of semantic analysis, so that during evaluation, the system only needs to compare pre-computed vectors and retrieve pre-determined cluster assignments, dramatically reducing evaluation time while maintaining high measurement precision
Data Source
AI summary
Provided is an apparatus for evaluating and improving responses, and a method and a computer readable recording medium thereof. The apparatus for evaluating responses according to the present disclosure obtains cluster classifying information for training responses, and based on distribution of clusters to which test responses output from the dialogue generation model are classified, evaluate semantic diversity of the responses output from the dialogue generation model.