Summary Evaluation Using Semantic Tuple Classes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing automated summary evaluation methods, such as the Basic Elements score, unfairly favor tuples that appear frequently across reference summaries, leading to large score differences and inability to distinguish tuples with similar meanings.
Innovation Solution
A summary evaluation device and method that extracts tuples of head and modifier words with dependency relations, replaces words with semantic classes, and calculates scores based on the overlap of these classes between reference and system summaries, ignoring frequency and focusing on semantic similarity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the Basic Elements score is calculated based on tuple frequency across reference summaries, then tuples appearing in multiple reference summaries receive higher scores, but this creates large score gaps and unfairly favors common tuples over contextually important tuples
Solution Approach 1:
The patent changes the parameter for tuple comparison from frequency-based scoring to semantic similarity-based scoring. By replacing words with their semantic classes and comparing tuples based on class similarity rather than occurrence frequency, the system eliminates the bias toward common tuples while maintaining the ability to identify contextually important information.
Solution Approach 2:
The patent creates a simplified representation of tuples by replacing actual word strings with semantic class labels. This copying approach preserves the structural information of tuples while eliminating the frequency bias, allowing fair comparison between tuples with different word choices but similar meanings.
2Measurement precision
If exact string matching is used to determine identical tuples, then precise matching is achieved, but tuples with substantially the same meaning cannot be distinguished as identical
Solution Approach 1:
The patent changes the matching parameter from exact string equality to semantic class equality. By mapping words to their semantic classes and comparing tuples based on class labels rather than word strings, the system recognizes semantic equivalence between tuples that would otherwise be considered different, improving both precision and adaptability.
3Quantity of substance
If article+noun tuples are scored based on frequency, then common nouns receive higher scores, but this does not account for the contextual importance of verb-noun relationships
Solution Approach 1:
The patent segments the evaluation into two components: tuple extraction that preserves contextual relationships, and semantic class mapping that enables meaningful comparison. By maintaining the head-word-modifier-word structure and mapping each component to its semantic class, the system preserves contextual information while enabling fair scoring based on semantic similarity rather than frequency.
Data Source
AI summary
The present disclosure relates to a method of evaluating accuracy of a summary of a document. The method includes receiving a plurality of reference summaries of a document and a system summary of the document. The system summary is generated by a machine. The method further includes extracting, for each reference summary, a tuple that is a pair of words composed of a modified word and a dependent word having a dependency relation to the modified word and a label representing the dependency relation. The method further includes replacing, for each of the extracted tuples, each of the modified word of the tuple's word pair and the dependent word with a class predetermined for the words. The method further generates a score of the system summary based on the class and a set of tuples of the system summary.