Combinatorial Summarizer Engine Ranking and Weighting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Human evaluation of summaries from different summarization engines is limited to binary qualitative data, lacking a comprehensive and unbiased method for evaluating and combining their outputs effectively.
Innovation Solution
A combinatorial approach that combines the outputs of multiple summarization engines using a processor with computer-readable instructions, incorporating both weighting schemes and human feedback to rank and weight sentences, enabling a quantitative and non-biased evaluation of summarization engines.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If human evaluation is used to rank summaries from different summarization engines, then qualitative feedback is obtained, but the evaluation is limited to binary data and lacks comprehensive quantitative analysis
Solution Approach 1:
The patent combines multiple summarization engine outputs by merging their respective sentence rankings and weights. The system integrates results from different engines, applying combination rules that consider both individual engine rankings and cross-engine agreement to produce a consolidated summary ranking that leverages the strengths of multiple engines simultaneously.
Solution Approach 2:
The system implements feedback mechanisms where human evaluation results are fed back into the summarization process. Human rankings of summary quality are used to adjust and reweight engine outputs, creating a closed-loop system where evaluation information continuously improves the combination strategy and enhances subsequent summarization performance.
2Reliability
If multiple summarization engines are used to improve summary quality, then accuracy and robustness increase, but the complexity of evaluating and combining their outputs increases
Solution Approach 1:
The patent segments the summarization system into independent engine components and a separate combination module. Each summarization engine operates independently to generate its own ranked summary, and the combination module separately processes these independent outputs using defined combination rules, allowing the system to leverage multiple engines without creating tightly coupled complexity.
Solution Approach 2:
The system dynamically adjusts parameters such as weights and combination rules based on engine performance characteristics and evaluation feedback. By changing these parameters, the system optimizes the combination of multiple engine outputs to achieve better reliability while managing complexity through adaptive parameter tuning rather than fixed complex structures.
3Measurement precision
If a combinatorial approach with weighting schemes is implemented, then quantitative evaluation and summary quality improve, but the computational processing requirements increase
Solution Approach 1:
The patent applies partial weighting schemes where not all engine outputs are weighted equally or processed with maximum computational effort. Instead, the system applies differential weighting based on engine performance, focusing computational resources on higher-quality engine outputs while still incorporating lower-quality outputs with reduced weight, achieving good evaluation precision without exhaustive computational processing of all possible combinations.
Data Source
AI summary
A combinatorial summarizer includes a plurality of summarization engines, a processor in selective communication with each summarization engine, and computer readable instructions executable by the processor and embodied on a tangible, non-transitory computer readable medium. Each summarization engine is to select a respective plurality of sentences, and generate a relative rank and an associated weight for each sentence of the respective plurality of sentences. The computer readable instructions include instructions to determine a combined weight for each sentence of each respective plurality of sentences. The combined weight is based upon the respective associated weight and a respective relative human rank for each sentence in a set of sentences, including all sentences of each respective plurality of sentences. The computer readable instructions further include instructions to determine a total weight for each summarization engine based, respectively, upon the combined weights for each sentence of each respective plurality of sentences.