Lossless Summarization Scoring for Accurate and Concise ML Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for training machine learning systems to generate text summaries are inefficient, often resulting in summaries that are similar to human-generated ones but not necessarily more useful or of higher quality, and obtaining human-generated summaries is burdensome and subjective.
Innovation Solution
A method involving generating a text summary using a machine learning system, evaluating its correctness against a question set with assigned weights, calculating a score based on correctness and brevity, and adjusting the system's training using this score to improve the quality and brevity of the summaries.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If machine learning systems are trained using human-generated text summaries for evaluation, then the system can be trained to generate summaries similar to human-generated ones, but the process becomes burdensome and subjective, and the obtained text summaries may vary greatly in writing styles
Solution Approach 1:
The patent creates synthetic human-like summaries by copying the structure and style of actual human summaries while using automated generation. These synthetic summaries are then used for training and evaluation, replacing the need for actual human-generated summaries and eliminating the burdensome collection process while maintaining evaluation accuracy.
Solution Approach 2:
The system uses itself to generate the evaluation data. The machine learning system generates summaries, which are then automatically evaluated against synthetic human-like summaries generated by the same or related systems, eliminating the need for external human annotators and simplifying the training process.
2Adaptability or versatility
If machine learning systems are trained using human-generated text summaries, then the system can learn from human writing styles, but the obtained text summaries are highly subjective and vary greatly in writing styles found present in business communications
Solution Approach 1:
The patent transforms the evaluation parameters from subjective human judgment to objective automated scoring. By generating synthetic summaries with controlled characteristics and using automated evaluation metrics, the system achieves consistent and reproducible evaluation results while still capturing the essence of human writing styles through the synthetic data generation process.
3Reliability
If the generated text summaries are evaluated to improve the machine learning system, then the system can be trained to generate better summaries, but the process is time-consuming and inefficient
Solution Approach 1:
The patent implements continuous automated evaluation where the system constantly generates summaries, evaluates them against synthetic references, and adjusts its parameters in an ongoing loop. This continuous feedback mechanism eliminates the need for periodic manual evaluation, maintaining high training efficiency while continuously improving summary quality and usefulness.
Data Source
AI summary
Systems, devices, and techniques are disclosed for lossless summarization. Data that including indications of whether a text summary generated by a machine learning system from an input document includes correct answers to questions about the input document may be received. Weights associated with the questions about the input document may be received. A first vector may be generated from the indications in the data. A second vector may be generated from the weights. A dot product of the first vector and the second vector may be determined. A ratio of a word count of the text summary to the word count of the input document may be determined. A score for the text summary based on the dot product and the ratio may be determined. The machine learning system may be adjusted based on the score.