Logical Inference Graph for Generative Model Evaluation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The evaluation of generative large models is challenging due to non-unique and non-fixed text generation results, making traditional evaluation methods inadequate.
Innovation Solution
Disassemble text generation results into logical inference units, generate a logical inference graph to characterize relationships among these units, and evaluate the correctness of the logical inference using the graph.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional evaluation methods are used to evaluate generative large models, then the evaluation process is simple, but the evaluation accuracy is insufficient due to non-unique and non-fixed text generation results
Solution Approach 1:
The patent segments the text generation result into multiple logical inference units, where each unit represents a discrete logical step (premise, intermediate conclusion, or final conclusion). This segmentation allows the evaluation system to assess each logical unit independently and then aggregate the results, thereby improving evaluation accuracy while managing system complexity through modular processing
Solution Approach 2:
The patent introduces a logical inference graph as an intermediary structure that characterizes the relationships among logical inference units. This graph serves as a mediator between the generated text and the evaluation criteria, enabling systematic assessment of logical coherence, completeness, and correctness without requiring direct complex comparison of entire text generations
2Productivity
If manual evaluation methods are used to assess text generation results, then evaluation accuracy can be maintained, but evaluation efficiency is low and time-consuming
Solution Approach 1:
By dividing the text generation result into discrete logical inference units, the system enables automated processing of each unit through predefined evaluation rules. This segmentation transforms the evaluation task from a complex manual review process into a series of manageable automated checks, significantly improving efficiency while maintaining accuracy through systematic assessment of each logical component
Solution Approach 2:
The patent replaces manual mechanical evaluation with an automated computational system that uses logical inference graphs and predefined evaluation criteria. This substitution eliminates the time-consuming nature of manual review while maintaining or improving accuracy through consistent application of logical rules and automated reasoning verification
3Measurement precision
If character string matching is used to evaluate text generation results, then the evaluation process is straightforward, but it cannot accurately assess logical inference correctness due to non-fixed generation results
Solution Approach 1:
The patent segments text generation results into logical inference units that explicitly represent premises, intermediate conclusions, and final conclusions. This segmentation transforms the evaluation task from simple character matching to structured logical relationship verification, enabling accurate assessment of logical inference correctness while maintaining operational simplicity through standardized unit formats and evaluation rules
Solution Approach 2:
The logical inference graph serves as an intermediary that translates unstructured text generation results into a structured representation of logical relationships. This graph enables the evaluation system to assess logical inference correctness by examining the relationships between logical units rather than performing complex text comparison, thereby improving accuracy while keeping the evaluation process manageable
Data Source
Figure 1
Figure 2
Figure 3~4
AI summary
The present disclosure provides a method and apparatus for processing a model generation result, an electronic device and a storage medium, and relates to the field of artificial intelligence technologies, such as machine learning technologies, natural language processing technologies. An implementation includes: disassembling a text generation result of a generative large model to obtain a plurality of result logic units; wherein each result logic unit includes a segment in the text generation result; each segment is capable of independently identifying one premise or conclusion in a logical inference relationship of the text generation result; and the text generation result is a response result generated by the generative large model based on text input information; generating a logical inference graph capable of characterizing a logical inference relationship among the plurality of result logic units based on the plurality of result logic units; and determining whether logical inference of generation of the text generation result by the generative large model is correct or not based on the logical inference graph. With the technology according to the present disclosure, whether logical inference of the generative large model is correct or not can be efficiently and accurately determined.