Chart De-Rendering Evaluation Using Index-Aware Accuracy Metrics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for evaluating the performance of chart de-rendering models, such as RNSS, fail to accurately assess the characteristics of charts due to their inability to consider data indices, leading to incorrect evaluations of chart accuracy.
Innovation Solution
A system and method that includes a performance evaluation model to objectively evaluate chart de-rendering models by calculating line accuracy, axis accuracy, meta accuracy, and length accuracy, using parameters like character error rate (CER) to quantify the model's performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If RNSS (Relative Number Set Similarity) is used to evaluate chart de-rendering accuracy, then the evaluation process is simple and fast, but the evaluation accuracy is insufficient because it does not consider data indices
Solution Approach 1:
The patent segments the evaluation process into multiple independent accuracy metrics: line accuracy (for trend evaluation), axis accuracy (for range evaluation), and data point accuracy (for value evaluation). Each metric focuses on a specific aspect of chart de-rendering performance, allowing comprehensive evaluation while maintaining computational efficiency through specialized sub-routines for each metric type.
Solution Approach 2:
The patent applies different evaluation criteria to different parts of the chart: line accuracy uses Euclidean distance for continuous line segments, axis accuracy uses overlap ratios for axis ranges, and data point accuracy uses index-matched distance calculations for discrete points. This localized approach ensures each chart element is evaluated by the most appropriate metric, improving overall evaluation accuracy.
2Device complexity
If only distance between data points is considered for evaluation, then the calculation is straightforward, but the evaluation cannot distinguish between correct and incorrect chart predictions
Solution Approach 1:
The patent performs preliminary actions by first matching data points based on their indices before calculating distances. This pre-matching step ensures that each predicted data point is compared against the corresponding ground truth point with the same index, preventing incorrect comparisons between unrelated points and ensuring evaluation reliability from the outset.
Solution Approach 2:
The patent introduces data indices as an intermediary element that mediates the comparison between predicted and ground truth data points. By using indices as a matching key, the system ensures that evaluations are performed on corresponding points rather than arbitrary point pairs, significantly improving evaluation reliability while adding minimal computational complexity.
3Productivity
If RNSS evaluates all data points collectively, then the overall accuracy score is obtained quickly, but individual prediction errors cannot be identified
Solution Approach 1:
The patent segments the overall evaluation into three distinct accuracy components: line accuracy for trend evaluation, axis accuracy for range evaluation, and data point accuracy for value evaluation. Each component is calculated independently and can be reported separately, preserving detailed error information while maintaining evaluation efficiency through specialized calculation methods for each segment type.
Solution Approach 2:
The patent adds dimensional granularity to the evaluation by providing accuracy metrics at multiple levels: overall chart accuracy, individual line accuracy, individual axis accuracy, and individual data point accuracy. This multi-dimensional approach allows users to drill down from aggregate metrics to specific error locations, preserving detailed information while maintaining efficient top-level evaluation.
Data Source
AI summary
Provided is a system for implementing model for evaluating accuracy of a chart de-rendering model. The system includes one processor, and one memory storing instructions for the processor. The processor inputs a chart stored in test set into artificial intelligence (AI) model to output data format in which information of the chart is predicted, and inputs the data format into performance evaluation model to output performance evaluation result for the AI model by comparing information of the data format with ground truth (GT), which is stored in the test set. The performance evaluation result includes line accuracy indicating degree of proximity between the chart of the data format and the chart of the GT, and axis accuracy indicating degree of overlap between range in which the chart of the data format is distributed and range in which the chart of the GT is distributed on one axis of the chart.