Chart De-Rendering Evaluation Using Multi-Metric AI Accuracy Checks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for evaluating the performance of chart de-rendering models, such as RNSS, fail to accurately consider data indices and chart characteristics, leading to incorrect accuracy assessments.
Innovation Solution
A system and method that includes a performance evaluation model to assess chart de-rendering models by calculating line accuracy, axis accuracy, meta accuracy, and length accuracy, using parameters like character error rate (CER) to objectively evaluate the AI model's performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If RNSS is used to evaluate chart de-rendering accuracy, then calculation simplicity is improved, but measurement precision deteriorates
Solution Approach 1:
The evaluation method is segmented into multiple independent accuracy metrics: data point accuracy, data index accuracy, chart layout accuracy, and visual characteristic accuracy. Each metric evaluates a specific aspect of chart de-rendering performance, allowing comprehensive assessment without requiring complex integrated calculations.
Solution Approach 2:
The patent introduces multiple intermediary evaluation metrics that bridge the gap between simple distance calculation and comprehensive accuracy assessment. These intermediaries include data index matching, layout structure comparison, and visual characteristic verification, which collectively mediate between simplicity and precision.
2Productivity
If only data point distance is considered in evaluation, then calculation speed is improved, but measurement precision deteriorates
Solution Approach 1:
The evaluation process is divided into multiple sequential stages: first calculating data point distances for quick assessment, then progressively evaluating data indices, layout structures, and visual characteristics. This segmentation allows speed-precision tradeoff at different evaluation levels.
Solution Approach 2:
The patent implements partial evaluation by allowing users to stop at any stage of the multi-stage evaluation process. If data point distance provides sufficient insight, further evaluation of indices and layout can be skipped, achieving partial assessment with reduced computation while maintaining option for complete evaluation.
3Device complexity
If data indices are not considered in evaluation, then calculation simplicity is improved, but information completeness deteriorates
Solution Approach 1:
The evaluation framework segments chart information into distinct components: numerical data points, data indices (labels, titles, axis names), layout structures, and visual characteristics. Each segment is evaluated separately with appropriate metrics, ensuring no information type is lost while maintaining modular evaluation complexity.
4Device complexity
If visual characteristics are not considered in evaluation, then measurement simplicity is improved, but accuracy assessment completeness deteriorates
Solution Approach 1:
The patent segments visual characteristic evaluation into distinct measurable attributes: color accuracy, shape accuracy, size accuracy, and position accuracy. Each attribute is evaluated with specific metrics that compare rendered chart visuals against reference charts, ensuring comprehensive visual assessment without overwhelming complexity.
Data Source
Figure 1A~1B
Figure 1C~1D
Figure 2~3
AI summary
A system, method, and program for evaluating performance of an artificial intelligence model capable of extracting information of a chart by de-rendering the chart are disclosed. The system includes a memory including an AI model and a performance evaluation model, and a processor that executes or trains the AI model and executes the performance evaluation model, wherein the AI model outputs at least one data format including meta information or data information included in the image of a chart by decoding an image of the chart stored in a test set, and the performance evaluation model outputs a result of evaluating the performance of the AI model. The performance evaluation model may output a performance evaluation result by comparing information included in the data format with ground truth (GT) including information about the chart stored in the test set, and more specifically, the performance evaluation result may include a line accuracy, an axis accuracy, a meta accuracy, and a length accuracy.