API-Based LLM Code Evaluation for Accuracy and Robustness

The patent introduces an evaluation system as an intermediary between the LLM code generator and the final code execution. This evaluation system acts as a mediator that assesses generated code across multiple dimensions (accuracy, robustness, consistency) before deployment, thereby maintaining fast generation speeds while ensuring code quality reliability through systematic evaluation.

The patent implements a feedback mechanism where the evaluation results of generated code are fed back into the system. By evaluating code quality across multiple runs and dimensions, the system generates feedback that can be used to improve future code generation, thus resolving the contradiction between speed and reliability through continuous improvement based on evaluation data.

The patent segments the code quality evaluation into three distinct dimensions: accuracy evaluation (whether code runs and produces correct output), robustness evaluation (performance across different difficulty levels and prompt variations), and consistency evaluation (results across multiple runs). This segmentation allows systematic evaluation of code quality without overwhelming complexity, as each dimension can be assessed independently with specific criteria.

The patent changes evaluation parameters by establishing specific measurable criteria for each evaluation dimension. Instead of vague quality assessments, the system uses concrete parameters such as execution success, output correctness, performance across difficulty levels, and result consistency across multiple runs. This parameterization makes the evaluation process systematic and manageable.

The patent applies partial action by evaluating code across multiple dimensions but using a structured approach that doesn't require exhaustive testing in all areas simultaneously. The evaluation system assesses accuracy, robustness, and consistency with specific targeted tests for each dimension, providing comprehensive evaluation without requiring excessive time by focusing on key evaluation points for each dimension.

The patent implements periodic action by evaluating code consistency across multiple runs (at least three runs) rather than a single exhaustive evaluation. This periodic approach allows the system to assess code reliability through repeated measurements while managing time efficiently by establishing a minimum number of runs rather than continuous evaluation.