Code Generation Prompt Repair Using Execution Error Feedback
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Crafting a prompt suitable for code generation is challenging due to the lack of transparency in how generative AI models generate code, leading to labor-intensive and iterative updates to address issues in the generated code.
Innovation Solution
A framework that automates the repair of code generation prompts by executing generated code samples, clustering execution errors, and generating a repair prompt to update the prompt based on identified errors, ensuring stable and error-free code generation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If a user manually crafts and iteratively updates code generation prompts to address issues in generated code, then code quality can be improved, but the process becomes labor-intensive and time-consuming
Solution Approach 1:
The system enables self-service by automatically analyzing execution errors and generating updated prompts without requiring manual user intervention. The framework executes generated code, clusters execution errors, and autonomously creates refined prompts, allowing the system to improve its own prompt quality through self-diagnosis and self-correction mechanisms
Solution Approach 2:
The system implements feedback loops by executing generated code samples and using execution errors as feedback to improve prompt quality. The framework collects execution results, clusters errors to identify patterns, and uses this feedback information to generate updated prompts that address the identified issues, creating a continuous improvement cycle
2Reliability
If a user manually updates code generation prompts to address execution errors, then code reliability can be improved, but the complexity of the process increases due to lack of transparency
Solution Approach 1:
The system introduces an intermediary automated framework that mediates between code generation and manual prompt updates. This intermediary layer executes code samples, clusters execution errors, and generates updated prompts, simplifying the complex process of prompt refinement while improving code reliability through systematic error analysis and correction
Solution Approach 2:
The system replaces the mechanical manual process of prompt updating with an automated computational system. Instead of manually analyzing errors and rewriting prompts, the framework uses automated code execution, error clustering algorithms, and prompt generation mechanisms to substitute the manual mechanical process with an automated system
3Measurement precision
If multiple code samples are generated and executed to identify execution errors, then prompt repair accuracy can be improved, but computational resources and time are consumed
Solution Approach 1:
The system applies partial action by generating and executing a limited number of code samples (e.g., 5-10 samples) rather than exhaustively testing all possible code variations. This partial sampling approach provides sufficient error identification accuracy while consuming reasonable computational resources, avoiding the need to test every possible code generation outcome
Data Source
AI summary
Systems and methods include input of a code generation prompt to a text generation model, reception of code from the text generation model in response to the input code generation prompt, execution of the received code, determination of execution information associated with the execution of the received code, input of a repair prompt, the code generation prompt and the execution information to the text generation model, and reception of an updated code generation prompt from the text generation model in response to the input repair prompt, code generation prompt and execution information.