Heterogeneous Context Evaluation for Trustworthy AI-Generated Code
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing generative artificial intelligence models for software code generation lack trustworthiness and explainability, particularly in programming languages without extensive semantic checkers, leading to vulnerabilities like SQL injection.
Innovation Solution
A pipeline using heterogenous context evaluation, involving multiple machine learning models and knowledge bases, to assess the semantic correctness of generated code by comparing it against different programming languages, employing semantic search and natural language processing to provide a trustworthiness score.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If existing generative AI models are used for code generation, then code generation capability is provided, but trustworthiness and explainability are insufficient
Solution Approach 1:
The evaluation system is segmented into multiple independent components: code-to-natural-language explanation model, text summarization model, and semantic correctness comparison model. Each component performs a specific function in the evaluation pipeline, allowing the system to assess trustworthiness through structured analysis rather than a monolithic approach.
Solution Approach 2:
The patent introduces an intermediary natural language explanation layer between the generated code and the evaluation process. The code is first translated to natural language explanations, then summarized, and finally compared semantically with the prompt. This intermediary representation enables thorough evaluation without directly analyzing complex code structures.
2Measurement precision
If code is evaluated using semantic comparison, then accuracy of correctness assessment is improved, but processing time increases
Solution Approach 1:
The system performs preliminary actions by first generating natural language explanations from the code and summarizing them before the final semantic comparison. This pre-processing transforms complex code into condensed natural language representations, making the subsequent semantic comparison more efficient and accurate.
Solution Approach 2:
The evaluation process changes the representation parameter of the code from raw code syntax to natural language explanations. This parameter transformation allows the system to leverage linguistic understanding for semantic comparison, improving accuracy while the summarization step controls the dimensionality of the comparison space.
Data Source
AI summary
A method, computer system, and a computer program product for code evaluation are provided. Knowledge bases in first and/or second programming languages are searched to find code and associated natural language explanations that correspond to code generated by a machine learning model. Additionally and/or alternatively, the code generated by the machine learning model undergoes code-to-natural language translation by another machine learning model and text summarization via an additional machine learning model. Machine learning semantic comparison of (A) an original prompt submitted to the first machine learning model to generate the code and (B) a natural language explanation obtained via one or more of the above approaches is performed. The machine learning semantic comparison generates a first semantic correctness score which is presented.