AI Code Detection Using Masked Line Perturbation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge of distinguishing between human-written and AI-generated computer code has become increasingly difficult due to the advancements in large language models, leading to concerns about code quality, security, plagiarism, and ethical implications, with existing detection methods being inadequate, especially for codes generated by models like GPT-4.
Innovation Solution
A method involving masking lines of code to create perturbed codes, generating missing code using an AI code generator model, predicting probabilities with a surrogate model, and comparing these against a detection threshold to identify AI-generated code, utilizing models such as Incoder-6B, PyCodeGPT-110M, and CodeParrot-1.5B, and employing techniques like n-gram divergence and model output probability comparison.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If large pre-trained language models are used to generate computer code, then code fluency and diversity are improved, but the ability to distinguish machine-generated from human-written code deteriorates
Solution Approach 1:
The patent applies preliminary action by masking certain lines of code before generation to create perturbed codes. This preprocessing step modifies the input to the AI code generator model, creating variations that preserve the original code's semantic meaning while introducing detectable patterns. The masking operation occurs before the generation process, allowing the detection system to capture subtle characteristics of AI-generated code that would otherwise be indistinguishable from human-written code.
Solution Approach 2:
The patent introduces a surrogate model as an intermediary between the AI code generator model and the detection process. This surrogate model is trained to predict whether code is AI-generated based on the perturbed codes and machine-filled codes. The intermediary model translates the complex patterns in generated code into probability predictions, making the detection process more effective without requiring direct analysis of the original generation process.
2Productivity
If AI code generator models generate missing code from perturbed codes, then code completion capability is improved, but detection accuracy deteriorates
Solution Approach 1:
The patent applies segmentation by dividing the code into visible and masked portions. The AI code generator model only generates the masked portions based on the visible context, creating machine-filled codes that complete the perturbed code. This segmentation allows the detection system to analyze both the original perturbed code and the generated completion, comparing their characteristics to identify AI-generated patterns while maintaining code completion functionality.
Solution Approach 2:
The patent changes parameters by creating multiple versions of perturbed codes with different masking patterns and applying the detection process to each version. By varying the masking parameters and generating multiple machine-filled codes, the system accumulates probability predictions that improve detection accuracy. The surrogate model processes these varied inputs to produce more reliable detection results.
Data Source
AI summary
Systems and methods for detecting artificial intelligence (AI) generated computer code. Lines of code can be masked from a candidate code to obtain perturbed codes. Missing code can be generated from the perturbed codes by employing an AI code generator model to obtain machine-filled codes. Probabilities of the candidate code probability and the machine-filled codes as AI-generated can be predicted by employing a surrogate model. The candidate code can be distinguished as AI-generated by comparing the probabilities against a detection threshold to obtain detection results.