LLM Pull Request Review Using Historical Comment Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing code review processes in cloud platforms are prone to inefficiencies, errors, and security concerns due to manual reviews susceptible to human error and biases, and off-the-shelf LLMs fail to account for up-to-date code review practices, leading to significant processing and memory overhead.
Innovation Solution
Implementing a system that uses a large language model (LLM) for automated code review, coupled with a retrieval-augmented generation (RAG) technique and a vector space to track historical code changes, allowing for prompt modification to include up-to-date code review comments, thereby reducing the need for LLM retraining and optimizing resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If manual code review is used, then code review accuracy can be maintained, but processing time and human resource consumption increase
Solution Approach 1:
The patent introduces an LLM-based automated code review system as an intermediary between code submission and final review. This automated system performs initial analysis, identifies issues, and generates review comments, thereby increasing processing speed while maintaining reliability through the mediator's specialized capabilities in code analysis
Solution Approach 2:
The patent replaces manual mechanical code review processes with an automated LLM-based system. The LLM analyzes code changes, identifies potential issues, and generates review comments automatically, substituting human manual inspection with an intelligent automated mechanism that operates continuously without fatigue
2Productivity
If off-the-shelf LLMs are used for code review, then processing speed increases, but code review accuracy decreases due to lack of up-to-date knowledge
Solution Approach 1:
The patent implements preliminary action by fine-tuning the LLM on historical code review data and organizational coding standards before it is deployed for actual code review tasks. This pre-training on domain-specific data ensures the LLM has up-to-date knowledge of best practices and patterns relevant to the organization's codebase
Solution Approach 2:
The patent incorporates feedback mechanisms where the LLM learns from historical code review outcomes and organizational coding standards. The system continuously refines its understanding by analyzing past review decisions and code changes, improving its accuracy over time through feedback loops that align its behavior with organizational requirements
3Reliability
If LLM retraining is performed to update code review knowledge, then code review accuracy improves, but processing time and computational resources increase
Solution Approach 1:
The patent performs preliminary fine-tuning of the LLM on historical code review data during the system setup phase. This one-time preliminary action prepares the model with organizational-specific knowledge, eliminating the need for continuous retraining and reducing ongoing computational resource consumption
Solution Approach 2:
The patent changes the approach from full LLM retraining to fine-tuning on domain-specific data. By adjusting the training parameters to focus only on organizational code review patterns rather than complete model retraining, the system achieves updated knowledge with minimal computational overhead and time investment
Data Source
AI summary
Methods, systems, apparatuses, devices, and computer program products are described. A processing device may support a large language model (LLM) for automatically improving pull requests to a codebase. To use the LLM, the processing device may create and maintain a vector space tracking information relating to historical pull requests to the codebase. The processing device may receive a new pull request indicating a change to code in the codebase and may determine, from the vector space, a vector corresponding to a code chunk affected by the pull request. The processing device may send, as an input to the LLM, a prompt including the code chunk affected by the pull request and one or more comments from a set of historical comments relating to the code chunk and indicated by the determined vector. The processing device may modify the pull request based on the one or more comments.