Code Generation Agent Selection With Multi-Stage Re-Ranking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing software engineering AI agents vary significantly in accuracy and efficiency when generating code to resolve software issues due to diverse agent capabilities and complex codebases, leading to inconsistent and suboptimal solutions.
Innovation Solution
A multi-stage rating and re-ranking pipeline selects the most suitable SWE agent for each context using a meta-policy that maximizes cumulative reward, combined with a feedback mechanism involving a multi-agent reviewing system for optimizing code snippets, including fault localization and code modification components.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple diverse SWE agents are used to generate code solutions, then the diversity of solutions increases, but the accuracy and consistency of code generation deteriorates
Solution Approach 1:
The system segments the code generation task into multiple stages: initial solution generation by diverse agents, followed by separate review and rating stages. Different agents specialize in different aspects (generation vs. review), allowing diversity in solution exploration while ensuring quality through dedicated review processes.
Solution Approach 2:
The system implements a feedback mechanism where generated code solutions are reviewed by other agents who provide ratings and critiques. This feedback loop allows the system to learn from diverse perspectives while maintaining quality standards through iterative improvement based on agent evaluations.
2Reliability
If a single SWE agent is used to generate code solutions, then the accuracy and consistency of code generation improves, but the adaptability to handle diverse software issues deteriorates
Solution Approach 1:
The system creates a universal framework where multiple agents with different specializations work together. Each agent can be optimized for specific task types (e.g., bug fixing, feature addition, code review), yet the overall system handles diverse software engineering issues through the coordinated effort of these specialized agents.
Solution Approach 2:
The system merges the capabilities of multiple specialized agents into a unified code generation system. By combining agents with different strengths (generation, review, rating, optimization), the system achieves both high accuracy through specialized review and high adaptability through diverse agent perspectives.
3Measurement precision
If extensive codebase navigation and complex function interaction analysis are performed, then the accuracy of bug detection improves, but the time and computational resources required deteriorates
Solution Approach 1:
The system performs preliminary code analysis and navigation before actual bug detection and fixing. Agents pre-process the codebase to identify relevant functions, relationships, and potential issue areas, so that when bug detection is needed, the analysis can be focused and efficient rather than starting from scratch.
Solution Approach 2:
The system applies partial analysis to most code areas and excessive (detailed) analysis only where needed based on initial screening. Agents use heuristics to identify high-risk areas requiring thorough analysis, while other areas receive lighter scrutiny, balancing accuracy with time efficiency.
4Reliability
If multiple review agents are used to evaluate code solutions, then the quality and reliability of code patches improves, but the device complexity and computational overhead deteriorates
Solution Approach 1:
The review process is segmented into distinct phases (syntax checking, logic verification, style review, security analysis) performed by different specialized agents. This segmentation allows each agent to focus on specific aspects, improving quality while managing complexity through clear division of labor.
Solution Approach 2:
The system introduces intermediary components that coordinate between multiple review agents, managing their interactions and synthesizing their feedback. This intermediary layer handles the complexity of multi-agent coordination, allowing individual agents to remain relatively simple while achieving comprehensive review through their combined effort.
Data Source
AI summary
Embodiments described herein provide a multi-stage rating and re-ranking pipeline for selecting SWE agents for an input issue description. Specifically, a meta-policy may be selected among available agent policies corresponding to a pool of available SWE agents which maximizes the cumulative reward along the trajectory of states (such as status of a file) and actions taken at a series of time steps, and a context of relevant repository information and issue descriptions. By dynamically choosing the most suitable agent policy for each context, the selection pipeline maximizes the expected cumulative reward across all possible contexts. In this way, software issue resolve rate is improved.