Multi-Agent LLM Story Pointing for CI/CD Effort Estimation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing software development processes face challenges in accurately estimating the effort required for completing work items, leading to inaccuracies and potential delays in project timelines due to underestimations, particularly in unfamiliar domains.
Innovation Solution
Implementing multiple large language model (LLM) agents within a CI/CD system to interact with an LLM model for automated generation of story points, which are estimates of development effort, and adjusting the software development process based on these estimates, with roles such as product owner and software engineer focusing on different relationships.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If automated story pointing is implemented using LLM agents, then estimation accuracy is improved, but system complexity increases
Solution Approach 1:
The system segments the story pointing process into multiple specialized LLM agents, each responsible for specific aspects of estimation. The product owner agent handles requirement analysis and value assessment, while the software engineer agent focuses on technical complexity and implementation effort. This segmentation allows each agent to specialize in particular estimation dimensions, improving overall accuracy while maintaining manageable system complexity through clear role division.
2Loss of information
If multiple LLM agents are used for story pointing, then transparency and explainability are improved, but processing time increases
Solution Approach 1:
The system performs preliminary actions by having the product owner agent analyze requirements and assess business value before the software engineer agent evaluates technical complexity. This sequential preliminary analysis ensures that each agent has relevant context from previous steps, improving transparency and explainability of the estimation process. The structured interaction protocol pre-defines the flow of information between agents, reducing redundant processing and mitigating time increases.
3Productivity
If automated estimation is implemented, then productivity is improved, but reliability of estimates worsens due to potential underestimation in unfamiliar domains
Solution Approach 1:
The system implements feedback mechanisms where the software engineer agent reviews and adjusts the initial estimates generated by the product owner agent. The agents engage in iterative discussions, with the software engineer providing technical insights that may reveal complexities overlooked in initial requirement analysis. This feedback loop allows the system to self-correct potential underestimations, particularly in unfamiliar domains, thereby improving reliability while maintaining automated productivity.
Data Source
AI summary
A method in an illustrative embodiment comprises configuring a software development process to include at least first and second software-based agents for interacting with one another and with at least one large language model (LLM). The method further comprises assigning a first role in the software development process to the first software-based agent, assigning a second role in the software development process to the second software-based agent, initiating interactions between the first and second software-based agents and between each of the first and second software-based agents and the at least one LLM, determining estimates for completion of respective work items of the software development process based at least in part on the interactions, and adjusting one or more characteristics of the software development process based at least in part on the estimates. The software development process is illustratively part of a continuous integration/continuous deployment (CI/CD) system.