AI Network Troubleshooting with Expert Feedback Routing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large language models (LLMs) face challenges in network monitoring and control due to delays in providing answers and potential mistakes, which can frustrate users and hinder effective network troubleshooting.
Innovation Solution
An AI-based network troubleshooting system that utilizes a large language model-based agent to identify tasks, determines when it cannot complete them, and requests assistance from subject matter experts to enhance task completion, tracking actions and metrics to improve success rates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a large language model-based agent is used to perform network troubleshooting tasks, then the system can handle complex troubleshooting scenarios, but the response time increases significantly
Solution Approach 1:
The system segments the troubleshooting task into two parts: complex analysis is handled by the LLM-based agent, while simple routine tasks are handled by a traditional agent. This segmentation allows the system to maintain fast response times for routine tasks while using the more versatile LLM agent only when necessary for complex troubleshooting scenarios.
Solution Approach 2:
A classification model acts as an intermediary between the user's troubleshooting query and the LLM-based agent. The classification model first analyzes the query and determines whether to route it to the traditional agent or the LLM-based agent, preventing unnecessary delays by avoiding LLM processing for simple tasks that can be handled more quickly by traditional agents.
2Adaptability or versatility
If a large language model-based agent is used for network monitoring, then the system can provide intelligent insights, but the model may make mistakes that are not apparent to users
Solution Approach 1:
The system incorporates feedback mechanisms where users can provide feedback on the LLM agent's responses. This feedback is used to continuously improve the model's performance through reinforcement learning from human feedback (RLHF), allowing the system to learn from user corrections and improve accuracy over time without requiring manual programming of correct responses.
Solution Approach 2:
The system uses a classification model to perform a preliminary action of filtering and routing tasks. By partially handling the task classification before it reaches the LLM agent, the system reduces the burden on the LLM and ensures that only appropriately complex tasks are submitted, improving overall reliability while maintaining intelligent analysis capability.
3Adaptability or versatility
If multiple steps are used to answer troubleshooting questions, then the system can provide comprehensive answers, but the overall process becomes slower
Solution Approach 1:
The system dynamically adjusts the number of steps based on the complexity of the troubleshooting task. Simple tasks are answered in a single step by the traditional agent, while complex tasks trigger multiple steps involving the LLM-based agent. This dynamic approach allows the system to maintain high productivity for simple tasks while providing comprehensive answers for complex scenarios.
Solution Approach 2:
The classification model performs a preliminary action of analyzing and categorizing the troubleshooting query before it is processed by the LLM-based agent. This preliminary classification step enables the system to prepare appropriate responses in advance and route tasks efficiently, reducing the overall number of steps needed and improving answer generation speed while maintaining comprehensiveness.
Data Source
AI summary
In one implementation, a device uses a large language model-based agent to identify a task to correct an issue in a network. The device makes a determination that the large language model-based agent cannot complete the task. The device identifies, based on the determination, a subject matter expert to help complete the task. The device sends a request to the subject matter expert to complete the task.