Metric Relevance Scoring Using NLP for Root-Cause Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud providers face challenges in identifying a smaller set of relevant metrics from thousands of metrics due to varying customer-defined names for the same or similar metrics across different workspaces, leading to inefficient root-cause analysis.
Innovation Solution
A system that converts customer-defined metric names into mathematical representations using word embedding featurization vectors, compares new metric names to existing representations, and assigns relevance scores to filter down to a smaller set of relevant metrics for efficient querying and ingestion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If all available metrics are queried and ingested for root-cause analysis, then the completeness of analysis is improved, but the time required and computational resources increase unreasonably
Solution Approach 1:
The system extracts only the relevant subset of metrics needed for root-cause analysis by comparing metric names using natural language processing. Instead of ingesting all thousands of available metrics, the system identifies and extracts only those metrics whose names semantically match the incident type, thereby reducing analysis time while maintaining analytical completeness.
Solution Approach 2:
The system performs partial action by querying only a subset of metrics rather than all metrics. The NLP-based relevance scoring enables the system to take a partial approach (querying only relevant metrics) while still achieving effective root-cause analysis, thus avoiding the excessive time and resource costs of complete metric ingestion.
2Productivity
If a smaller set of relevant metrics is identified, then the querying and analysis time is reduced, but the difficulty of identifying relevant metrics increases due to varying customer-defined names
Solution Approach 1:
The system introduces natural language processing as an intermediary mechanism to bridge the gap between varying customer-defined metric names and the need to identify relevant metrics. The NLP model acts as a mediator that translates different naming conventions into comparable semantic representations, enabling automatic relevance determination without manual intervention.
Solution Approach 2:
The system changes the parameter of metric comparison from exact string matching to semantic similarity based on NLP embeddings. By transforming metric names into vector representations and comparing them using cosine similarity, the system can identify relevant metrics despite variations in naming conventions, thus reducing identification difficulty while maintaining speed.
3Measurement precision
If manual labeling of metrics is performed to determine relevance, then the accuracy of metric selection is improved, but the scalability to millions of workspaces is reduced
Solution Approach 1:
The system enables self-service by allowing metric relevance determination to occur automatically without manual labeling. The NLP-based system autonomously compares metric names and assigns relevance scores, enabling the solution to scale to millions of workspaces without requiring manual intervention for each workspace while maintaining accurate metric selection.
Solution Approach 2:
The system uses copying by training an NLP model on labeled data from one workspace and then applying the learned semantic comparison capabilities to automatically score metrics in other workspaces. This allows the accurate metric selection approach to be replicated and scaled across millions of workspaces without repeating the manual labeling process.
Data Source
AI summary
Disclosed herein is a system for determining scores that are usable to filter a larger set of metrics (e.g., thousands of metrics) down to a smaller set of relevant metrics (e.g., hundreds of metrics) that can be more efficiently queried and ingested for root-cause analysis of an incident. During a training stage, the system analyzes known incidents and converts the names of the metrics, as described via customer-defined words, into mathematical representations (e.g., word embedding featurization vectors). When a new metric with a new name is received for a new incident, the system implements an incident inference stage during which the new name is converted into a new mathematical representation. The system compares the new mathematical representation to the mathematical representations to identify a similar mathematical representation. The system retrieves the score for the metric associated with the similar mathematical representation and assigns the retrieved score to the new metric.