AIOps Capability Measurement via Simulation and Accuracy Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current AIOps systems lack an effective method to measure their predictive capability, leading to potential issues in computing environments such as performance degradation and resource shortages, as existing approaches do not adequately assess the applicability of machine learning models based on practical software and hardware considerations.
Innovation Solution
The system evaluates the capability of AIOps systems by running simulations using historical data from similar systems, determining accuracy scores, and generating an enablement score that considers AI models, data types, and computing constraints, providing a measure of predictive accuracy and suggesting improvements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If AIOps systems are deployed to predict performance issues using AI/ML models, then predictive capability is improved, but measurement precision of the system's actual capability deteriorates due to lack of effective evaluation methods
Solution Approach 1:
The patent creates a virtual copy of the AIOps system that replicates its behavior and characteristics. This virtual instance allows for controlled experimentation and evaluation without affecting the actual production system, enabling precise measurement of predictive capability through repeated testing with different datasets and scenarios.
Solution Approach 2:
The patent introduces an intermediary evaluation framework that acts as a mediator between the AIOps system and the metrics used to assess its capability. This framework standardizes the measurement process, providing consistent and comparable evaluations across different systems and time periods, thereby improving measurement precision.
2Measurement precision
If simulations are run using historical data from similar systems, then measurement precision is improved, but device complexity increases due to additional simulation infrastructure
Solution Approach 1:
The patent designs a universal simulation framework that can evaluate multiple AIOps systems using the same infrastructure and methodology. This multi-functional approach allows the simulation environment to serve various evaluation purposes without requiring separate complex setups for each system, thereby reducing overall device complexity while maintaining measurement precision.
Solution Approach 2:
The patent utilizes parameter changes in the simulation environment, such as varying historical datasets, performance thresholds, and system configurations, to comprehensively evaluate AIOps capability without building multiple physical test environments. By changing parameters rather than physical infrastructure, measurement precision is improved while device complexity is controlled.
3Measurement precision
If accuracy scores are determined through multiple simulations, then measurement precision is improved, but loss of time increases due to extensive simulation processes
Solution Approach 1:
The patent performs preliminary actions by pre-processing historical data, pre-configuring simulation parameters, and pre-establishing evaluation criteria before running the actual simulations. This preparation work is done once and reused across multiple evaluation runs, reducing the time required for each simulation while maintaining high measurement precision through comprehensive pre-analysis.
Solution Approach 2:
The patent implements continuous evaluation where simulations run continuously or in overlapping batches rather than sequential discrete processes. This allows accuracy scores to be updated incrementally as new data becomes available, maintaining high measurement precision without requiring complete re-simulation, thereby reducing total time loss.
Data Source
AI summary
An aspect of the present disclosure facilitates measuring the capability of AIOps (Artificial Intelligence for IT operations) systems deployed in computing environments. In one embodiment, a first simulation of a target AIOps system is run using a first historical input set having a corresponding first actual output set of a first AIOps system different from the target AIOps system. A second simulation of a reference AIOps system is run using a second historical input set having a corresponding second actual output set of the same first AIOps system. A first and second accuracy scores are determined based on outputs of the first and second simulations and the corresponding first and second actual output sets. An enablement score representing a measure of the capability (in terms of accuracy of prediction) of the target AIOps system is generated based on the first accuracy score and the second accuracy score.