Alert-Based Learning for Multidimensional KPI Condition Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large data warehouses, or 'big data' platforms, face challenges in detecting and alerting users to conditions within complex graph cube networks due to the vast number of records and interdependence between data elements, making timely evaluation and decision-making unfeasible with conventional data processing techniques.
Innovation Solution
Implementing alert-based learning systems that capture and analyze multidimensional data, automatically generate alerts, and execute instructions to correct conditions, while storing historical alert events for learning and decision-making, using machine learning to identify recurrent patterns and root causes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional data processing techniques are used to evaluate data in a big data platform, then the system can process data, but it cannot timely detect and alert conditions in the graph cube network due to the vast number of records and interdependence between data elements
Solution Approach 1:
The patent segments the evaluation process into distinct phases: data collection from multiple sources, data preparation and processing, model training, and deployment. This segmentation allows each phase to be optimized independently, with parallel processing of data collection and preparation, enabling timely condition detection in complex graph cube networks
Solution Approach 2:
The system performs preliminary actions by pre-processing data during the data preparation phase and pre-training models before deployment. Historical data is processed and models are trained in advance, so when conditions need to be detected in the graph cube network, the system can immediately apply pre-trained models to newly collected data, significantly reducing evaluation time
2Measurement precision
If the system monitors all data elements in the graph cube network, then it can detect conditions accurately, but the complexity of processing interdependent data elements increases
Solution Approach 1:
The patent extracts only the necessary features and data elements needed for specific condition detection tasks from the vast graph cube network. Rather than processing all data elements, the system identifies and extracts relevant features through the machine learning pipeline, reducing processing complexity while maintaining detection precision for targeted conditions
Solution Approach 2:
The system introduces machine learning models as intermediaries between raw data collection and condition detection. These models serve as mediators that automatically learn and represent the complex interdependencies between data elements, transforming the processing task from manually managing all interdependencies to training models that capture these relationships, thereby reducing overall system complexity
3Speed
If the system processes multidimensional data in real-time, then it can provide near real-time alerts, but the computational resources and processing time required increase
Solution Approach 1:
The system implements periodic action by collecting data continuously but processing and evaluating it at defined intervals through batch processing. Data is collected from multiple sources continuously, then processed in batches through the machine learning pipeline at regular intervals, providing periodic updates and alerts. This approach enables near real-time responsiveness while controlling computational resource consumption by avoiding continuous full-system processing
Data Source
AI summary
Systems and techniques for alert-based learning for multi-dimensional data processing are described herein. Alert data may be received from a big data platform. An alert definition may be generated that includes a set of key performance indicators (KPIs) for an alert based on a determination that an alert definition does not exist for the alert. A calculation configuration may be created for the alert. Historical data from the big data platform may be evaluated with the calculation configuration to calculate the KPIs. An alert condition may be established for the alert based on the KPIs. The alert condition may be stored in the big data platform.