Unsupervised Learning for Distributed Systems Management
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current approaches to managing distributed systems, particularly those with microservices-based architectures, face challenges in accurately predicting system anomalies and determining root causes due to the complexity and high dimensionality of data, which burdens network and storage resources.
Innovation Solution
A computer-implemented method and system that utilizes unsupervised machine learning to cluster data, perform causal extraction, construct causal graphs, embed them into vector spaces, and train artificial neural networks to manage distributed computing environments, effectively reducing data processing burdens and improving anomaly detection and resource forecasting.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data collection and analysis are performed using current expert-based approaches, then system monitoring capability is maintained, but network and storage resources are burdened and analysis complexity increases
Solution Approach 1:
The system enables self-service by training machine learning models to automatically perform anomaly detection and root cause analysis without requiring expert intervention. The models learn from historical data and autonomously identify patterns, reducing both resource burden and analysis complexity while maintaining monitoring reliability.
Solution Approach 2:
The patent extracts the complex analysis task from the expert-based approach and separates it into distinct machine learning components: anomaly detection models and root cause analysis models. This extraction allows the system to handle complexity through automated algorithms rather than human expertise, reducing the burden on network and storage resources.
2Measurement precision
If comprehensive data collection is performed in distributed systems, then anomaly detection accuracy could be improved, but network and storage resource consumption increases
Solution Approach 1:
The system applies partial action by collecting and processing only the necessary data for anomaly detection rather than all possible data. The machine learning models are trained to identify anomalies using selective features and data samples, achieving sufficient detection accuracy without the resource consumption of comprehensive data collection.
Solution Approach 2:
The patent segments the data collection and processing into distributed components across multiple nodes. Each node collects local data and runs local models, with only essential information transmitted to central coordination. This segmentation reduces network traffic and storage requirements while maintaining detection accuracy through distributed intelligence.
3Loss of information
If expert-based data analysis is used, then system state understanding is achieved, but the burden on network and storage resources increases
Solution Approach 1:
The patent replaces the mechanical system of expert-based analysis with automated machine learning models. These models process system data using learned patterns and relationships, achieving comprehensive system state understanding without requiring human expert intervention. This substitution significantly reduces the network and storage resource burden associated with expert analysis workflows.
Data Source
AI summary
Data associated with performances of microservices functioning in a distributed computing environment is clustered by executing an unsupervised machine learning algorithm. A representative data is selected from a cluster, selecting performed for a plurality of the clusters. Based on time series data of the representative data associated with the plurality of the clusters, causal extraction is performed. Based on the causal extraction and the plurality of the clusters, a causal graph is constructed. The causal graph is embedded into vector space. Based on the embedded vector space, an artificial neural network model can be trained for managing the distributed computing environment.