Synthetic Observability Data Generation via CVAE
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for generating synthetic observability data for computing systems lack the quality and diversity needed to accurately reflect real-world system behavior, particularly in retaining the association between traces and logs.
Innovation Solution
The method involves collecting traces and logs from a system as a seed dataset and training multiple conditional variational autoencoder (CVAE) models to learn the association between traces and logs. Synthetic traces and logs are then generated using these models while preserving the learned associations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional methods use real-world data collected from production environment, then data authenticity is improved, but data quantity and accessibility are limited
Solution Approach 1:
The patent uses conditional variational autoencoder (CVAE) models to generate synthetic copies of observability data that replicate the statistical properties and associations of real production data. The models learn from limited real-world data and generate synthetic traces and logs that maintain the same correlation patterns, effectively copying the essential characteristics without requiring large volumes of actual production data.
2Quantity of substance
If synthetic data is generated by traditional tooling, then data quantity is improved, but data diversity and quality are limited
Solution Approach 1:
The patent employs conditional variational autoencoder models that can adjust and control various parameters during data generation. By modifying the conditional inputs and latent space sampling, the system generates synthetic data with controlled diversity in terms of system behaviors, error conditions, and trace patterns, thereby achieving both quantity and quality improvements.
3Object-affected harmful factors
If sensitive data is filtered from collected data, then privacy compliance is improved, but data quality and completeness are reduced
Solution Approach 1:
The patent generates synthetic copies of observability data that replicate the structural and statistical properties of real data without containing actual sensitive information. The CVAE models learn the patterns and associations from real data during training, then generate entirely synthetic samples that maintain data quality and completeness while inherently complying with privacy requirements since no real sensitive data is present in the generated output.
4Manufacturing precision
If multiple CVAE models are trained to learn association between traces and logs, then data association quality is improved, but model complexity increases
Solution Approach 1:
The patent segments the complex task of generating associated trace-log pairs into multiple specialized CVAE models. Each model is trained on specific aspects of the data (e.g., trace generation, log generation, association patterns), allowing for more focused and efficient learning. This segmentation enables high association quality while managing complexity through modular model architecture and specialized training objectives for each model component.
Data Source
AI summary
In several aspects for generation of high quality synthetic observability data for computing systems, traces and logs from a system are collected as a seed dataset. Multiple conditional variational autoencoder (VAE) models are trained using the seed dataset for learning association between the traces and the logs. Synthetic traces and logs are generated using the multiple CVAE models while retaining the association between the traces and the logs for the synthetic traces and logs.