Log Sampling Storage System Reducing Cloud Log Volume
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increased volume of backend log messages in cloud environments leads to resource consumption and prolonged diagnostic processes due to the complexity and unpredictability of frontend-backend interactions, resulting in elevated operational costs and inefficient log searching.
Innovation Solution
A log storage system that compares incoming log messages to historical patterns, generating a similarity score to identify outlier messages, which are stored in full, while non-outlier messages are stored as links to sample patterns, reducing the volume of stored log data and conserving hardware resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If all backend log messages are stored in full detail, then complete diagnostic information is preserved, but hardware resources are consumed and operational costs increase
Solution Approach 1:
The patent creates representative copies (patterns) of log message series that capture the essential diagnostic information. Instead of storing every individual log message, the system stores pattern templates that represent groups of similar log sequences, along with references to sample instances. This copying approach preserves diagnostic capability while dramatically reducing storage requirements.
Solution Approach 2:
The system extracts and stores only the distinctive outlier log message series that deviate from established patterns, while referencing sample logs for typical patterns. This extraction principle separates the essential diagnostic information (outliers and pattern definitions) from redundant data (repetitive log messages), storing only what is necessary for effective diagnostics.
2Measurement precision
If all backend log messages are stored in full detail, then complete search capability is available, but diagnostic process time increases due to large data volume
Solution Approach 1:
The patent segments log messages into distinct patterns and categories based on their characteristics. By organizing logs into pattern groups with representative samples, the system creates a segmented structure that enables efficient searching. Queries can quickly identify relevant pattern groups rather than searching through every individual log message, significantly reducing diagnostic time while maintaining search accuracy.
3Adaptability or versatility
If cloud environment complexity increases with more frontend-backend interactions, then system functionality improves, but log volume and storage requirements increase
Solution Approach 1:
The patent creates universal pattern templates that can represent multiple specific log message series. A single pattern can capture the essence of many different but structurally similar log sequences generated by complex frontend-backend interactions. This multi-functionality allows the system to handle diverse log types from complex cloud environments using a compact set of reusable patterns, preventing log volume from scaling linearly with system complexity.
Data Source
AI summary
A log sampling and storage system reduces volumes of stored log information. A log storage engine receives a series of log messages responsive to a request. The engine compares that series to various patterns previously generated from historical log data. This comparison can reference •a name of the request; •a total time to process the request; •an item count; •a message length; and/or •a message hash code. Comparing incoming log series with the predefined patterns, results in generation of a similarity score. If the score falls outside a tolerance, the existence of an outlier log message series is indicated, and that outlier log message series is stored in its entirety. However, if the similarity score falls within a tolerance, similarity to an existing predefined pattern is indicated. Rather than storing the (non-outlier) log message series, the engine directs storing a link to a sample log message series reflecting the pattern.