Log Tiering via Event Rate Change Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Manual extraction of logs for low-cost tier storage is challenging due to the volume of logs that need to be examined, making it difficult for users to identify and separate logs that only require storage without advanced features like analytics and alerting.
Innovation Solution
A system and method that use a log classification algorithm to group logs into event types and calculate the rate change of occurrence from a base time window to a current time window, identifying candidate event types for low-cost tier log storage by checking these rate changes against a threshold range, and transferring selected logs to the appropriate storage tier.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If manual extraction of logs for low-cost tier storage is performed, then cost optimization is achieved, but the time and effort required increases significantly due to the volume of logs that must be examined
Solution Approach 1:
The system automatically performs log analysis and tiering without human intervention. The log management service autonomously evaluates logs, determines their suitability for low-cost storage, and transfers them accordingly, eliminating the need for manual examination while achieving cost optimization
Solution Approach 2:
The manual mechanical process of examining and sorting logs is replaced with an automated computational system. The log management service uses algorithms and automated workflows to analyze log characteristics, classify them into event types, and determine storage tier placement, substituting human effort with machine-based automation
2Adaptability or versatility
If all logs are stored in high-cost tier storage with advanced features, then full functionality is available, but storage costs increase unnecessarily for logs that only require basic storage
Solution Approach 1:
Different storage tiers are assigned to different log types based on their specific requirements. Logs that only need basic storage are placed in low-cost tiers, while logs requiring advanced features remain in high-cost tiers. This localized optimization ensures each log receives appropriate functionality without paying for unnecessary capabilities
Solution Approach 2:
The system changes the storage parameter (cost tier) based on log characteristics. By evaluating log event types and their properties, the system dynamically assigns logs to appropriate storage tiers, transforming the storage configuration from a uniform high-cost approach to a differentiated approach that matches actual log requirements
3Loss of energy
If manual log selection is performed to identify logs for low-cost storage, then cost savings are achieved, but the complexity and difficulty of the process increases due to the volume of logs
Solution Approach 1:
The log management service autonomously performs the entire log evaluation and tiering process without requiring user intervention. The system self-manages the complexity by implementing automated workflows that evaluate logs, classify them into event types, and determine optimal storage placement, eliminating the need for users to navigate complex manual selection processes
Data Source
AI summary
A system and method for managing logs from computing environments uses a rate change in a rate of occurrence of same event type logs from a base time window to a current time window for each of the event types to identify candidate event types for a particular tier log storage. The rate changes of the event types are checked against a threshold rate change range to identify the candidate event types. In response to selection of some of the candidate event types, the logs in the selected candidate event types are transferred to the particular tier log storage.