Dynamic Log Retention via Neural Network Estimation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing complexity of distributed computer systems leads to inefficiencies and significant overheads in managing and administering log/event messages, including high computational and storage costs, due to the large volumes of data generated, transmitted, and stored, which results in latency and decreased available bandwidth for client applications and services.
Innovation Solution
The implementation of additional pre-processing steps, including time-series analysis and neural network-based estimation, to efficiently identify and store log/event messages, reducing data volume and latency, and employing low-cost, low-latency storage facilities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional approaches are used to manage and store log/event messages in distributed computer systems, then all log messages are retained for potential analysis, but storage costs and computational overheads increase significantly
Solution Approach 1:
The system dynamically changes the retention parameter of log messages based on their event type and historical usage patterns. By analyzing query frequencies and access patterns, the system adjusts which log messages are retained and for how long, transitioning from a static retention policy to a dynamic one that adapts to actual usage needs.
Solution Approach 2:
Different retention policies are applied to different event types based on their specific characteristics and usage patterns. High-value events that are frequently queried retain longer, while low-value events are discarded sooner. This local differentiation of retention quality optimizes storage efficiency while maintaining reliability for important events.
2Loss of information
If all log/event messages are stored with equal retention periods, then comprehensive data is available for analysis, but processing time and computational resources increase
Solution Approach 1:
The system performs preliminary classification of log messages by event type and pre-determines retention periods based on historical analysis before the actual querying occurs. This advance preparation allows the system to quickly retrieve relevant logs without performing complex analysis at query time, reducing processing delays.
Solution Approach 2:
The system retains only the necessary portion of log messages required for actual analysis needs, rather than preserving all logs indefinitely. By using time-series analysis and neural networks to predict which event types will be queried, the system retains sufficient data for analysis while discarding excess logs that would consume resources without providing value.
3Adaptability or versatility
If manual determination of retention periods is used, then retention policies can be customized per event type, but operational complexity and costs increase
Solution Approach 1:
The system automatically determines and enforces retention periods for log messages without requiring manual intervention. It uses time-series analysis and neural network models to self-analyze query patterns and self-adjust retention policies for different event types, eliminating the need for manual policy configuration and reducing operational complexity.
Solution Approach 2:
The system continuously monitors query patterns and usage statistics, using this feedback to dynamically adjust retention policies. The neural network analyzes historical data and provides feedback loops that automatically refine retention decisions, enabling customized policies to adapt to changing needs without manual reconfiguration.
4Duration of action of stationary object
If long retention periods are applied to all log messages, then historical analysis capability is enhanced, but storage costs and bandwidth consumption increase
Solution Approach 1:
The system transitions from static retention periods to dynamic retention durations that automatically adjust based on event type characteristics and usage patterns. Retention duration becomes a flexible parameter that extends longer for high-value events and shorter for low-value events, optimizing the balance between historical analysis capability and storage cost.
Data Source
AI summary
The current document is directed to methods and systems that efficiently process and store log/event messages generated within distributed computer facilities. Various different types of initial processing steps may be applied to a stream of log/event messages received by a message-collector system or a message-ingestion-and-processing subsystem. The currently disclosed methods and systems employ additional pre-processing steps to identify the types of received log/event messages, monitor event-type-associated log/event-message-usage-delay histories, and employ time-series-analysis-based and/or neural-network-based estimation of event-type-associated log/event-message usage to efficiently store log/event-messages in low-cost and low-latency storage facilities.