Edge Log Filtering for Data Center Volume Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The high volume of log messages generated by edge systems burdens data centers, leading to increased storage costs and prolonged troubleshooting times, which can result in errors and damage to an organization's reputation.
Innovation Solution
Implementing a system where edge systems collect and process log messages to identify representative messages, discarding non-representative ones, and sending only these representative messages to the data center for storage and analysis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If all log messages are sent to the data center, then complete log data is available for analysis, but storage volume and processing time increase significantly
Solution Approach 1:
The edge system performs preliminary filtering and selection of log messages before transmission to the data center. By identifying and marking representative log messages at the source (edge system) before sending them to the data center, the system reduces the volume of transmitted data while ensuring that the most important messages are preserved for analysis.
Solution Approach 2:
The system extracts only the essential and representative log messages from the complete set of generated logs. The edge system identifies representative messages that capture the essential information needed for troubleshooting, separating them from redundant or less important messages, and sends only these extracted messages to the data center.
2Reliability
If all log messages are stored in the data center, then comprehensive analysis is possible, but storage costs and processing time increase
Solution Approach 1:
The edge system performs preliminary filtering and selection of log messages before transmission to the data center. By identifying and marking representative log messages at the source (edge system) before sending them to the data center, the system reduces the volume of transmitted data while ensuring that the most important messages are preserved for analysis.
Solution Approach 2:
The system extracts only the essential and representative log messages from the complete set of generated logs. The edge system identifies representative messages that capture the essential information needed for troubleshooting, separating them from redundant or less important messages, and sends only these extracted messages to the data center.
3Quantity of substance
If representative log messages are selected at the edge system, then transmission volume is reduced, but filtering complexity increases
Solution Approach 1:
The edge system autonomously performs the filtering and identification of representative log messages without requiring external intervention or complex centralized processing. Each edge system independently analyzes its own log messages, identifies representatives, and prepares them for transmission, making the system self-sufficient and reducing overall complexity.
Solution Approach 2:
The filtering and selection process is distributed to local edge systems rather than centralized at the data center. Each edge system applies filtering logic locally to its own log messages, allowing the system to handle different types of log messages with appropriate local rules without requiring a single complex centralized filtering system.
Data Source
AI summary
Computer-implemented methods and systems described herein are directed to reducing volumes of log messages sent from edge systems to a data center. The computer-implemented methods performed at each edge system includes collecting a stream of log messages generated by one or more event sources of the edge system. Representative log messages of the stream of log messages are determined. The edge systems may discard non-representative log messages from data storage devices at the edge system. The representative log messages are sent from each of the edge systems to the data center where the representative log messages are received and stored in data storage devices of the data center, thereby reducing the volumes of log messages sent from the edge systems to the data center.