On-Demand Aggregated Logging for Distributed Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed systems, administrators face significant challenges in troubleshooting due to the need to collect and sift through large amounts of irrelevant data from local logs, often requiring lengthy processes and potentially involving application restarts to change logging levels, which can be impractical due to business continuity needs.
Innovation Solution
Implementing on-demand aggregated logging, where a logging condition is identified and sent to multiple logging elements, which determine if their local runtime conditions match, and if so, send relevant logs to an aggregating logger, creating an aggregated log that is easily accessible without requiring application restarts, thus reducing data storage and improving troubleshooting efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If traditional logging mechanisms collect all local logs from distributed system components, then complete system-wide log data is obtained, but the amount of irrelevant data increases significantly and troubleshooting time increases
Solution Approach 1:
The system performs preliminary filtering of log events at the source components before aggregation, using event type filters and filter expressions to pre-process and select only relevant events. This preliminary action reduces the volume of data that needs to be collected and analyzed later, directly addressing the contradiction by maintaining completeness of relevant information while reducing overall data volume and analysis time
Solution Approach 2:
The logging system is segmented into distributed logging elements at each system component that independently evaluate and filter their own log events. Each logging element operates autonomously to segment the log generation process, allowing local filtering decisions that reduce the total amount of data transmitted and aggregated centrally, thus reducing troubleshooting time while preserving relevant information
2Measurement precision
If logging levels are changed to capture more detailed information, then troubleshooting insight is improved, but application restart is required which disrupts business continuity
Solution Approach 1:
The system implements dynamic filtering where filter criteria and event type filters can be modified at runtime without requiring application restart. The logging elements continuously evaluate events against dynamically updated filter expressions, allowing administrators to adjust logging detail levels on-demand while the application remains running, thus maintaining both measurement precision and business continuity
Solution Approach 2:
The system allows runtime modification of filter parameters and evaluation criteria that control the detail level of logged events. By changing these software parameters rather than requiring restart, the system can adjust logging precision dynamically, resolving the contradiction between obtaining detailed troubleshooting information and maintaining business continuity
3Reliability
If comprehensive logs are collected from all components, then all potential issues can be identified, but data storage requirements increase significantly
Solution Approach 1:
The system extracts and filters out irrelevant log events at the source components using event type filters and filter expressions before aggregation. This extraction process removes unnecessary data elements while preserving relevant troubleshooting information, directly addressing the contradiction by reducing stored data volume while maintaining the ability to identify actual issues
Solution Approach 2:
Filtering and event selection operations are performed preliminarily at each logging element before data is transmitted for storage. This preliminary processing action reduces the volume of data that needs to be stored centrally while ensuring that all potentially relevant events for issue identification are preserved, thus maintaining reliability while reducing storage requirements
Data Source
AI summary
On-demand aggregated logging for distributed systems is described. An identification of a logging condition is received via a first user interface. The logging condition is sent to multiple logging elements, wherein each of the multiple logging elements is executing on a corresponding one of multiple distributed systems components. A logging element determines whether a local runtime condition matches the logging condition. The logging element sends a local log associated with the local runtime condition to an aggregating logger, wherein the local log is supplemented with information that identifies the logging condition, in response to a determination that the local runtime condition matches the logging condition. The aggregating logger creates an aggregated log based on the local log and the supplemented information. The aggregated log is output via a second user interface to enable resolution of a problem associated with the multiple distributed systems components.