Dynamic Log Management for Distributed Stream Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed computing environments, such as streaming applications and stream analytics systems, the continuous operation and reduced log settings lead to a challenge in diagnosing underlying causes of frequent warnings/errors, as only important warning/error messages are recorded, limiting the ability to generate lower-level log data without disrupting the system.
Innovation Solution
A system and method for dynamic log management that allows for the generation of lower-level log data by deploying a log-configuration application to modify logger configurations on worker nodes, enabling the recording of info, debug, or trace messages without stopping the running application, using a cluster manager to coordinate tasks and adjust logging levels dynamically.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If logging is configured to record only important warning/error messages, then system performance is improved and space requirements are reduced, but the ability to diagnose underlying causes of frequent warnings/errors is limited
Solution Approach 1:
The patent implements dynamic log management by allowing the logging level configuration to be changed at runtime without stopping the streaming application. The system can switch between different logging levels (e.g., ERROR, WARN, INFO, DEBUG) dynamically based on diagnostic needs, resolving the contradiction between maintaining high performance with minimal logging and enabling comprehensive diagnostics when needed.
Solution Approach 2:
The patent changes the logging level parameter dynamically by committing a configuration job that updates the logger configuration on worker nodes. This parameter change allows the system to adjust the amount of information recorded without affecting the continuous operation of the streaming application, thus resolving the contradiction between performance and diagnostic capability.
2Loss of information
If logging level is increased to record lower-level log data, then diagnostic capability is improved, but space requirements and system performance deteriorate
Solution Approach 1:
The system dynamically adjusts logging levels based on operational needs. During normal operation, the system maintains high performance with minimal logging, but can switch to detailed logging when diagnostics are required, without impacting the continuous execution of the streaming application.
Solution Approach 2:
The system can periodically switch between different logging configurations based on operational context. The configuration job can be committed to change logging levels temporarily for diagnostics and then reverted to performance-optimized settings, creating a periodic action that balances both requirements.
3Adaptability or versatility
If the streaming application is stopped to modify logger configuration, then logging level can be changed, but system availability and performance are reduced
Solution Approach 1:
The patent enables continuous operation of the streaming application while modifying logging configurations. The configuration job is committed to the cluster and executed on worker nodes without stopping the main application, ensuring that system availability is maintained while still allowing logging level changes for diagnostic purposes.
Solution Approach 2:
The system uses an intermediary configuration mechanism (configuration job committed to the cluster) that mediates between the need to change logging configurations and the requirement to maintain system availability. This intermediary allows configuration changes to be applied dynamically without direct interruption to the streaming application execution.
Data Source
AI summary
A system and method for dynamic log management of stream processing in a distributed computing environment, such as, for example, a streaming application or stream analytics system. A streaming application can be deployed or published to a cluster, to execute as a client application. A cluster manager coordinates with worker nodes, to commit tasks associated with the streaming application. If a need arises to generate lower-level log data associated with the streaming application, for example to diagnose an underlying cause of a warning/error message, a configuration job can be committed to the cluster to execute as a separate log-configuration application. The log-configuration application operates with the cluster manager to determine the set of working nodes currently associated with the streaming application, and modify the logger configuration at those nodes, to record or otherwise provide log data according to a modified logging level, for example to provide lower-level log messages.