Microservice Logging Level Adjustment for Failure Diagnosis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current logging practices in microservice chains of cloud native applications are inadequate for identifying the root cause of failures, as they typically record only critical events, and detailed logging is often not enabled until after a failure occurs, requiring manual configuration changes and simulation of failure scenarios.
Innovation Solution
A computer-implemented method using a rule-based classifier predicts high-risk failure periods and automatically adjusts the logging level to detailed logging during these times, eliminating the need for manual restarts and simulation, and optionally employs techniques like weakest link weight-based pruning and bagging to improve prediction accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If detailed logging is enabled continuously to capture all events, then sufficient information is available for failure diagnosis, but storage space is wasted and system overhead increases
Solution Approach 1:
The system performs preliminary action by predicting potential failure periods before they occur, and proactively enables detailed logging only during these predicted high-risk windows. This prevents information loss by ensuring detailed logs are captured when failures are most likely, while avoiding continuous logging that would waste storage space during normal operation.
Solution Approach 2:
The logging level is made dynamic rather than static, automatically adjusting between detailed and critical-only modes based on real-time failure predictions. The system transitions logging granularity dynamically in response to changing system conditions, enabling detailed logging only when predicted failures occur while maintaining storage efficiency during stable periods.
2Loss of information
If logging level is changed manually to detailed level, then sufficient information is available for failure diagnosis, but system operation is interrupted and time is lost
Solution Approach 1:
The system performs self-service by automatically adjusting its own logging level based on failure predictions, eliminating the need for manual intervention. The microservice autonomously transitions between logging modes without requiring human operators to stop the system, thus preventing both information loss and operational interruption simultaneously.
Solution Approach 2:
The system enables detailed logging in advance during predicted failure periods before actual failures occur, ensuring diagnostic information is captured without requiring post-failure manual configuration. This preliminary action eliminates both the information loss that would occur with continuous critical-only logging and the time loss from manual logging level changes after failures.
3Loss of information
If continuous detailed logging is performed to capture failure information, then root cause can be identified, but system performance degrades
Solution Approach 1:
The system implements periodic action by enabling detailed logging only during specific predicted failure periods rather than continuously. This creates alternating cycles between detailed logging mode (during high-risk periods) and critical-only logging mode (during normal periods), capturing necessary failure information while maintaining system throughput during stable operation.
Solution Approach 2:
The logging intensity is made dynamic, automatically scaling between detailed and minimal modes based on real-time failure risk assessment. This dynamic adjustment ensures sufficient diagnostic information is captured during predicted failures while minimizing performance degradation during normal operation when detailed logging would unnecessarily burden the system.
4Loss of information
If manual simulation of failure scenarios is performed to recreate errors, then failure information can be collected, but time is lost and normal operation is disrupted
Solution Approach 1:
The system performs preliminary action by predicting and preparing for failures before they occur in production, enabling detailed logging during predicted failure windows. This eliminates the need for time-consuming manual simulation of failure scenarios, as the actual failures are captured in real-time with full diagnostic detail already enabled by the prediction system.
Solution Approach 2:
Instead of manually recreating failure scenarios through simulation, the system captures authentic failure data directly from production environments when failures naturally occur during predicted high-risk periods. This real-copy approach preserves actual failure information without the time loss and operational disruption associated with artificial simulation.
Data Source
AI summary
Intelligent logging of microservice failures is provided. It is predicted that failure of a microservice in a microservice chain will occur during a predicted high-risk failure period for a predicted failure type while performing a transaction corresponding to an application. A detailed logging level is determined for the microservice to record sufficient information to identify a root cause of the failure of the microservice for the predicted failure type. The microservice is directed to increase a current logging level of the microservice to the detailed logging level during the predicted high-risk failure period to record sufficient information to identify the root cause of the failure of the microservice.