Log Management Framework With OS-Level Failover Monitoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing log management systems in cloud environments face disruptions when container platform or container runtime services crash, rendering log management services useless and interrupting system monitoring and troubleshooting.
Innovation Solution
A framework that includes a task scheduler and deployer daemon to continuously monitor and maintain operational instances of log collection and storage services, automatically correcting non-operational states by redeploying services at predetermined intervals.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If log management services are deployed on container platform services or container runtime services, then hardware resource utilization is optimized and automated management is implemented, but the log management services become vulnerable to crashes of these container services
Solution Approach 1:
The system segments log management services into multiple independent instances distributed across different container platform services and container runtime services. This segmentation ensures that a crash in one container service does not affect all log management services, as other instances continue to operate independently on different platforms.
Solution Approach 2:
The system dynamically changes the operational state parameters of log management service instances based on health check results. When a service instance is detected as non-operational, the system transitions from a state where all instances run on container services to a state where at least one instance runs directly on the operating system, thereby adapting to changing conditions and maintaining reliability.
2Loss of energy
If container platform services or container runtime services crash, then hardware resource utilization is freed, but log management services are interrupted and logs become useless for searching
Solution Approach 1:
The system performs preliminary actions by deploying at least one log management service instance directly on the operating system before any container service crashes occur. This ensures that when container services crash and free up resources, the log management functionality remains uninterrupted because the OS-level instance continues to operate independently.
Solution Approach 2:
The operating system acts as an intermediary layer between container services and log management services. By running at least one log management instance directly on the OS, the system creates a stable intermediary that maintains log management continuity even when container services above it crash and are restarted.
3Reliability
If multiple instances of log management services are maintained across different platforms, then service availability is improved, but system complexity increases
Solution Approach 1:
The system applies local quality by differentiating the deployment locations of log management service instances based on specific needs. At least one instance is deployed directly on the operating system with high availability characteristics, while other instances can be deployed on container platform services with automated management characteristics. This localized differentiation optimizes both reliability and complexity management.
Data Source
AI summary
A system includes one or more data processors and a non-transitory computer-readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform operations. The operations include executing a deployer daemon on an operating system of a local host to maintain operation of at least one instance of combined services, which include a log collection service and a storage and retrieval service. The operations include performing, via a task scheduler executing on the operating system, a cycle of operational status checks on the combined services, and repeating the cycle at a predetermined time interval. The operations further include triggering, via the task scheduler, the deployer daemon to correct a non-operational result if the operational status check returns the non-operational result, and adding an entry, via the task scheduler, into a results file indicative of corrective action taken.