Telemetry Agent Restart and Configuration Update
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Virtual computing systems face challenges in reliably transmitting telemetry data due to limited internal buffers in agents, data loss during transmission, and mis-behaving agents that consume excessive resources, leading to reduced system reliability and efficiency.
Innovation Solution
Implementing a telemetry data replication system using a commit log to store and transmit data in small chunks, ensuring reliable delivery and handling mis-behaving agents through self-policing mechanisms that monitor and manage resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If agents use limited internal buffers to store telemetry data, then data transmission is simplified, but data loss occurs during transmission
Solution Approach 1:
The patent segments telemetry data into small chunks and stores them in a commit log on durable storage rather than relying on limited in-memory buffers. This segmentation approach allows data to be persisted reliably and transmitted incrementally, preventing data loss while maintaining transmission simplicity.
Solution Approach 2:
The patent performs preliminary action by writing telemetry data to a commit log on durable storage before transmission occurs. This ensures data is safely persisted and ready for transmission without relying on volatile memory, thereby preventing data loss during transmission interruptions.
2Productivity
If mis-behaving agents consume excessive resources, then system resource utilization increases, but system reliability decreases
Solution Approach 1:
The patent implements feedback mechanisms where the telemetry control system monitors agent behavior and resource consumption. When agents exhibit mis-behavior or consume excessive resources, the system detects this through telemetry data and takes corrective actions such as restarting agents or updating configuration files to prevent resource exhaustion and maintain system reliability.
Solution Approach 2:
The patent enables agents to self-polic their resource usage through configuration management. The telemetry control system updates agent configuration files based on observed behavior, allowing agents to self-regulate their resource consumption and prevent mis-behavior without external intervention, thereby maintaining both productivity and reliability.
3Reliability
If agents are restarted frequently to handle terminations, then agent availability improves, but system stability deteriorates
Solution Approach 1:
The patent performs preliminary action by updating the agent configuration file before restarting the agent. This ensures that the agent is configured correctly ahead of time, reducing the likelihood of immediate re-termination and promoting system stability while maintaining agent availability through controlled restarts.
Solution Approach 2:
The patent changes configuration parameters of the agent based on termination records and system state before restarting. By adjusting configuration parameters such as resource limits or operational settings, the system addresses the root cause of terminations, improving agent availability while preventing instability from repeated restart cycles.
Data Source
AI summary
A system and method include determining, by a telemetry control system of a telemetry system that an agent associated with the telemetry control system terminated during operation. The agent collects telemetry data from data sources associated with the telemetry system. The system and method also include determining that a number of times the agent has terminated is greater than a predetermined threshold, restarting the agent after a first predetermined delay in response to exceeding the predetermined threshold, and determining that the agent terminated again within a predetermined time period upon restarting. The system and method further include updating a configuration file of the agent in response to the termination within the predetermined time period and restarting the agent with the updated configuration file. The updating is based upon an agent termination record of the agent.