Log Collection Agent Metadata Supplementation via Unique Identifiers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed computing environments, particularly in Platform as a Service (PaaS) cloud services, locating and processing log messages from multiple hosts is logistically challenging due to the high volume of requests and large metadata sizes, leading to increased network traffic and bandwidth utilization.
Innovation Solution
A method where a log collection agent on the same host as an executing process requests a unique identifier (UID) for immutable metadata from a centralized management system, which is then used to supplement log messages instead of the actual metadata, reducing the need for repeated requests and minimizing message size.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If log collection agents send complete metadata with each log message to a centralized logging system, then log messages contain comprehensive information for analysis, but network traffic and bandwidth utilization increase significantly
Solution Approach 1:
The patent segments metadata into two categories: immutable metadata (sent once as UIDs) and mutable metadata (sent with each log message). This segmentation allows the system to minimize network traffic by sending only essential information repeatedly, while still providing comprehensive log analysis capabilities through the centralized system's ability to retrieve full metadata using the UIDs.
Solution Approach 2:
The patent introduces UIDs as intermediary identifiers that replace complete metadata in log messages. These UIDs act as references that the centralized logging system can use to retrieve full metadata information, thereby reducing network bandwidth utilization while preserving information completeness.
2Loss of information
If log collection agents request complete metadata from the centralized management system for each log message, then log messages are supplemented with comprehensive metadata, but the number of requests to the management system increases
Solution Approach 1:
The patent applies preliminary action by having log collection agents request immutable metadata (UIDs) once when processes are first created, before any log messages are generated. This preliminary retrieval eliminates the need for repeated requests to the centralized management system, as the same UIDs are reused for all subsequent log messages from those processes.
Solution Approach 2:
The patent uses copying by creating local caches of immutable metadata (UIDs) at the log collection agents. These cached UIDs are copied once from the centralized management system and then reused locally for supplementing multiple log messages, eliminating repeated network requests and reducing the number of interactions with the management system.
3Loss of information
If the centralized logging system stores complete metadata for all log messages, then log analysis can access full information, but storage requirements increase
Solution Approach 1:
The patent extracts the bulk of metadata information from individual log messages and stores it separately in the centralized management system under process identifiers. Only compact UID references are embedded in log messages, while the full metadata resides in the management system's database. This extraction significantly reduces the quantity of data stored in the logging system while preserving information availability through on-demand retrieval.
Data Source
AI summary
Mechanisms for supplementing log messages are disclosed. A log collection agent initiates, to a centralized management system, a request for metadata associated with a first executing process of the computing device. The log collection agent determines that a log message has been generated by the first executing process. The log collection agent generates a log entry that includes the log message and the metadata. The log collection agent sends the log entry to a centralized logging system that consolidates log messages from a plurality of executing processes including the first executing process.