Virtual Machine Log Aggregation via Index Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Computing resource service providers face challenges in efficiently evaluating and processing data logs from virtual computing resources, as existing methods are resource-intensive and delay diagnosis of issues due to periodic monitoring.
Innovation Solution
Implementing a system of computing system instances, such as containers or virtual machines, to aggregate and process data logs from virtual computing system servers, including index analysis and data log analysis instances that parse and transmit event data to a diagnostics datastore for further processing and report generation, reducing resource consumption on virtual computing servers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If computing resources themselves are used to generate and evaluate data logs, then event identification and tracking capability is improved, but resource availability for customer use decreases
Solution Approach 1:
The system segments the computing infrastructure into customer-visible virtual computing resources and customer-invisible computing resources. The segmentation separates event monitoring functions (performed on customer-invisible resources) from customer workloads (running on customer-visible resources), resolving the contradiction by enabling reliable event identification without consuming customer-available resources.
Solution Approach 2:
A data log store acts as an intermediary component between virtual computing resources and monitoring systems. Virtual computing resources write logs to this intermediary store, and separate computing system instances evaluate the logs. This intermediary mechanism enables comprehensive event tracking while keeping customer resources available for their intended purposes.
2Quantity of substance
If monitoring systems evaluate data logs periodically, then resource consumption on virtual computing servers is reduced, but event identification and diagnosis speed decreases
Solution Approach 1:
The monitoring system transitions from static periodic evaluation to dynamic event-driven evaluation. Computing system instances continuously monitor the data log store and immediately evaluate logs when new events occur, as indicated by log index notifications. This dynamic approach maintains low resource consumption while eliminating diagnostic delays.
Solution Approach 2:
The system implements feedback mechanisms where log index notifications trigger immediate log evaluation. When new logs are written to the data log store, the updated log index generates notifications that feedback to computing system instances, prompting immediate evaluation. This feedback loop ensures timely event identification without requiring continuous periodic monitoring.
3Reliability
If multiple computing system instances are used to process data logs, then event processing capability and availability are improved, but system complexity increases
Solution Approach 1:
Computing system instances are designed as universal, multi-functional components that can evaluate various types of logs (data logs, processor state logs, crash dumps) for multiple virtual computing resources. This universality allows the system to achieve high availability through instance replication without proportionally increasing overall system complexity.
Solution Approach 2:
The system uses copying to replicate computing system instances across multiple physical or virtual computing resources. These copied instances independently evaluate logs and provide redundant event processing capability. The copying approach increases availability while maintaining manageable complexity through standardized instance templates.
Data Source
AI summary
A service provider launches an index analysis computing system instance to evaluate logs generated by a virtual computing system server to identify events encountered by the server. The index analysis computing system instance obtains, from a datastore, a data log for the virtual computing system server and uses the log to identify the events. The index analysis computing system instance identifies, based on data from the log associated with the events, tasks to be performed to obtain processor state data for the server. The index analysis computing system instance generates, based on the identified tasks, a metadata for a file, where the metadata represents the tasks to be performed. This file is transmitted to a scheduler computing system instance which can identify the tasks based on the metadata for the file.