Modular Pipeline for Single-Pass Log Metric Aggregation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In cloud computing environments, generating performance metrics from large user interface logs is computationally expensive and inefficient, requiring multiple passes through the data set, which decreases performance and increases complexity as the number of metrics increases.
Innovation Solution
A modular data processing pipeline that generates performance metrics by consuming the log data only once, allowing for an arbitrary number of aggregated metrics to be computed efficiently, with new metrics added simply by updating aggregation expressions, and maintaining a fixed output data format.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple performance metrics are generated from the same data set, then the number of performance metrics increases, but the data set needs to be consumed multiple times resulting in decreased efficiency
Solution Approach 1:
The system performs preliminary actions by consuming the data set once and pre-computing and storing intermediate aggregation results in a cache memory during the first pass. This preliminary computation enables subsequent metrics to be generated from the cached results without requiring additional passes through the original data, thereby resolving the contradiction between generating multiple metrics and maintaining efficiency
Solution Approach 2:
The metric generation process is segmented into distinct passes: a first pass that computes and caches intermediate results, and subsequent passes that consume the cached data to generate specific metrics. This segmentation allows the system to handle multiple metrics efficiently by avoiding repeated full data scans, thus improving productivity while maintaining versatility
2Adaptability or versatility
If the number of performance metrics being generated increases, then more comprehensive performance analysis is achieved, but the data set needs to be consumed multiple times increasing computational expense
Solution Approach 1:
The system performs preliminary computations during the first data pass by calculating and storing intermediate aggregation results in cache memory. This preliminary action eliminates the need for repeated expensive full data scans when generating additional metrics, thereby reducing computational expense while maintaining the ability to generate comprehensive performance analyses
Solution Approach 2:
The system creates copies of intermediate computation results and stores them in cache memory for reuse. Instead of re-processing the original large data set multiple times, the system works with cached copies of aggregated data, significantly reducing computational expense while enabling generation of multiple performance metrics
Data Source
AI summary
System and methods are described for parsing log data items to find events in the log data items, wherein an event comprises at least a portion of a log data item matching an event definition; aggregating a plurality of performance metric definitions into a single expression; and generating a plurality of performance metrics from the events by applying the single expression to the events.