Run-time Application Performance Scoring via Telemetry and ML
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional benchmarking applications for determining run-time performance of computing systems are resource-intensive, disrupt user experience by requiring exclusive machine access, and lack standardization, making them unsuitable for continuous performance measurement and application-agnostic performance evaluation.
Innovation Solution
A method involving time series telemetry data streams, statistical feature extraction, and machine learning performance score models to determine run-time performance with minimal system impact, using a service provider system to train models that can predict performance scores for various applications across different configurations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional benchmark applications are used to determine run-time performance, then performance measurement capability is improved, but system performance and user experience deteriorate due to high resource consumption
Solution Approach 1:
The patent creates a virtual copy of the benchmarking environment using virtual machines or containers that run alongside the production system. This allows performance testing to occur on a replicated system rather than the actual production system, eliminating the performance impact on users while maintaining measurement accuracy. The virtual copy captures telemetry data that mirrors production behavior without consuming production resources.
Solution Approach 2:
The patent replaces the mechanical execution of benchmark applications with a software-based telemetry collection and analysis system. Instead of running compute-intensive benchmark workloads, the system collects runtime telemetry data from actual application execution and uses machine learning models to predict performance metrics. This substitution eliminates the need for resource-intensive mechanical benchmarking while maintaining measurement capability.
2Measurement precision
If custom benchmark applications are designed for specific applications, then measurement precision for that application is improved, but device complexity and ease of operation worsen due to lack of standardization
Solution Approach 1:
The patent develops a universal benchmarking system that can measure performance across multiple different applications and workloads using a single standardized platform. The system collects generic telemetry data that can be analyzed for various application types without requiring custom benchmark applications for each specific case. This universal approach reduces complexity while maintaining measurement precision through application-agnostic performance indicators.
Solution Approach 2:
The patent transforms the benchmarking approach by changing from application-specific test procedures to standardized telemetry parameter collection. The system monitors a consistent set of performance parameters (CPU usage, memory allocation, I/O operations, etc.) across different applications and uses machine learning to derive application-specific insights from these universal parameters. This parameter-based approach standardizes the process while maintaining measurement relevance.
3Productivity
If benchmark applications run during user operations, then performance optimization capability is improved, but ease of operation deteriorates due to resource contention
Solution Approach 1:
The patent performs performance measurements and optimizations in advance during off-peak hours or using virtualized environments, so that when users need to operate the system, the performance characterization is already complete. The system proactively collects telemetry data and generates performance recommendations before users require system resources, eliminating contention while maintaining optimization capability.
Solution Approach 2:
The patent introduces an intermediary layer between the application and the performance measurement process. This intermediary collects telemetry data in a non-intrusive manner, using hooks or instrumentation that do not interfere with application execution. The intermediary processes performance data separately, allowing users to operate the application normally while performance optimization occurs in the background without resource contention.
Data Source
AI summary
Techniques are disclosed for determining the run-time performance of an application executing on a computing system with low impact on the performance of the computing system. For example, a time series telemetry data stream is obtained for each of a plurality of key performance indicators during run-time execution of the application on a computing system having a given system configuration. One or more statistical features are extracted from each time series telemetry data stream. Model parameters of a machine learning performance score model are populated with values of the extracted statistical features. A run-time performance score of the application is then determined using the model parameters of the machine learning performance score model populated with the values of the extracted statistical features.