Continuous Pipeline Synthetic Performance Evaluation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current solutions for testing and verifying the quality of service (QoS) and performance of software programs fail to simulate all possible hardware, software, and network conditions, leading to blind spots in field-based testing, and rely on unrealistic absolute performance objectives that are often disregarded due to variations in 'last mile' network performance.
Innovation Solution
Integrating continuous development pipeline systems with in-field synthetic performance measurements to use relative comparative performance values, allowing automated promotion or rejection of code updates based on performance across various geographic locations and client devices, thereby simulating real-user experiences and accommodating unpredictable distributed computing environments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If synthetic performance tests are used to monitor website performance, then visibility into application health is improved, but the ability to capture actual end-user interactions is limited
Solution Approach 1:
The patent combines synthetic performance tests (proactive monitoring) with passive monitoring (reactive monitoring of actual user transactions) into a unified monitoring system. This merging allows the system to both simulate user behavior and capture real user interactions, resolving the contradiction between measurement precision and adaptability by integrating both approaches.
Solution Approach 2:
The monitoring system is designed to perform multiple functions: it can execute scripted synthetic tests to proactively measure performance, simultaneously passively monitor actual user transactions, and correlate both data sources. This multi-functionality allows a single system to address both the precision of synthetic testing and the versatility of capturing real user behavior.
2Extent of automation
If absolute performance thresholds are used for code promotion, then automated control is simplified, but accommodation of last mile variations in client device conditions is reduced
Solution Approach 1:
The patent implements location-aware and device-aware performance thresholds by geolocating synthetic test results and comparing them against device-specific baseline performance data. Instead of applying a single absolute threshold globally, the system adjusts thresholds locally based on geographic location and client device characteristics, maintaining automated control while accommodating last mile variations.
Solution Approach 2:
The system dynamically adjusts performance thresholds based on multiple parameters including geographic location, time of day, network conditions, and client device type. By changing thresholds as parameters vary, the system maintains automated promotion control while adapting to different operating conditions, resolving the contradiction between automation simplicity and adaptability.
3Measurement precision
If development pipeline testing is performed, then initial performance visibility is provided, but simulation of all real-world conditions is limited
Solution Approach 1:
The patent executes synthetic performance tests at multiple stages in the development pipeline before production deployment, performing preliminary performance validation in controlled environments. This preliminary action provides initial performance visibility while the system is still being developed, and the results are later correlated with production data to verify real-world performance.
Solution Approach 2:
The system establishes feedback loops between production performance data and development pipeline testing. Production synthetic test results and passive monitoring data are fed back to update baseline performance expectations and refine testing strategies, enabling the development pipeline to better simulate real-world conditions over time through continuous learning and adaptation.
Data Source
AI summary
Continuous development pipeline systems and in-field synthetic performance test systems are interlocked to provide for automated control of promotion of program code elements within the development pipeline and in the deployment environment using relative comparative performance values rather than absolute performance thresholds, in order to better accommodate “last mile” variations in client device conditions.