Automated Software Regression Testing via N-gram Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing software tracing technologies face challenges in efficiently analyzing execution sequences and input streams to identify performance issues and security breaches, particularly in asynchronous operations, where predicting behavior under varying loads is difficult and resource overhead is high.
Innovation Solution
The use of n-gram analysis to quantify and visualize execution sequences and input streams, allowing for the creation of behavior models that compare production data to test suite coverage, identify security breaches, and perform regression testing across software versions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a heavyweight tracer is used to gather comprehensive performance data, then measurement precision is improved, but device complexity and resource overhead increase significantly
Solution Approach 1:
The patent segments the tracing system into multiple lightweight tracer instances that can be distributed across different devices or processes. Instead of using a single heavyweight tracer, the system divides the monitoring task into smaller units that collect data independently and can be aggregated later, reducing the overhead on any single device while maintaining comprehensive coverage.
Solution Approach 2:
The patent introduces an intermediary component that acts as a data aggregation and analysis layer between the lightweight tracers and the final performance analysis. This intermediary receives data from multiple lightweight tracers, performs preliminary processing and correlation, and presents consolidated results, thereby reducing the complexity burden on individual tracer components while maintaining measurement precision.
2Device complexity
If a lightweight tracer is used to reduce resource overhead, then device complexity is reduced, but measurement precision and data accuracy deteriorate
Solution Approach 1:
The patent merges data from multiple lightweight tracer instances to achieve comprehensive measurement coverage. By combining the outputs of several lightweight tracers that each monitor specific aspects of system behavior, the system reconstructs a complete and accurate performance picture without requiring any single tracer to be heavyweight, thus maintaining low overhead while improving measurement precision through aggregation.
Solution Approach 2:
The patent designs the lightweight tracer with multi-functional capabilities, enabling it to perform multiple monitoring tasks and collect various types of performance data through a single unified interface. This universal tracer can adapt to different monitoring requirements and work across multiple contexts, maximizing the measurement precision achieved from a lightweight, low-overhead design.
3Measurement precision
If comprehensive tracing is performed to capture all application behavior, then measurement precision is improved, but loss of time increases due to data processing overhead
Solution Approach 1:
The patent performs preliminary filtering, aggregation, and prioritization of trace data at the source before the data requires extensive processing. Lightweight tracers pre-process their collected data by identifying and flagging only the most significant events and patterns, reducing the volume of data that needs detailed analysis later. This preliminary action maintains comprehensive behavior capture while significantly reducing subsequent processing time.
Solution Approach 2:
The patent applies partial action by focusing tracing efforts on the most critical performance aspects and time periods. Instead of uniformly processing all trace data with equal depth, the system applies intensive analysis only to identified problem areas and uses lighter processing for routine operations, thereby maintaining measurement precision for critical behaviors while reducing overall processing time through selective attention.
Data Source
AI summary
Regression testing of an application may gather performance tests for multiple functions within an application and determine when performance changes from one version of the application to another. The analysis may be further broken down by input sequences that may be processed by various functions. A detailed regression analysis may be presented as a heat map or other visualizations. A regression testing system may be launched during a build process by automatically launching a set of performance tests against an application. In many cases, the application may be executed in a system with a known or consistent performance capabilities. The application may be executed and tested in a new version and at least one prior version on the same hardware and software execution environment, so that results may be normalized from one execution run to another. A regression testing system may be deployed as a paid-for service that may integrate into a source code repository.