Multi-version Regression Tester Using Production Traffic
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional unit, component, and integration testing for software projects require excessive resources and time, often fail to achieve 100% code path coverage, and may produce false positives due to the use of staged traffic that does not accurately simulate production conditions.
Innovation Solution
A multi-version regression tester that uses production traffic and compares outputs from multiple instances executing different versions of the source code to identify faults, filtering out noisy data fields and using thresholds to determine actual defects.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional unit, component, and integration testing are performed to ensure software quality, then reliability is improved, but resource consumption and time requirements increase significantly
Solution Approach 1:
The patent creates virtual copies of production environments and traffic patterns to perform regression testing without affecting actual production systems. By copying production traffic data and system states, the testing process can evaluate software changes under realistic conditions while consuming fewer resources than traditional testing methods.
Solution Approach 2:
The system performs preliminary analysis of production traffic patterns and system behavior before executing regression tests. By pre-processing and categorizing traffic data, the system prepares test inputs in advance, reducing the time and resources needed during actual testing execution while maintaining comprehensive coverage.
2Measurement precision
If comprehensive regression testing is performed to detect all defects, then measurement precision is improved, but time consumption increases
Solution Approach 1:
The patent applies different testing strategies to different components and traffic patterns based on their specific characteristics. High-risk areas with greater defect probability receive more intensive testing, while lower-risk areas use lighter testing approaches. This localized quality assessment improves defect detection precision without uniformly increasing testing time across the entire system.
Solution Approach 2:
The system dynamically adjusts testing parameters such as traffic volume, test depth, and validation thresholds based on the specific software change being tested. By changing these parameters adaptively rather than using fixed comprehensive testing, the system achieves high measurement precision for critical areas while reducing overall testing duration.
3Adaptability or versatility
If staged traffic is used for testing to simulate production conditions, then adaptability is improved, but false positives increase due to inaccurate simulation
Solution Approach 1:
The patent introduces an intermediary layer that translates and adapts production traffic patterns for testing purposes. This intermediary process preserves the essential characteristics of real production traffic while removing elements that could cause false positives, such as transient network issues or temporary system states. The intermediary layer acts as a buffer that maintains adaptability to production conditions while ensuring test result reliability.
4Measurement precision
If multiple baseline instances are used to filter noisy data fields, then measurement precision is improved, but device complexity increases
Solution Approach 1:
The patent merges the functionality of multiple baseline instances into a unified noise filtering mechanism. By combining the outputs of multiple baselines and using statistical analysis to identify consistent patterns, the system achieves high measurement precision in noise filtering while avoiding the full complexity of managing completely separate baseline systems. The merging process simplifies the overall architecture while maintaining the benefits of multiple baselines.
Data Source
AI summary
A system and method for performing regression testing on source code provides a data sample to one or more candidate instances executing a candidate version of source code for a code base and provides the data sample to plurality of baseline instances executing a baseline version of the source code. The baseline version of the source code and the candidate version of the source code differ. The regression testing system and method obtains candidate outputs from the one or more candidate instances and obtains baseline outputs from the plurality of baseline instances based on the provided data sample. One or more faults in the candidate version of the source code can be determined by comparing the candidate outputs the baseline outputs. A filter for excluding one or more data fields from the comparison can be generated based on a comparison between baseline outputs from one of the plurality of baseline instances and baseline outputs from another of the plurality of baseline instances. Faults can also be determined based on satisfaction of a sample count, a relative threshold, an absolute threshold, or a median absolute deviation comparison.