Balanced Software Testing With Shared Treatment Downloads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing control-treatment experiments in software testing face biases due to computing devices in the control group and treatment group downloading different data with varying characteristics, leading to skewed performance data comparisons and network usage inefficiencies.
Innovation Solution
A computing system deploys a treatment version of a software application to both groups and triggers re-installation of the control version in the control group, ensuring consistent data comparison and reducing network usage by avoiding redundant downloads.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If computing devices in the control group download the treatment version of the software application, then network usage is reduced when upgrading to the new release version, but the performance data comparison becomes biased due to different download characteristics
Solution Approach 1:
The patent applies homogeneity by ensuring that both control group and treatment group devices download and install the same treatment version of the software application. This creates uniform download characteristics across both groups, eliminating the bias that would otherwise occur from downloading different versions with different characteristics. The control group devices download the treatment version just like treatment group devices, ensuring comparable performance metrics.
Solution Approach 2:
The patent applies preliminary action by having control group devices download and install the treatment version in advance, before the actual A/B test performance measurement begins. This preliminary installation ensures that when performance is measured, both control and treatment groups have identical software versions installed, eliminating version-related performance variations. The treatment version is pre-installed on control devices so that no additional download is needed during the test period.
2Measurement precision
If computing devices in the control group reinstall the control version from local storage, then installation performance data can be accurately compared, but local storage space is required to retain the control version
Solution Approach 1:
The patent applies copying by creating a local copy of the control version software on control group devices' storage. This local copy serves as the baseline for installation performance comparison. Instead of downloading the control version during the test, devices use the pre-stored local copy, enabling accurate measurement of installation performance without network dependency. The control version is copied to local storage in advance for this purpose.
3Measurement precision
If the treatment version is deployed to both control and treatment groups, then download bias is eliminated, but the control group must re-download or re-install software
Solution Approach 1:
The patent applies preliminary action by pre-installing the treatment version on control group devices before the A/B test begins. This preliminary installation means that when the test starts, control devices already have the treatment version locally available, eliminating the need to download it during the test period. The treatment version is installed in advance on control devices, so no download time is consumed during performance measurement.
Data Source
AI summary
A computing system may determine that a software application installed at a computing device in a control group associated with a control-treatment experiment for the software application is a control version of the software application. The computing system may determine. based at least in part on the computing device being in the control group and the software application installed at the computing device being the control version of the software application. a software update for updating the software application to be sent to the computing device. The computing system may send. to the computing device. the software update to trigger the computing device to. in response to receiving the software update. reinstall the control version of the software application at the computing device from a storage device of the computing device.