Device Matrix Prediction Using Runtime Error Log Correlation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Developing applications requires testing on thousands of OS and device combinations, which is computationally infeasible due to resource constraints and unknown dependencies, making it difficult to determine which devices and operating systems should be tested.
Innovation Solution
A machine learning-based device matrix prediction system that uses iterative and collaborative processes to predict optimal device and OS combinations for testing, leveraging machine learning models to analyze run-time error logs and API package information for efficient device selection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If exhaustive testing on all device and OS combinations is performed, then testing completeness is improved, but computational resources and time required increase significantly
Solution Approach 1:
The system performs partial testing by predicting a subset of critical device-OS combinations that are most likely to expose bugs, rather than exhaustively testing all possible combinations. The machine learning model identifies and prioritizes the most relevant test cases based on historical error data, application characteristics, and device attributes, enabling effective testing with reduced computational resources
Solution Approach 2:
The system performs preliminary analysis by training machine learning models on historical error logs and device data before actual testing begins. This preliminary action enables the system to predict which device-OS combinations are most likely to fail, allowing testers to prepare and prioritize test cases in advance rather than performing exhaustive testing during the actual testing phase
2Measurement precision
If machine learning models are trained on historical error data, then prediction accuracy is improved, but data processing time and computational resources increase
Solution Approach 1:
The machine learning system is segmented into multiple components that can be trained and updated independently. Different models handle different aspects of prediction (e.g., one model for device compatibility, another for OS version issues), allowing selective training and updating of specific model components rather than retraining entire systems, thus reducing overall training time while maintaining accuracy
Solution Approach 2:
The system implements periodic model training and updating schedules rather than continuous training. Models are retrained at intervals using newly accumulated error data, balancing the need for accurate predictions with the computational cost of training. This periodic approach allows the system to maintain prediction accuracy while managing resource consumption efficiently
Data Source
AI summary
Storing a set of emulated devices, the set of emulated devices including a plurality of different emulated devices. Obtaining one or more filter parameters. Selecting, based on the filter parameters, a subset of emulated devices from the set of emulated devices. Initiating testing of an application on a particular emulated device of the subset of emulated devices. Obtaining, at run-time while the application is being tested, first error log data from the testing of the application on the particular emulated device of the subset of emulated devices. Parsing, at run-time while the application is being tested, the first error log data, thereby generating a set of first error log data components. Adjusting, at run-time while the application is being tested, one or more weights of a machine learning model based on the set of first error log data components. Providing, at run-time while the application is being tested, the set of first error log data components to the machine learning model. Determining, at run-time while the application is being tested by the machine learning model based on the set of the first error log data components, a second subset of emulated devices from the set of emulated devices, wherein each emulated device of the second subset of emulated devices correlates to at least one of the first error log data components of the set of first error log data components. Testing the application on at least one of the second subset of emulated devices.