Automated Test Case Generation for Deep Neural Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional automated testing techniques for deep neural networks (DNNs) face challenges in ensuring adequate test coverage due to uninterpretable systems, error propagation, and scalability issues with large numbers of nodes, particularly in 'black box' testing scenarios where internal workings are unknown.
Innovation Solution
The implementation of 'black box' testing techniques that use local explainer functions to generate test cases based on decision trees with linear constraints, which are easily solvable, and incorporate fairness checking to prevent discrimination, allowing for improved test coverage and scalability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional black box testing techniques are used for DNNs, then testing can be performed without knowledge of internal workings, but test coverage determination becomes unduly difficult
Solution Approach 1:
The patent introduces an explainer model as an intermediary between the DNN under test and the testing system. This explainer model provides interpretable representations of the DNN's decision-making process, enabling the testing system to determine test coverage without requiring direct knowledge of the DNN's internal workings. The explainer acts as a mediator that translates the uninterpretable DNN behavior into understandable patterns that can be analyzed for coverage assessment.
Solution Approach 2:
The patent segments the complex DNN testing problem into manageable components by using the explainer model to break down the DNN's decision process into interpretable segments. These segments can be individually analyzed and measured, making it possible to assess test coverage systematically rather than attempting to evaluate the entire uninterpretable system at once.
2Productivity
If conventional testing techniques are used for DNNs, then testing can proceed, but errors can propagate from one test case to another
Solution Approach 1:
The patent applies preliminary action by using the explainer model to analyze and understand the DNN's decision-making process before generating test cases. This preliminary analysis ensures that each test case is independently designed based on accurate understanding of the system behavior, preventing error propagation that would occur if test cases were generated sequentially without such analysis.
Solution Approach 2:
The explainer model provides feedback about the DNN's internal decision processes to the testing system. This feedback mechanism allows the testing system to adjust and refine test case generation based on accurate understanding of system behavior, preventing the propagation of errors that would occur without such feedback.
3Adaptability or versatility
If conventional techniques are used for DNN testing, then testing can be performed, but scalability to very large numbers of nodes is not readily achievable
Solution Approach 1:
The explainer model serves as a scalable intermediary that can handle DNNs of any size. By translating the behavior of even very large DNNs into interpretable representations, the explainer enables the testing system to scale to systems with thousands of nodes without being overwhelmed by the complexity of the underlying system.
Solution Approach 2:
The patent creates an interpretable copy or representation of the DNN's decision-making process through the explainer model. This copied representation can be analyzed and tested without dealing with the full complexity of the original large-scale DNN, enabling scalability to systems with very large numbers of nodes.
Data Source
AI summary
Methods, systems and computer program products for automated test case generation are provided herein. A computer-implemented method includes selecting sample input data as a test case for a system under test, executing the test case on the system under test to obtain a result, and applying the result to a local explainer function to obtain at least a portion of a corresponding decision tree. The method further includes determining at least one path constraint from the decision tree, solving the path constraint to obtain a solution, and generating at least one other test case for the system under test based at least in part on the solution of the path constraint. The steps of the method are illustratively repeated in each of one or more additional iterations until at least one designated stopping criterion is met. The resulting test cases form a test suite for testing of a deep neural network (DNN) or other system.