Supervised ML Validation Using Corrected Failure Probability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current machine learning methods, particularly those using artificial intelligence, face challenges in ensuring the reliability and validation of supervised learning processes, especially in safety-critical applications, where providing suitable security evidence for ongoing operation is a significant hurdle.
Innovation Solution
A method for supervised machine learning that involves dividing data sets into training and validation sets, allowing for a thorough training phase followed by a validation phase where the system checks the output data against known correct results, with a tolerance range for accuracy and a statistical significance correction to ensure reliability, thereby meeting stringent security requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If machine learning is used to optimize safety-relevant applications during ongoing operation, then the system can adapt and improve performance, but it becomes difficult to provide suitable safety proof and meet approval requirements
Solution Approach 1:
The data pool is segmented into training data records and validation data records. The training phase uses training data to optimize the algorithm during ongoing operation, while the validation phase uses validation data to provide safety proof by checking quality and calculating failure probabilities. This segmentation allows both adaptability and reliability to coexist.
Solution Approach 2:
Before the algorithm is deployed for safety-critical operations, a validation phase is performed in advance to check the quality of the learned algorithm. The system calculates an actual failure probability and compares it against a target failure probability, ensuring safety requirements are met before ongoing operation begins.
2Measurement precision
If the system is trained with sufficient data to learn correct functional relationships, then the learning accuracy improves, but the complexity of validation and safety verification increases
Solution Approach 1:
The validation process extracts specific measurable parameters from the learned algorithm, including the actual failure probability calculated from validation data. By focusing on these extracted metrics rather than comprehensive manual verification, the system maintains high learning accuracy while managing validation complexity through automated statistical analysis.
3Reliability
If a strict target failure probability is set for safety validation, then the reliability of the system improves, but the difficulty of achieving successful validation increases
Solution Approach 1:
The system implements feedback by calculating the actual failure probability from validation data and comparing it against the target failure probability. This feedback mechanism guides whether the algorithm meets safety requirements, providing clear guidance for iterative optimization while maintaining strict reliability standards.
Data Source
Figure 1~2
Figure 3
Figure 4
AI summary
The invention relates to supervised machine learning of a computer-implemented method for carrying out a technical process (TPR), in which a data pool (DP) containing data sets (x;y) with input data and output data describing a correct process result is created, the data sets (x;y) are divided into training data sets (TD) and validation data sets (VD), the computer-implemented method is trained in a training phase (TP), wherein method parameters of the method are varied during repeated process executions, in a validation phase (VP) the trained method is checked by comparing the output data calculated with the input data using the trained method with the output data describing the correct process result, and as a result of the comparison, an actual probability of failure (Ptrue) for the method is calculated.It is proposed that an empirical failure probability (Pemp) be set for the training phase (TP), which is at most as large as a predetermined target failure probability (PFD). The validation phase (VP) is initiated after it has been determined in the training phase (TP) that the empirical failure probability (Pemp) is not exceeded. Furthermore, it is proposed that statistical significance (α) of the result of the validation phase (VP) be taken into account by setting a corrected failure probability (Pval) in the validation phase (VP) for comparison with the actual failure probability (Ptrue). This corrected failure probability should be lower than the predetermined target failure probability (PFD).