Autonomy Safety Envelope for ML Control Verification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Automated systems, particularly in safety-critical applications, face challenges in achieving both safety and security due to complex tasks that are difficult to define with variables and conditional statements, and the reliance on potentially unreliable machine learning algorithms that are hard to verify and certify.
Innovation Solution
The implementation of a system that combines machine learning algorithms with control-theory based algorithms and a safety monitor to ensure safe and secure operations, using a safety envelope to preempt maneuvers outside a predetermined operational safety boundary, thereby ensuring both safety and security.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If machine learning algorithms are used to perform complex tasks, then the system's ability to handle difficult tasks is improved, but the reliability and verifiability of the system deteriorates
Solution Approach 1:
The system is divided into distinct modules: a machine learning module for complex task processing and a control theory module for safety-critical decision making. This segmentation allows each module to specialize - the ML module handles adaptability while the control theory module ensures reliability through verifiable algorithms.
Solution Approach 2:
A safety monitor acts as an intermediary between the machine learning module and the final control outputs. It receives outputs from the ML algorithm, verifies them against safety constraints using control theory, and only allows execution if they meet safety requirements. This mediator enables the system to use untrusted ML algorithms while maintaining certified safety.
2Productivity
If automation is increased to handle complex tasks, then productivity is improved, but safety and security assurances deteriorate
Solution Approach 1:
The system dynamically switches between different control modes based on the task requirements. For non-critical tasks, it uses flexible machine learning for high productivity. For safety-critical decisions, it transitions to verifiable control theory algorithms, ensuring safety assurances are maintained even as automation increases.
Solution Approach 2:
The safety monitor is pre-configured with safety constraints and emergency protocols before operation. It continuously monitors system state and is prepared to intervene with predetermined safety measures if anomalies are detected, providing beforehand cushioning against potential safety failures.
3Adaptability or versatility
If machine learning algorithms are used without verification, then adaptability is improved, but harmful factors increase
Solution Approach 1:
The safety monitor provides continuous feedback to the machine learning module by monitoring its outputs against safety constraints. When the ML algorithm produces outputs that violate safety requirements, the safety monitor feeds back correction signals or overrides the commands, preventing harmful actions while allowing the ML system to maintain its adaptability.
Data Source
AI summary
Safe practical autonomy is ensured by encapsulating an unreliable or untrusted machine learning algorithm within a control-based algorithm. A safety envelope is utilized to ensure that the machine learning algorithm does not output control signals that are beyond safe thresholds or limits. Secure practical autonomy is ensured by verification using digital certificates or cryptographic signatures. The verification may be for individual partitions of an autonomous system or apparatus. The partitions include trusted and untrusted partitions. Trusted partitions are verified for security, while untrusted partitions are verified for safety and security.