Logical Consistency Monitoring for Deep Neural Networks

The patent introduces concept models as intermediary components that bridge the DNN output and the plausibility checking mechanism. These concept models translate complex neural network outputs into interpretable concept-level representations, enabling simplified logical consistency checks without requiring complex attribution methods. The concept models act as mediators that convert high-dimensional DNN activations into meaningful concepts that can be evaluated against domain knowledge rules.

The patent segments the monitoring system into independent components: the existing DNN model, concept models that can be trained separately, and plausibility checking rules. This segmentation allows the concept models to be trained independently on concept labels while the original DNN remains unchanged. The modular architecture enables the system to monitor any pre-trained DNN by adding the concept model layer, thus maintaining adaptability while providing ease of operation through interpretable concept outputs.

The patent performs concept model training and concept-label association in advance during an offline training phase. The concept models are pre-trained to correctly associate concepts with vectors in the intermediate output of the DNN. This preliminary action allows the runtime system to simply evaluate pre-trained concept models against pre-defined logical rules, achieving real-time monitoring without the complexity of trainable predicates during execution.

The patent enables continuous plausibility checking throughout the entire operational lifecycle of the DNN. The concept models and logical rules are designed to work continuously during runtime, providing ongoing monitoring of output consistency. The system maintains continuous evaluation of whether DNN outputs conform to domain knowledge, ensuring reliability is maintained throughout operation rather than only during initial verification.