Shortcut Debiasing for Online Continual Learning Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Online Continual Learning (OCL) models face challenges in suppressing shortcut bias, which leads to low transferability and high catastrophic forgetting, especially in environments with limited training data and computational resources.
Innovation Solution
The method involves fusing feature maps from different layers of the OCL model to generate a fused feature map, identifying shortcut features based on attention scores, and removing these features by applying a drop mask, all without requiring prior knowledge or auxiliary data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If DNNs learn shortcut features early in the learning process, then learning efficiency is improved, but transferability deteriorates and catastrophic forgetting increases
Solution Approach 1:
The patent extracts and removes shortcut features from the feature maps by identifying high-attention regions through attention maps and applying drop masks. This extraction process eliminates the harmful shortcut features (such as background cues or local patterns) while preserving genuine discriminative features, thereby resolving the contradiction between learning efficiency and transferability.
Solution Approach 2:
The patent implements a feedback mechanism where attention maps are generated from feature maps, used to identify shortcut features, and then feed back into the modification of feature maps through drop masks. This iterative feedback loop continuously refines the feature extraction process, ensuring that shortcut features are systematically removed while maintaining learning efficiency.
2Power
If DNNs rely on easy-to-learn peripheral features, then computational resources are reduced, but catastrophic forgetting increases
Solution Approach 1:
The patent extracts and removes shortcut features that cause catastrophic forgetting by generating attention maps to identify high-attention regions, then applying drop masks to eliminate these features. This extraction process reduces reliance on easy-to-learn but harmful features, thereby reducing catastrophic forgetting without significantly increasing computational resources.
Solution Approach 2:
The patent changes the parameter of feature map modification by introducing drop masks with varying drop intensities. This parameter change allows the model to dynamically adjust the degree of shortcut feature removal, optimizing the balance between computational resources and preventing catastrophic forgetting.
3Reliability
If existing debiasing methods use prior knowledge or auxiliary data, then shortcut bias is suppressed, but device complexity and memory requirements increase
Solution Approach 1:
The patent enables the model to self-identify and self-correct shortcut features by generating attention maps from its own feature maps. This self-service mechanism eliminates the need for external prior knowledge or auxiliary data, suppressing shortcut bias while maintaining low device complexity and memory requirements.
Solution Approach 2:
The patent makes the attention map generation and drop mask application process universal, applicable to any task without requiring task-specific prior knowledge or auxiliary data. This multi-functionality allows the same mechanism to suppress shortcut bias across different scenarios, reducing overall device complexity and memory budget requirements.
Data Source
AI summary
Disclosed is an operating method of an apparatus operated by at least one processor, the operating method including: fusing at least some feature maps generated by layers of a target model performing online continual learning to generate a fused feature map; identifying features with high attention in the fused feature map as shortcut features in a ratio based on a drop intensity; and removing the shortcut features from a target feature map output from a predetermined layer of the target model and input to a next layer.