Few-Shot Class-Incremental Learning With Hyperbolic Prompts
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning models struggle to efficiently integrate and retain newly acquired knowledge across multiple training sessions with limited data, leading to substantial computational overhead and the risk of catastrophic forgetting, especially in environments with sparse data, and they do not effectively leverage complex hierarchical structures in image-text pair learning scenarios.
Innovation Solution
A dual-component system employing Session Specific Prompts (SSP) and hyperbolic distance metrics projects image and text embeddings into hyperbolic space, leveraging its expansive geometrical properties to improve class separability and reduce trainable parameters, while optimizing for incremental learning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional machine learning approaches are used to train models with large amounts of labeled data for each class, then classification accuracy is improved, but data requirements and training time increase significantly
Solution Approach 1:
The patent transforms the training approach by changing the parameter of data quantity from traditional large-scale labeled data to minimal few-shot examples. The CLIP model architecture enables this parameter change by using contrastive learning to create robust feature representations that generalize well from limited data, achieving high accuracy with only a few labeled examples per class.
Solution Approach 2:
The patent applies preliminary action by pre-training the CLIP model on large-scale image-text pairs before the few-shot classification task. This pre-training phase prepares the model with general visual and linguistic knowledge, enabling it to perform well on downstream tasks with minimal task-specific data.
2Adaptability or versatility
If models are retrained from scratch to learn new classes in class-incremental learning, then new class recognition is improved, but computational overhead and training time increase substantially
Solution Approach 1:
The patent implements dynamics by enabling the model to adapt its parameters dynamically during incremental learning sessions. The model freezes certain layers and fine-tunes others, allowing flexible adaptation to new classes without complete retraining. This dynamic parameter adjustment enables continuous learning across multiple sessions with reduced computational cost.
Solution Approach 2:
The patent applies segmentation by dividing the model into different trainable components during incremental learning. Specifically, it freezes the image encoder parameters while fine-tuning the text encoder and classification layers, segmenting the training process to reduce computational overhead while maintaining adaptability to new classes.
3Adaptability or versatility
If models continuously learn new classes over time, then adaptability to new classes is improved, but the risk of catastrophic forgetting of previously learned classes increases
Solution Approach 1:
The patent uses session-specific prompts as an intermediary mechanism to bridge new and old classes. These prompts act as a mediator that helps the model distinguish between different learning sessions and retain knowledge of previously learned classes while adapting to new ones, reducing catastrophic forgetting through structured parameter management.
Solution Approach 2:
The patent applies preliminary action by freezing the image encoder parameters before training on new classes. This preliminary freezing preserves the visual feature representations learned from previous classes, preventing catastrophic forgetting while allowing the text encoder and classification layers to adapt to new class information.
4Adaptability or versatility
If all prompt weights are unfrozen and retrained during incremental sessions, then flexibility in learning new classes is improved, but computational demand and risk of overfitting increase
Solution Approach 1:
The patent implements dynamics by dynamically adjusting which parameters are frozen and which are trainable during incremental sessions. Based on the learning stage and available data, the model flexibly freezes the image encoder while fine-tuning text prompts, optimizing the balance between adaptability and computational efficiency for each specific learning scenario.
Data Source
AI summary
Methods and systems for Few-Shot Class-Incremental Learning (FSCIL) that utilizes a combination of Session Specific Prompts (SSP) and hyperbolic distance metrics to enhance session-wise learning and representation of image-text pairings across differing classes. The methods and systems include a base training session where both text and image features are projected into hyperbolic space for accurate class pairing using a cross-entropy loss function. Subsequent incremental sessions incorporate previously learned SSPs to retain and augment the separability of classes while minimizing the trainable parameters. This enhances performance in image-text classification tasks by leveraging a minimalistic approach, achieving higher accuracy with fewer trainable parameters compared to traditional models.