Neural Network Prompt Tuning for Recognition Consistency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural network training and processing methods require significant memory and computing resources, especially when updating or recognizing images and text, which can be inefficient and disrupt the training process.
Innovation Solution
A method for tuning neural network prompts using a single test image, where a processor augments the image, encodes text and images, filters noisy inputs, and updates prompts to improve recognition consistency across different views, leveraging techniques like Contrastive Language-Image Pre-Training (CLIP) and A Large-Scale Image and Noisy-Text Embedding (ALIGN) for zero-shot generalization without additional training data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If neural network training and processing methods are used to recognize images and text, then recognition capability is improved, but memory and computing resources are significantly consumed
Solution Approach 1:
The patent applies preliminary action by pre-training the neural network on large datasets before deployment. The network is pre-trained with diverse image-text pairs to learn robust features, so that during actual operation with a single test image, it can perform accurate recognition without requiring extensive computing resources for training at that moment.
Solution Approach 2:
The patent changes parameters by using prompt tuning instead of full network training. Instead of updating all network weights during inference, only the prompt parameters are adjusted based on the test image, significantly reducing computational burden while maintaining recognition capability.
2Reliability
If neural network updating is performed after training, then recognition accuracy is improved, but training stability is disrupted and additional resources are consumed
Solution Approach 1:
The patent segments the neural network into fixed pre-trained components and adjustable prompt components. The main network structure remains stable and frozen, while only the prompt parameters are updated during inference. This segmentation allows accuracy improvement without disrupting the stability of the trained network.
Solution Approach 2:
The patent introduces dynamics by making the prompt parameters adaptive during inference. The prompts are dynamically adjusted based on the specific test image to improve recognition accuracy, while the core network remains static. This creates a flexible updating mechanism that doesn't require full retraining.
3Adaptability or versatility
If multiple training datasets are used to improve generalization, then zero-shot capability is improved, but data processing time and resources increase
Solution Approach 1:
The patent applies preliminary action by incorporating diverse training data during the pre-training phase using methods like CLIP and ALIGN. These large-scale pre-training processes teach the network general concepts across multiple domains, enabling zero-shot transfer to new tasks without requiring additional processing time during deployment.
Data Source
AI summary
Apparatuses, systems, and techniques to perform neural networks. In at least one embodiment, a most consistent output of one or more pre-trained neural networks is to be selected. In at least one embodiment, a most consistent output of one or more pre-trained neural networks is to be selected based, at least in part, on a plurality of variances of one or more inputs to the one or more neural networks.