Image Classification With Modality Dropout for OCR-Free Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image classification systems struggle to accurately classify images for generating relevant image sets without relying on time-consuming and costly OCR processes, especially when images contain encoded text that is incidental or primary to the depicted objects.
Innovation Solution
A multi-branch image classification model is trained using modality dropout to process both visual and text data, followed by refining the model to remove the text branch, enabling efficient classification of images into specific classes without OCR, using a teacher-student model distillation for improved performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If OCR processes are used to classify images containing text, then classification accuracy is improved, but processing time and computational cost increase significantly
Solution Approach 1:
The patent extracts and removes the text processing branch from the multi-branch model after training, creating a streamlined model that achieves comparable accuracy without the computational overhead of OCR. The text branch is taken out because its function has been transferred to the visual branch during training through modality dropout and teacher-student distillation.
Solution Approach 2:
The patent replaces the mechanical OCR process with a learned visual classification approach. Instead of using traditional OCR mechanics to extract and process text, the model learns to directly classify images based on visual features, substituting a complex mechanical processing system with a more efficient neural network-based approach.
2Measurement precision
If a multi-branch model with text processing is used, then classification accuracy for images with text is improved, but device complexity increases
Solution Approach 1:
The text processing branch is extracted and removed from the final deployed model. During training, the multi-branch model learns to classify images using both visual and text modalities, but the text branch is subsequently removed, leaving a simpler single-branch model that maintains classification accuracy without the complexity of multiple branches.
Solution Approach 2:
The model structure parameter changes from multi-branch to single-branch after training. The training phase uses a multi-branch architecture to learn from both visual and text data, but the deployed model uses a simplified single-branch architecture, changing the structural parameter to reduce complexity while preserving performance.
3Measurement precision
If OCR-based classification systems are deployed, then image classification accuracy is maintained, but computational overhead and resource consumption increase
Solution Approach 1:
The computationally intensive OCR and text processing components are extracted and removed from the deployed system. The model is trained to perform text-based classification tasks through visual processing alone, eliminating the need for separate OCR pipelines and reducing overall computational overhead.
Solution Approach 2:
The model performs preliminary learning during training of how to classify images that would traditionally require text processing. Through modality dropout training, the visual branch learns to compensate for the absence of text information, performing the classification work in advance so that no additional OCR processing is needed at inference time.
Data Source
AI summary
Systems and methods are provided for classifying images associated with an item, and generating an image set for that item which includes image classifications determined to be helpful for the item type of the item. To classify images, an image classification model is generated and trained using two phases. The first phase uses intermediate model with text and visual processing to teach the model to recognize patterns created by text without requiring OCR at inference. The second phase uses visual processing to refine the model for use at inference. To generate an image set, image classifications helpful to an item type are identified, items are associated with item types, images are obtained for an item, the images are classified using the image classification model, missing image classifications set out in the preferred image set are identified, and a request or requests is generated for the missing image classifications.