Multi-Modality Classification Using Augmented Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning classifiers face accuracy issues due to inadequate training data, particularly when classifying objects across multiple modalities like images, sound, or touch, as conventional methods fail to effectively utilize additional information and blend disjoint content from different time slots.
Innovation Solution
A method and system utilizing Long Term Short Memory (LSTM) classifiers and Layer-wise Relevance Propagation (LRP) to generate patterns for augmented data, which includes features from multiple modalities, and re-classifying input images using Convolutional Neural Networks (CNNs) when initial classification accuracy is below a threshold, leveraging augmented data to improve classification precision.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional deep learning classifiers use basic training data for classification, then the classification process is simple and fast, but the classification accuracy is insufficient due to lack of additional information
Solution Approach 1:
The system performs preliminary actions by generating augmented training data with additional modality information (audio, text descriptions) before the actual classification task. This pre-enrichment of training data with multi-modal augmented information enables the classifier to achieve higher accuracy without increasing the complexity of the classification algorithm itself.
Solution Approach 2:
The system introduces an intermediary component that generates augmented data with additional modality information. This intermediary layer bridges the gap between basic training data and the requirements for accurate classification, allowing the classifier to access enriched information without direct modification of the classification algorithm.
2Measurement precision
If additional augmented data with multiple modalities is used to improve classification accuracy, then classification precision improves, but the system complexity and computational requirements increase
Solution Approach 1:
The system generates augmented data with additional modality information in advance, before the classification task. This preliminary enrichment allows the classifier to work with comprehensive data without requiring real-time processing of multiple modalities, thus improving precision while managing computational load.
Solution Approach 2:
The system creates copies of existing training data with augmented modality information. Instead of collecting new multi-modal data, the system generates synthetic augmented versions of existing data, preserving the original data volume while adding informational depth through multiple modalities.
3Measurement precision
If the system re-classifies images using LSTM classifier when CNN accuracy is below threshold, then classification accuracy improves, but the processing time and computational resources increase
Solution Approach 1:
The system performs preliminary classification using the faster CNN algorithm. Only when the preliminary classification accuracy falls below a threshold does the system engage the more computationally intensive LSTM classifier for re-classification. This staged approach ensures accurate results while minimizing unnecessary processing time for cases where CNN suffices.
Solution Approach 2:
The system applies the more resource-intensive LSTM classifier only partially - specifically, only when needed (when CNN accuracy is below threshold). This selective application of excessive action ensures that the additional computational resources are used only when necessary to achieve the required accuracy, avoiding waste in cases where simpler methods suffice.
Data Source
AI summary
A method and system of multi-modality classification using augmented data is disclosed. The method includes generating a pattern for each of a plurality of augmented data associated with each of a plurality of object classes, based on at least one modality associated with each of the plurality of objects classes using a Long Term Short Memory (LSTM) classifier and a Layer-wise Relevance Propagation (LRP). The method further includes classifying an input image into a first object class of the plurality of object classes based on one or more objects within the input image using a Convolution Neural Network (CNN). The method further includes re-classifying the input image into one of the first object class or a second object class of the plurality of object classes when the accuracy of classification by the CNN into the first object class is below a matching threshold.