Fine-Grained Image Categorization via Alignment Module

Solution Overview

Problem

Deep convolutional neural networks (CNNs) face challenges in recognizing subtle differences between similar types of items due to their finite discriminatory capacity, which is often spent on recognizing items regardless of orientation, making it difficult to classify fine-grained categories effectively.

Innovation Solution

The implementation of a system that uses a convolutional neural network (CNN) to identify a coarse category, followed by an alignment module that rotates and scales the image to a standard orientation, and then a fine-grained categorization module to determine specific categories within the coarse category, utilizing techniques like grabCut masks and convex hulls to isolate the object and enhance discrimination.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If CNN is used for image classification, then classification capability is provided, but discriminatory capacity is limited and cannot recognize subtle differences between similar items

Engineering Contradiction:

Improveclassification precisionVSAvoiddiscriminatory capacity limitation

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The patent segments the classification task into two distinct stages: coarse-grained classification (broad categories) and fine-grained classification (subtle differences). This segmentation allows each stage to specialize - the first stage handles orientation and basic categorization, while the second stage focuses on subtle discriminative features, thereby overcoming the limited discriminatory capacity of a single CNN model.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent applies preliminary action by performing coarse-grained classification and orientation normalization before fine-grained classification. The system first identifies the broad category and standardizes object orientation, which prepares the data in a way that enables the subsequent fine-grained classifier to focus exclusively on subtle discriminative features without being distracted by orientation variations.

Inventive Principle:

Principle #10Preliminary action

2Adaptability or versatility

If CNN recognizes items without regard to orientation, then orientation invariance is achieved, but ability to recognize subtle differences is reduced

Engineering Contradiction:

Improveorientation invarianceVSAvoidfine-grained discrimination ability

Core Design Contradiction:

Adaptability or versatilityVSMeasurement precision

Solution Approach 1:

The patent segments the classification task into two distinct stages: coarse-grained classification (broad categories) and fine-grained classification (subtle differences). This segmentation allows each stage to specialize - the first stage handles orientation and basic categorization, while the second stage focuses on subtle discriminative features, thereby overcoming the limited discriminatory capacity of a single CNN model.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent applies preliminary action by performing coarse-grained classification and orientation normalization before fine-grained classification. The system first identifies the broad category and standardizes object orientation, which prepares the data in a way that enables the subsequent fine-grained classifier to focus exclusively on subtle discriminative features without being distracted by orientation variations.

Inventive Principle:

Principle #10Preliminary action

3Measurement precision

If fine-grained categorization is implemented, then classification accuracy for similar items improves, but computational resources required increase

Engineering Contradiction:

Improvefine-grained classification accuracyVSAvoidcomputational resource consumption

Core Design Contradiction:

Measurement precisionVSUse of energy by moving object

Solution Approach 1:

The patent applies preliminary action by performing coarse-grained classification and orientation normalization before fine-grained classification. The system first identifies the broad category and standardizes object orientation, which prepares the data in a way that enables the subsequent fine-grained classifier to focus exclusively on subtle discriminative features without being distracted by orientation variations.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent segments the classification task into two distinct stages: coarse-grained classification (broad categories) and fine-grained classification (subtle differences). This segmentation allows each stage to specialize - the first stage handles orientation and basic categorization, while the second stage focuses on subtle discriminative features, thereby overcoming the limited discriminatory capacity of a single CNN model.

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS10885394B2Fine-grained categorization

Publication Date: 2021.01.05 EBAY INC

AI summary

An image is passed through an image identifier to identify a coarse category for the image and a bounding box for a categorized object. A mask is used to identify the portion of the image that represents the object. Given the foreground mask, the convex hull of the mask is located and an aligned rectangle of minimum area that encloses the hull is fitted. The aligned bounding box is rotated and scaled, so that the foreground object is roughly moved to a standard orientation and size (referred to as calibrated). The calibrated image is used as an input to a fine-grained categorization module, which determines the fine category within the coarse category for the input image.