Transformer Image Token Masking for Reliable Classifier Testing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image classifiers rely on irrelevant image features for decision-making, and existing methods for testing decision limits of these classifiers often introduce artificial modifications that are easily detectable, undermining the reliability of the assessment.
Innovation Solution
A method using transformer networks to generate semantically modified image variations by masking specific tokens and preserving relevant information, ensuring that downstream classifiers react only to the absence of masked information, thus accurately determining the decision-making reliance on critical features.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If image pixels are replaced by minimum or maximum available values to remove distracting objects, then the distracting information is removed from the image, but the replaced pixels become visible as artificial spots that were not present in the original image
Solution Approach 1:
The patent extracts and removes distracting information from images by identifying regions containing distracting objects (e.g., neon signs, advertisements) and eliminating them through inpainting techniques. The system segments the image, identifies distracting regions based on classification, and removes only those specific areas while preserving the rest of the image content.
Solution Approach 2:
The patent applies asymmetric treatment to different regions of the image: distracting regions are removed through inpainting while non-distracting regions are preserved as-is. This asymmetric approach ensures that only problematic areas are modified, preventing artificial spots from appearing in the overall image while still achieving the goal of removing distracting information.
2Reliability
If image classifiers are tested with semantically modified variations to assess decision limits, then the reliability of the classification system can be evaluated, but the modifications may introduce artificial artifacts that compromise the validity of the assessment
Solution Approach 1:
The patent converts the potential harm of artificial modifications into a benefit by using controlled inpainting techniques. Instead of introducing visible artificial spots through simple pixel replacement, the system uses sophisticated inpainting algorithms that blend removed regions seamlessly with surrounding areas, transforming what would be harmful artifacts into beneficial, imperceptible modifications that validly test classifier robustness.
3Adaptability or versatility
If transformer networks are used to process images by dividing them into patches and converting to tokens, then the processing capability and semantic understanding are improved, but the complexity of the system increases
Solution Approach 1:
The patent applies segmentation by dividing images into patches and converting them to tokens for processing by the transformer network. This segmentation approach enables the system to handle complex semantic understanding tasks effectively while managing computational complexity through structured processing of smaller image segments rather than treating the entire image as a single unit.
Data Source
AI summary
A method for generating a semantically modified variation of an image. In the method: the image is divided into equally sized, non-overlapping patches; the patches are converted with a patch encoding function of a transformer network into a chain of tokens in a workspace; the tokens are grouped into preservation tokens, whose information is to be preserved in the variation, and masked tokens, whose information is to be masked in the variation; the preservation tokens are converted with an encoder of the transformer network into a chain of processed tokens; the chain is supplemented through application of an insertion operator that inserts these masked tokens at positions corresponding to the positions of the masked tokens in the original chain to form a chain that represents the sought variation; the chain is converted with a decoder of the transformer network into the sought variation.