Multi-scale Transformer for Native Resolution Image Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image analysis techniques, such as those using convolutional neural networks (CNNs), are constrained by fixed size input requirements, which can lead to distortions and changes in image quality when resizing or cropping is necessary.
Innovation Solution
A patch-based multi-scale Transformer approach that processes images at their native resolution, transforming them into a multi-scale representation to capture information across different granularities, while using hash-based 2D spatial embedding and scale embedding to handle images with arbitrary aspect ratios and resolutions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If images are resized or cropped to conform to fixed size input requirements, then the model can be trained with mini batches, but image composition is impacted, distortions or artifacts are introduced, and image quality changes
Solution Approach 1:
The image is divided into multiple patches of different sizes (e.g., 16x16, 32x32, 64x64 pixels) that are processed separately and then aggregated. This segmentation allows the model to handle images of arbitrary resolutions without resizing, as each patch is processed independently and then combined to form the final representation, thereby preserving image quality while enabling efficient batch processing.
Solution Approach 2:
The patent introduces a multi-scale dimension by processing the same image at multiple resolution levels simultaneously. Instead of forcing a single fixed input size, the model creates patches at different scales (fine-grained, medium-grained, coarse-grained) and processes them through separate Transformer encoders, then aggregates the results. This multi-scale approach resolves the contradiction by allowing variable input sizes while maintaining consistent processing capabilities.
2Manufacturing precision
If images are processed at native resolution without resizing, then image quality and composition are preserved, but fixed size input requirements cannot be met
Solution Approach 1:
The image is divided into multiple patches of different sizes (e.g., 16x16, 32x32, 64x64 pixels) that are processed separately and then aggregated. This segmentation allows the model to handle images of arbitrary resolutions without resizing, as each patch is processed independently and then combined to form the final representation, thereby preserving image quality while enabling efficient batch processing.
Solution Approach 2:
The model dynamically adapts to different input image sizes by creating a hierarchical patch structure where the number and size of patches vary based on the input resolution. The Transformer encoder processes a variable number of patches depending on the original image dimensions, allowing the system to maintain adaptability while processing images at their native resolution without fixed size constraints.
3Measurement precision
If multi-scale representation is used to capture information at different granularities, then image analysis effectiveness is improved, but computational complexity increases
Solution Approach 1:
The patent merges the outputs from multiple Transformer encoders that process patches at different scales. By aggregating the representations from fine-grained, medium-grained, and coarse-grained patches through a pooling mechanism, the model combines the strengths of each scale to improve image analysis accuracy while sharing computational resources across the different processing streams, thereby managing complexity.
Solution Approach 2:
The same Transformer encoder architecture is used across all scales, making the model universal and reducing the need for scale-specific components. This multi-functionality allows a single encoder design to handle patches of varying sizes effectively, simplifying the overall system while still capturing multi-scale information, thus improving accuracy without proportionally increasing complexity.
Data Source
AI summary
The technology employs a patch-based multi-scale Transformer (300) that is usable with various imaging applications. This avoids constraints on image fixed input size and predicts the quality effectively on a native resolution image. A native resolution image (304) is transformed into a multi-scale representation (302), enabling the Transformer's self-attention mechanism to capture information on both fine-grained detailed patches and coarse-grained global patches. Spatial embedding (316) is employed to map patch positions to a fixed grid, in which patch locations at each scale are hashed to the same grid. A separate scale embedding (318) is employed to distinguish patches coming from different scales in the multiscale representation. Self-attention (508) is performed to create a final image representation. In some instances, prior to performing self-attention, the system may prepend a learnable classification token (322) to the set of input tokens.