Compressing Image Segmentation Models for Mobile via Knowledge Distillation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing Segment Anything Model (SAM) is not compatible with mobile devices due to its heavy ViT-Huge based image encoder with over 600,000,000 parameters, making it resource-intensive and difficult to run on devices with limited resources.
Innovation Solution
A knowledge distillation method is employed to compress the image segmentation model by training a lightweight second image segmentation model with a ViT-Based image encoder and a lightweight image embedding layer, while maintaining performance by copying the lightweight mask decoder from the existing SAM.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the existing SAM uses a ViT-Huge based image encoder, then segmentation performance is improved, but device compatibility deteriorates due to over 600,000,000 parameters
Solution Approach 1:
The patent copies the mask decoder from the original SAM model to the compressed model, preserving the segmentation functionality while using a lighter image encoder. This allows the model to maintain performance on mobile devices without requiring the full ViT-Huge architecture
Solution Approach 2:
The patent replaces the expensive ViT-Huge image encoder with a more lightweight image encoder that has fewer parameters. This substitution reduces the computational burden and memory requirements, making the model suitable for mobile deployment while accepting some performance trade-off
2Measurement precision
If the image encoder is based on ViT-Huge with over 600,000,000 parameters, then segmentation accuracy is improved, but resource consumption increases making it unsuitable for mobile devices
Solution Approach 1:
The patent substitutes the resource-intensive ViT-Huge encoder with a lighter image encoder that consumes fewer computational resources. This replacement directly addresses the high energy and memory consumption while maintaining acceptable segmentation accuracy for mobile applications
3Adaptability or versatility
If the model is compressed to reduce parameters, then device compatibility is improved, but segmentation performance deteriorates
Solution Approach 1:
By copying the mask decoder from the original SAM, the patent preserves the core segmentation logic and functionality. This allows the compressed model to maintain segmentation performance despite using a lighter image encoder with fewer parameters
Solution Approach 2:
The patent segments the model into two parts: the image encoder and the mask decoder. By keeping the mask decoder intact and only compressing the image encoder, it achieves model compression while preserving the critical segmentation functionality in the mask decoder
Data Source
AI summary
A knowledge distillation method for compressing an image segmentation model that is performed in a computing device including one or more processors and a memory storing one or more programs executed by the one or more processors includes training a first image segmentation model including a first image encoder, a first image embedding layer, and a first mask decoder, constructing a second image segmentation model including a second image encoder, a second image embedding layer, and a second mask decoder according to preset constraints, and performing knowledge distillation for the second image embedding layer of the second image segmentation model based on the trained first image segmentation model.