Feature Vector Distillation Weighting for Simplified Student Models

The patent applies parameter changes by dynamically adjusting the weighting coefficients in the loss function during the distillation process. Specifically, it modifies the relative weights of different loss components (e.g., feature vector matching loss, intermediate layer loss) to optimize the balance between model simplification and feature vector equivalence. This allows the student model to achieve better feature representation despite significant architectural reduction.

The patent employs a composite loss function that combines multiple loss components with different weighting coefficients. This composite approach integrates feature vector matching, intermediate layer feature alignment, and standard classification loss, allowing the student model to learn from both the teacher model's features and the limited labeled data effectively, thereby maintaining feature vector similarity despite using fewer labeled images.

The patent implements a feedback mechanism through the distillation loss function that continuously compares the student model's feature vectors with the teacher model's feature vectors during training. The weighting coefficients are adjusted based on the magnitude of differences observed, providing feedback that guides the student model to converge toward producing more similar feature vectors while maintaining its simplified architecture for efficient inference.