3D Object Detection Model Weight Reduction via Knowledge Distillation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional knowledge distillation techniques are not effectively applied for weight reduction in three-dimensional (3D) object detection models, particularly due to the unique characteristics of point cloud data and the complex structure of 3D object detection models, which include multiple detection heads for predicting 3D coordinates, size, and position.
Innovation Solution
An electronic apparatus and method that employs a backbone network module for knowledge distillation by making feature maps of a teacher and student network identical through an auto-encoder, and a self-attention module to calculate relation-aware self-attention for improving detection information from multiple detection heads, thereby reducing model weights while maintaining high performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional knowledge distillation techniques are applied to 3D object detection models, then model performance may be improved through self-distillation or feature fusion, but model weight reduction is not achieved
Solution Approach 1:
The patent creates a student network that copies the architecture and learns from a teacher network through knowledge distillation. The student network replicates the teacher's detection heads and feature extraction capabilities but with optimized weights, achieving weight reduction while maintaining detection performance through learned knowledge transfer rather than direct copying of parameters.
Solution Approach 2:
The patent segments the knowledge distillation process into two distinct stages: backbone network distillation (feature map level) and detection head distillation (detection information level). This segmentation allows targeted optimization at each stage, enabling comprehensive weight reduction across the entire 3D object detection model while preserving performance.
2Measurement precision
If point cloud data is used for 3D object detection, then accurate 3D object detection is enabled, but the unique characteristics of point cloud data limit the application of conventional knowledge distillation techniques
Solution Approach 1:
The patent changes the parameter space for knowledge distillation by operating on feature maps and detection information specific to point cloud data rather than conventional image data. It adapts the distillation process to handle the unique characteristics of point cloud representations, including 3D spatial relationships and point cloud-specific feature extractions, making the technique versatile for 3D detection tasks.
3Loss of information
If a 3D object detection model includes multiple detection heads for predicting direction, size, and position, then comprehensive 3D object information is obtained, but the model structure becomes more complex
Solution Approach 1:
The patent extracts and distills detection information from each detection head separately during the knowledge distillation process. By taking out the detection information from multiple heads and processing it through the self-attention module, the system maintains comprehensive 3D object information while simplifying the student network structure through learned knowledge transfer, reducing overall model complexity.
4Weight of stationary object
If knowledge distillation is performed to reduce model weight, then model size is reduced, but detection accuracy may deteriorate without proper adaptation to 3D object detection characteristics
Solution Approach 1:
The patent performs preliminary action by first conducting knowledge distillation at the backbone network level to establish accurate feature map relationships before proceeding to detection head distillation. This preliminary feature-level knowledge transfer ensures that the student network learns correct 3D object features early in the process, preventing accuracy deterioration and enabling effective subsequent weight reduction at the detection head level.
Data Source
AI summary
Provided is an electronic apparatus for lightweight of a three dimensional (3D) object detection model based on knowledge distillation, the electronic device including: a backbone network module configured to perform knowledge distillation such that a first feature map of a teacher network and a second feature map of a student network extracted through input point cloud data are made identical to as each other; and a self-attention module configured to acquire a plurality of pieces of detection information from a plurality of detection heads for 3D object detection, respectively, using the first feature map and the second feature map, and perform knowledge distillation using a relation-aware self-attention calculated based on the acquired plurality of pieces of detection information.