AI Base Caller Compression Using Knowledge Distillation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep neural networks for base calling on mobile devices are resource-intensive and require significant computational power, making them unsuitable for deployment on resource-constrained platforms like FPGAs without compromising accuracy.
Innovation Solution
Implement knowledge distillation to transfer knowledge from a larger, teacher base caller to a smaller, student base caller by using hybrid ground truth data combining discrete and continuous valued labels, reducing the number of processing modules and parameters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a larger teacher base caller model is used, then base calling accuracy is improved, but computational resource requirements increase
Solution Approach 1:
The patent creates a student model that copies the essential knowledge and functionality of the teacher model through knowledge distillation. The student model replicates the base calling capabilities while using significantly fewer computational resources, effectively creating a lightweight copy that maintains accuracy without the resource burden of the original teacher model.
Solution Approach 2:
The patent transforms the model parameters by distilling knowledge from the teacher model's parameters into a condensed form suitable for the student model. This involves changing the parameter representation and organization to achieve comparable accuracy with reduced parameter count and computational complexity, enabling deployment on resource-constrained devices.
2Productivity
If model size is reduced for mobile deployment, then computational efficiency is improved, but base calling accuracy deteriorates
Solution Approach 1:
The patent introduces soft labels as an intermediary mechanism during knowledge distillation. These soft labels serve as a mediator that transfers nuanced information from the teacher model to the student model, enabling the smaller student model to learn complex decision boundaries and maintain high accuracy despite its reduced size.
Solution Approach 2:
The patent performs preliminary training of the teacher model on comprehensive datasets before distillation. This preliminary action ensures that the teacher model has already optimized its parameters and knowledge, allowing the subsequent distillation process to effectively transfer maximum knowledge to the student model in a single pass, achieving high accuracy efficiently.
3Device complexity
If knowledge distillation is applied, then model compression is achieved, but training complexity increases
Solution Approach 1:
The patent merges the training processes by computing both hard loss (from ground truth labels) and soft loss (from teacher model predictions) simultaneously during student model training. This combining of loss functions into a unified training objective simplifies the overall training procedure while achieving effective knowledge transfer, avoiding the need for separate complex training stages.
Data Source
AI summary
The technology disclosed compresses a larger, teacher base caller into a smaller, student base caller. The student base caller has fewer processing modules and parameters than the teacher base caller. The teacher base caller is trained using hard labels (e.g., one-hot encodings). The trained teacher base caller is used to generate soft labels as output probabilities during the inference phase. The soft labels are used to train the student base caller.