Knowledge Distillation for Collaborative Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing recommendation systems in multiple-class environments face challenges in accurately recommending items with both high pre-use and post-use preferences due to limitations in knowledge distillation techniques, which struggle to transfer pre-use preferences effectively from teacher models to student models.
Innovation Solution
A method and system that utilize a plurality of teacher models to learn and transfer knowledge about pre-use and post-use preferences, with a knowledge transfer unit implementing this knowledge in a student model to recommend items with high pre-use and post-use preferences, employing a knowledge distillation technique to reduce model size and inference time while maintaining accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a large-sized neural model is used to capture nonlinear and complex patterns, then recommendation accuracy is improved, but inference time increases causing delays
Solution Approach 1:
The patent segments the recommendation task into two distinct preference types (pre-use and post-use) and employs separate teacher models for each, allowing the student model to learn from specialized knowledge sources rather than a monolithic large model, thus reducing inference time while maintaining accuracy
Solution Approach 2:
The patent creates a student model that copies essential knowledge from multiple teacher models through knowledge distillation, enabling the smaller student model to achieve comparable accuracy to large models while significantly reducing inference time and computational resources
2Device complexity
If knowledge distillation uses only one teacher model from feedback, then model size is reduced, but pre-use preference cannot be transferred to the student model
Solution Approach 1:
The patent merges multiple teacher models that specialize in different preference types (pre-use and post-use) into a unified knowledge distillation framework, allowing the student model to simultaneously learn from diverse knowledge sources without increasing its own size
Solution Approach 2:
The patent introduces a new dimension to knowledge distillation by distinguishing between pre-use and post-use preferences, creating multiple teacher models along this dimensional axis rather than relying on a single teacher model, thus enabling comprehensive preference transfer
3Measurement precision
If multiple teacher models are used to learn pre-use and post-use preferences, then recommendation accuracy for items with both preferences is improved, but system complexity increases
Solution Approach 1:
The patent extracts and separates the two distinct preference types (pre-use and post-use) from the unified recommendation task, assigning each to a specialized teacher model, which simplifies the knowledge distillation process despite having multiple teachers by giving each a focused function
Solution Approach 2:
The student model acts as an intermediary that consolidates knowledge from multiple specialized teacher models, managing the complexity by serving as a unified interface that learns from diverse sources without requiring the system to maintain all teacher models during inference
Data Source
AI summary
A recommendation method performed by a recommendation system in a multiple-class collaborative filtering environment includes learning pre-use preference and post-use preference by a plurality of teachers; selecting items to be transferred to a student model by predicting pre-use preference for items unobserved by a user based on the learned pre-use preference; determining a soft label based on post-use preference, which is predicted for the selected items based on the learned post-use preference; and transferring the determined soft label to the student model as distilled knowledge, and recommending, by the student model, items having high pre-use preference and high post-use preference based on the received distilled knowledge.