Contextual Bandit Training With Entropy-Guided Human Feedback
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing contextual bandit models rely heavily on implicit feedback, leading to biased and incomplete evaluations of user preferences and behaviors, which hinders accurate decision-making in dynamic and uncertain environments.
Innovation Solution
Incorporate human feedback into the training process of reinforcement learning models, specifically through entropy-based thresholds to determine when to seek expert input, using methods like action restriction and reward penalties, to enhance decision-making accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If implicit feedback (user clicks) is used for training contextual bandit models, then the training process is simple and automated, but the evaluation of user preferences becomes biased and incomplete
Solution Approach 1:
The patent introduces explicit human feedback (expert annotations) to supplement implicit feedback (user clicks). The system combines both feedback types, using explicit feedback to correct biases in implicit feedback and provide more accurate ground truth for training the contextual bandit model, thereby improving measurement precision while maintaining automation through hybrid feedback processing
Solution Approach 2:
The patent introduces an intermediary layer of expert annotation and entropy-based feedback processing between the implicit user clicks and the model training. This intermediary process transforms biased implicit feedback into more reliable training signals by incorporating expert knowledge and uncertainty measurements, resolving the contradiction between automation and accuracy
2Measurement precision
If human feedback is incorporated into the training process, then decision-making accuracy improves, but the training complexity and computational resources increase
Solution Approach 1:
The patent applies partial action by using entropy-based thresholds to selectively incorporate human feedback only when the model's uncertainty exceeds a certain level. This partial approach balances accuracy improvement with complexity management by avoiding unnecessary human feedback processing when the model is already confident, thus improving decision-making accuracy without proportionally increasing training complexity
Solution Approach 2:
The patent changes the parameter of feedback incorporation from binary (always/never) to continuous based on entropy levels. By adjusting the entropy threshold parameter, the system can control the degree of human feedback integration, allowing flexibility in balancing accuracy improvement against training complexity and computational resource consumption
3Productivity
If entropy-based thresholds are used to determine when to seek expert input, then feedback efficiency improves, but the computational overhead for calculating entropy increases
Solution Approach 1:
The patent performs preliminary entropy calculation during the training process to proactively identify when expert feedback is needed. By calculating entropy in advance and comparing against thresholds before seeking expert input, the system optimizes feedback efficiency by avoiding unnecessary expert queries while managing computational overhead through batch processing and efficient entropy estimation algorithms
Data Source
AI summary
One or more computer processors are used to train a reinforcement learning machine learning model, such as a contextual bandit machine learning model. A training dataset is inputted to the reinforcement learning machine learning model. The reinforcement learning machine learning model is trained based on the training dataset. During the training, an entropy of the reinforcement learning machine learning model is determined. Based on the feedback, feedback is generated. The reinforcement learning machine learning model is further trained based on the feedback.