Reinforcement Learning Reward Modification via Local Explainability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Reinforcement learning models can be biased or maliciously trained due to features that are biased or intentionally misleading, which affects the decision-making process, especially in complex and dynamic environments.
Innovation Solution
The use of local explainability parameters to update reward values in reinforcement learning models by determining feature weights through local explainability operations, such as LIME, SHAP, or DeepLIFT, to identify and penalize biased or malicious features, thereby modifying the reward values and updating the model parameters to prevent reliance on such features.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If reinforcement learning models use features from complex and dynamic environments to make decisions, then the model's decision-making capability is improved, but the model becomes susceptible to biased or malicious features that can train the model incorrectly
Solution Approach 1:
The patent introduces local explainability parameters as an intermediary layer between the reinforcement learning model and the environment features. These parameters act as mediators that analyze and evaluate feature importance, allowing the model to benefit from complex environmental features while filtering out biased or malicious ones through the explainability analysis mechanism
Solution Approach 2:
The patent implements a feedback mechanism where local explainability parameters continuously evaluate feature importance and provide feedback to adjust reward values. This feedback loop enables the model to learn from its decisions while correcting for biased features, improving reliability without sacrificing decision-making capability in complex environments
2Productivity
If the reinforcement learning model relies on features that maximize cumulative reward, then the model's performance is improved, but the model may learn biased or malicious patterns from the training history
Solution Approach 1:
The patent changes the reward parameter by introducing local explainability weights that modify the original reward values. Instead of directly maximizing cumulative reward from potentially biased features, the model maximizes a adjusted reward that incorporates feature importance analysis, thereby maintaining productivity while improving training fairness
Solution Approach 2:
The patent converts the harmful effect of biased features into a beneficial learning opportunity. By using local explainability parameters to identify and penalize biased features through reward modification, the model transforms potential training corruption into a refinement process that improves both performance and fairness
3Reliability
If local explainability operations are performed to determine feature weights and modify reward values, then the model's reliability is improved, but the computational complexity and training time increase
Solution Approach 1:
The patent applies local explainability operations selectively rather than universally. By focusing explainability analysis on specific features or decision points that have the greatest impact on reliability, the system achieves improved model reliability while avoiding the computational overhead of applying explainability methods to every feature and decision
Data Source
AI summary
A method and related system of operations include providing a set of feature values to determine a first reward value to a prediction model configured with a set of model parameters and obtaining a set of feature weights for features of the set of feature values by performing a local explainability operation that comprises providing the prediction model with a set of test inputs to determine a set of feature weights. The method also includes selecting a subset of feature weights of the set of feature weights based on a feature subset of the features indicated by a policy parameter of the prediction model, determining a reward modification value based on the subset of feature weights, and determining a second reward value based on the first reward value and the reward modification value. The method also includes updating the set of model parameters based on the second reward value.