Gated Linear Contextual Bandits for Low-Overhead Action Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing contextual bandit systems face challenges in accurately selecting actions while efficiently managing computational resources, particularly in scenarios where rewards are either binary or continuous, leading to suboptimal action selection and high computational overhead.
Innovation Solution
The use of gated linear networks (GLNs) to predict action probabilities and scores, allowing for efficient computation of pseudo-counts and reducing the need for backward passes, thereby enhancing action selection accuracy and resource efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional deep neural networks are used to generate action scores, then action selection accuracy can be improved, but computational overhead increases due to the required backward pass for weight updates
Solution Approach 1:
The patent extracts and removes the computationally intensive backward pass from the training process by using pre-trained gated linear networks. The action scores are generated through forward propagation only, eliminating the need for gradient computation and weight updates during action selection, thus reducing computational overhead while maintaining accuracy.
Solution Approach 2:
The gated linear networks are pre-trained offline using historical data and rewards. This preliminary training action allows the models to be ready for deployment with pre-learned weights, so that during actual action selection, only efficient forward propagation is needed without requiring additional backward passes or weight updates.
2Reliability
If exploration of the space of possible actions is enhanced, then action selection quality improves, but computational resources consumed increases
Solution Approach 1:
The patent uses multiple pre-trained gated linear networks, each corresponding to different actions. These networks are copied from the same architecture and pre-trained model, allowing parallel evaluation of multiple actions through efficient forward propagation without requiring additional training computations for each action evaluation.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer-readable storage media, for training a neural network to control a real-world agent interacting with a real-world environment to cause the real-world agent to perform a particular task. One of the methods includes training the neural network to determine first values of the parameters by optimizing a first task-specific objective that measures a performance of the policy neural network in controlling a simulated version of the real-world agent; obtaining real-world data generated from interactions of the real-world agent with the real-world environment; and training the neural network to determine trained values of the parameters from the first values of the parameters by jointly optimizing (i) a self-supervised objective that measures at least a performance of internal representations generated by the neural network on a self-supervised task performed on the real-world data and (ii) a second task-specific objective.