Reinforcement learning method, reinforcement learning device, control program, and method for producing trained model

Abstract

[Problem] To provide a reinforcement learning method capable of increasing learning speed and increasing discounted cumulative reward. [Solution] A reinforcement learning method executed by a computer, said reinforcement learning method comprising: performing first reinforcement learning in which a fixed value is set for a discount rate and a policy is determined so as to increase a discounted cumulative reward, and acquiring information indicating the relationship between the number of trials and a parameter corresponding to the learning progress; approximating, on the basis of the information, the relationship between the number of trials and the parameter as a descriptive model; setting a reference model for the relationship between the number of trials and the parameter, wherein the reference model is represented by a transfer function of a first-order lag system; setting a discount rate corresponding to the number of trials so that the parameter output from the descriptive model approaches the parameter output from the reference model; and using a discount rate set corresponding to the number of trials and parameters to perform second reinforcement learning in which a policy is determined so as to increase the discounted cumulative reward.

Description

Reinforcement Learning Method, Reinforcement Learning Apparatus, Control Program, and Method for Manufacturing Trained Model

[0001] The present invention relates to a reinforcement learning method, a reinforcement learning apparatus, a control program, and a method for manufacturing a trained model.

[0002] Reinforcement learning is a technique in which an agent learns a policy for maximizing a discounted cumulative reward through interaction with an environment. In this method, a parameter called a discount rate, which is a value between 0 and 1, is set to adjust the importance of future rewards. By setting the discount rate to a value close to 1, there is a possibility of maximizing long-term rewards. However, it is empirically known that setting the discount rate to a value close to 1 causes problems such as an increase in learning time and non-convergence of learning. In response to this problem, Non-Patent Document 1 proposes a reinforcement learning method in which the discount rate is dynamically changed. Specifically, in the initial stage of learning, the discount rate is set relatively low, and the value of the discount rate is set to gradually increase according to the progress of learning.

[0003] Vincent Francois-Lavet, Raphael Fonteneau, Damien Ernst "How to Discount Deep Reinforcement Learning: Towards New Dynamic Strategies," arXiv:1512.02011

[0004] However, in order to increase the learning speed and the discounted cumulative reward, it is necessary to change the discount rate in a preferable manner.

[0005] In view of the above background, an aspect of the present invention aims to provide a reinforcement learning method, a reinforcement learning apparatus, a control program, and a method for manufacturing a trained model that can increase the learning speed and maximize the discounted cumulative reward.

[0006] To solve the above problems, one aspect of the present invention is a reinforcement learning method performed by a computer, which involves performing a first reinforcement learning in which a fixed value is set for the discount rate and a policy is determined to increase the cumulative discount reward; obtaining information showing the relationship between the number of trials and parameters corresponding to the learning progress; approximating the relationship between the number of trials and the parameters as a descriptive model based on the information; setting a normative model for the relationship between the number of trials and the parameters, where the normative model is represented by a transfer function of a first-order lag system; setting the discount rate corresponding to the number of trials so that the parameters output from the descriptive model approach the parameters output from the normative model; and performing a second reinforcement learning in which a policy is determined to increase the cumulative discount reward using the discount rate set in accordance with the number of trials and the parameters.

[0007] Another aspect of the present invention is a reinforcement learning device that performs first reinforcement learning to determine a policy such that the discounted cumulative reward increases while fixing the discount rate, acquires information showing the relationship between the number of trials and parameters corresponding to the learning progress, approximates the relationship between the number of trials and the parameters as a descriptive model based on the information, sets a normative model for the relationship between the number of trials and the parameters, where the normative model is represented by a transfer function of a first-order lag system, sets the discount rate corresponding to the number of trials such that the parameters output from the descriptive model approach the parameters output from the normative model, and performs second reinforcement learning to determine a policy such that the discounted cumulative reward increases using the discount rate set corresponding to the number of trials.

[0008] Another aspect of the present invention is a control program for causing a computer to execute a reinforcement learning method. The computer executes a first reinforcement learning to determine a policy so that the discounted cumulative reward increases with a fixed discount rate, obtains information indicating the relationship between the number of trials and a parameter corresponding to the learning progress, approximates the relationship between the parameter and the number of trials as a descriptive model based on the information, sets a normative model for the relationship between the parameter and the number of trials, where the normative model is represented by a transfer function of a first-order lag system, sets the discount rate corresponding to the number of trials so that the parameter output from the descriptive model approaches the parameter output from the normative model, and executes a second reinforcement learning to determine the policy so that the discounted cumulative reward increases using the discount rate set corresponding to the number of trials.

[0009] Another aspect of the present invention is a method for manufacturing a learned model executed by a computer. The method executes a first reinforcement learning to determine a policy so that the discounted cumulative reward increases with a fixed discount rate, obtains information indicating the relationship between the number of trials and a parameter corresponding to the learning progress, approximates the relationship between the parameter and the number of trials as a descriptive model based on the information, sets a normative model for the relationship between the parameter and the number of trials, where the normative model is represented by a transfer function of a first-order lag system, sets the discount rate corresponding to the number of trials so that the parameter output from the descriptive model approaches the parameter output from the normative model, and executes a second reinforcement learning to determine the policy so that the discounted cumulative reward increases using the discount rate set corresponding to the number of trials.

[0010] Another aspect of the present invention is a reinforcement learning method for a learned model implemented in a control device for a drive unit having an electric motor and a two-speed transmission, wherein the two-speed transmission includes a transmission input shaft connected to the output shaft of the electric motor, a sun gear coupled to the transmission input shaft, a carrier rotatably supporting a plurality of planetary gears that mesh with the sun gear, a ring gear that meshes with the plurality of planetary gears, a case that rotatably supports the ring gear in one direction via a one-way clutch, a transmission output shaft coupled to the carrier, and a clutch that connects and disconnects the transmission input shaft and the carrier, and the control device obtains a target motor torque of the electric motor by inputting state quantities of the two-speed transmission to the learned model, and based on the target motor torque Next, the electric motor is controlled, a fixed value is set for the discount rate, and a first reinforcement learning is performed to determine a policy that increases the cumulative discount reward. Information is obtained showing the relationship between the number of trials and parameters corresponding to the learning progress. Based on this information, the relationship between the number of trials and the parameters is approximated as a descriptive model, and a normative model is set for the relationship between the number of trials and the parameters. Here, the normative model is represented by the transfer function of a second-order lag system, and the discount rate corresponding to the number of trials is set so that the parameters output from the descriptive model approach the parameters output from the normative model. A second reinforcement learning is performed to determine a policy that increases the cumulative discount reward using the discount rate set in accordance with the number of trials and the parameters.

[0011] According to the above embodiments, it is possible to provide a reinforcement learning device, a reinforcement learning method, a control program, and a method for manufacturing a trained model that can increase the learning speed and increase the discounted cumulative reward.

[0012] Block diagram showing the configuration of the reinforcement learning device according to the embodiment Block diagram conceptually showing the functions of the processor Diagram conceptually showing the model structure of DQN Graph showing the relationship between the discount rate and the average cumulative reward Graph showing the relationship between the discount rate and the value of the loss function Graph of the average cumulative reward and discount rate with respect to the number of episodes Flow diagram showing the procedure of the reinforcement learning method Explanatory diagram approximating the value of the loss function with a first-order lag descriptive model Block diagram showing the normative model and a PID-controlled descriptive model Graph showing the relationship between the discount rate and the value of the loss function Explanatory diagram showing a model of the hill climbing problem Graph showing the loss function values ​​of the real data, descriptive model and normative model Graph showing (A) average cumulative reward, (B) value of the loss function, (C) discount rate Explanatory diagram of the drive unit and control device Graph showing clutch force Graph showing the rotational speed of the sun gear, carrier and ring gear of a two-speed transmission Graph showing the friction characteristics of the clutch Explanatory diagram of the DDPG algorithm Graph showing the loss function values ​​of the descriptive model and normative model Graph showing clutch force (A) average reward, (B) Critical Loss, (C) Discount rate graph. Graph showing (A) Rotational speed of each element of the two-speed transmission, (B) Rotational speed of each element of the clutch, (C) Motor torque, (D) Output torque of the example. Graph showing (A) Rotational speed of each element of the two-speed transmission, (B) Rotational speed of each element of the clutch, (C) Motor torque, (D) Output torque of the comparative example.

[0013] The following describes embodiments of a reinforcement learning method, a reinforcement learning apparatus, a control program, and a method for manufacturing a trained model, with reference to the drawings.

[0014] The reinforcement learning method is executed by the reinforcement learning device 1. As shown in Figure 1, the reinforcement learning device 1 is a computer having a processor 2 and a memory 3 that is communicatively connected to the processor 2. The processor 2 may include at least one of the following as its core: a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a RISC (Reduced Instruction Set Computer). The memory 3 stores the control program executed by the processor 2 and various data. The memory 3 may include at least one of volatile memory and non-volatile memory. The volatile memory may be, for example, DRAM (Dynamic Random Access Memory) or SRAM (Static Random Access Memory). The non-volatile memory may be an SSD (Solid State Drive), flash memory, magnetic disk storage device, or optical disk storage device. At least a part of the reinforcement learning device 1 may be implemented by hardware such as an LSI (Large Scale Integration), ASIC (application specific integrated circuit), FPGA (field-programmable gate array), or by a combination of software and hardware. The reinforcement learning device 1 may be composed of a single piece of hardware, or it may be composed of multiple pieces of hardware that can communicate with each other. Part of the reinforcement learning device 1 may be composed of an external server located outside.

[0015] Processor 2 executes the reinforcement learning method by running a control program stored in memory 3. The control program may be stored on a removable recordable medium such as a DVD or CD-ROM, and installed in memory 3 when the recordable medium is read by a reader. Alternatively, the control program may be downloaded to and installed in memory 3 via a communication network such as the Internet.

[0016] As shown in Figure 2, the processor 2 functions as agent 5 and environment 6 by executing a control program stored in memory 3. Agent 5 selects an action based on information from environment 6 and learns based on the reward obtained in accordance with that action. Agent 5 has a state observation unit 7 that receives state information provided by environment 6, an action decision unit 8 that determines the action that agent 5 should take based on the state information obtained from the state observation unit 7, and a learning unit 9 that optimizes the action decision unit 8 based on the experience data (state, action, reward, next state) obtained by agent 5 from its interaction with environment 6. Environment 6 feeds back the results of agent 5's actions to agent 5. Environment 6 has a state generation unit 11 that generates the state of environment 6 based on agent 5's actions, a reward generation unit 12 that generates a reward which is an evaluation value for the action taken by agent 5, and a transition model 13 that generates the next state based on agent 5's actions and returns it to agent 5.

[0017] The reinforcement learning method according to this embodiment can be applied to various known reinforcement learning algorithms. These reinforcement learning algorithms may include, for example, Q-learning, SARSA, DQN (Deep Q Network), Actor-Critic method, DDPG (Deep Deterministic Policy Gradient), etc. In this embodiment, DQN, a type of deep reinforcement learning, will be described as an example.

[0018] In the case of DQN, the state observation unit 7, the action decision unit 8, and the learning unit 9 are composed of a DQN model 15. As shown in Figure 3, the DQN model 15 has an input layer 16, an intermediate layer 17, and an output layer 18. DQN approximates the Q function using a deep neural network.

[0019] The input layer 16 has multiple nodes 16A. Each node 16A receives different state information. When creating a model to control a vehicle, the state information may include information about the vehicle's state and information about the vehicle's surrounding environment. Information about the vehicle's state may include, for example, the vehicle's position, speed, acceleration, steering angle, yaw rate, etc. Information about the vehicle's surrounding environment may include, for example, the position and speed of other vehicles, the position and speed of pedestrians, the position of obstacles such as guardrails, the weather, etc. The number of states may be set according to the task and environment to be controlled. The number of nodes 16A in the input layer 16 may correspond to the number of states. The input layer 16 passes the input information to the intermediate layer 17.

[0020] The intermediate layer 17 includes multiple layers. Each layer constituting the intermediate layer 17 has multiple nodes 17A. The intermediate layer 17 compresses the information input to the input layer 16 and extracts feature quantities of the information.

[0021] The output layer 18 has multiple nodes 18A, each node 18A outputting value information for each action. The value information is the expected value of the discounted cumulative reward obtained when a specific action is taken in a specific state, i.e., the state-action value function (Q-value). It is preferable that each node 18A in the output layer 18 corresponds to the number of actions.

[0022] In DQN-based learning, the update formula for the action-state value function is used, as shown in equation (1). Here, s is the current state, a is the current action, Q(s,a) is the current state-action-value function, α is the learning rate, r is the reward (immediate reward) for performing action a in state s, γ is the discount rate, and maxQ(s') is the state-action-value function when the action that maximizes value is selected in the next state s'.

[0023] The loss function (error) in updating the Q value can be expressed by equations (2) and (3), for example, by considering the loss as the mean squared error. Here, Li(θi) is the loss function, Q π (s, a; θ) is the predicted value (Q value output from the current model), Q μ'(s, a; θ) is the value at the time of sampling (training data), and E is the expected value. In DQN-based learning, the weight coefficients of the DQN model 15 are optimized using methods such as backpropagation and gradient descent so that the loss function Li(θi) approaches zero.

[0024] The DQN model 15 with optimized weight coefficients is used as a trained model for specific control. The trained DQN model 15 outputs the optimal action for a given state input.

[0025] In the learning described above, a fixed value is generally used for the discount rate γ. The discount rate γ is a coefficient used when calculating the present value of future rewards. The discount rate is introduced to adjust for the temporal importance of the rewards and is usually set in the range of 0 ≤ γ ≤ 1. Using the discount rate γ, the discount rate cumulative reward Rt at time t is expressed by equation (4). Furthermore, the average value of the cumulative rewards obtained within a certain period is called the average cumulative reward.

[0026] Figure 4 is a graph showing the relationship between the discount rate and the average cumulative reward. As shown in Figure 4, when the discount rate is close to 0 (low discount rate), agent 5 tends to choose actions that maximize the current reward, with little consideration given to future rewards. This results in shorter learning time, but also the problem that more optimal actions are not discovered, leading to a lower average cumulative reward.

[0027] When the discount rate is close to 1 (i.e., the discount rate is high), agent 5 places almost equal importance on future rewards as on present rewards and chooses actions based on long-term benefits. However, this can lead to problems such as longer learning times and failure to converge on learning.

[0028] Figure 5 is a graph showing the relationship between the discount rate and the value of the loss function. When a fixed value is used for the discount rate, the value of the loss function increases in proportion to the increase in the number of episodes during the initial stages of training, and then decreases. When the value of the loss function approaches zero, it can be understood that training is complete. From Figure 5, it can be seen that the higher the discount rate, the larger the maximum value of the loss function tends to be. Also, the lower the discount rate, the faster the value of the loss function decreases, and the faster training is completed. Thus, the value of the loss function changes depending on the discount rate.

[0029] In the reinforcement learning method according to this embodiment, the discount rate is dynamically set according to the number of trials, as shown in Figure 6. In the graphs of average cumulative reward and discount rate in Figure 6, the horizontal axis scale is the same. As shown in Figure 6, the initial value of the discount rate is set to a relatively low value, and in the early stages of learning, the discount rate increases in accordance with the increase in the number of trials. Then, in the latter half of learning, the discount rate is set to a value close to 1, such as 0.99 or higher. As a result, the average cumulative reward increases and the learning speed increases.

[0030] The reinforcement learning device 1 executes a reinforcement learning method based on the flowchart shown in Figure 7. First, the reinforcement learning device 1 performs a first reinforcement learning process in which it sets a fixed value for the discount rate and determines a policy to increase the cumulative discount reward, and obtains information showing the relationship between the number of trials and the parameters corresponding to the learning progress (ST1).

[0031] Here, the number of trials may be either the number of episodes or the number of steps. Parameters corresponding to the learning progress may be, for example, the loss function and the TD error. Furthermore, it is preferable that parameters corresponding to the learning progress increase in proportion to the increase in the number of trials in the early stages of learning, and decrease in proportion to the increase in the number of trials in the later stages of learning. Parameters corresponding to the learning progress may also be the average cumulative reward, the state value function, and the state action value function.

[0032] In this embodiment, the parameter corresponding to the progress of learning is the value of the loss function. When the discount rate is fixed, the loss function increases in proportion to the increase in the number of trials in the early stages of learning, and then decreases in proportion to the increase in the number of trials in the later stages of learning, converging to a value close to zero (see Figure 5).

[0033] The fixed value set for the discount rate in the first reinforcement learning process may be a relatively low value in order to speed up the learning process. The value set for the discount rate may be, for example, 0.8 or less. The first reinforcement learning process should ideally be carried out at least until the value of the loss function is maximized. Alternatively, the first reinforcement learning process may be carried out until the value of the loss function converges.

[0034] Next, the reinforcement learning device 1 approximates the relationship between the number of trials and the parameters corresponding to the learning progress, based on the information obtained in step ST1, as a descriptive model P (ST2). Here, the descriptive model P may be represented, for example, by the transfer function P(s) of a first-order lag system, as shown in Figure 8. The transfer function P(s) is expressed by equation (5). Here, k is the gain, τ is the time constant, and s is the Laplace operator. The reinforcement learning device 1 determines the values ​​of the gain k and the time constant τ by fitting the information representing the relationship between the number of trials and the value of the loss function obtained in step ST1 to equation (5). The fitting is preferably performed in the early stages of learning, where the value of the loss function increases with increasing number of trials.

[0035] Next, the reinforcement learning device 1 sets a normative model M for the relationship between the number of trials and the parameters (ST3). Here, the normative model M is preferably expressed by the transfer function M(s) of a first-order lag system, as shown in equation (6). Here is the first K ｍ is the gain of the normative model M, T ｍ is the time constant of the normative model M. The gain K of the normative model M. ｍ and time constant T ｍ It is preferable that this be a pre-set value. Also, the gain K of the normative model M. ｍ and time constant T ｍmay be set based on the values of the gain k and the time constant τ of the descriptive model P. For example, the gain K of the normative model M ｍ may be set by multiplying the gain k of the descriptive model P by a predetermined coefficient. Similarly, the time constant T of the normative model M ｍ may be set by multiplying the time constant τ of the descriptive model P by a predetermined coefficient for k. Further, the reinforcement learning model may receive numerical input from a user and set the input numerical value to the gain K of the normative model M ｍ and the time constant T ｍ . The normative model M may be one obtained by changing the coefficients of the descriptive model P.

[0036] Next, the reinforcement learning device 1 sets a discount rate corresponding to the number of trial runs so that the parameter output from the descriptive model P approaches the parameter output from the normative model M (ST4). At this time, the reinforcement learning device 1 may set a discount rate corresponding to the number of trial runs by executing model matching (partial model matching method) based on PID control so that the parameter output from the descriptive model P approaches the parameter output from the normative model M. Specifically, as shown in FIG. 9, consider a system (referred to as model G) in which the output value to the descriptive model P is PID-controlled by the controller C and performs the same output as the normative model M.

[0037] The transfer function C(s) of the controller C is represented by Equation (7). Here, k ｐ is the proportional gain, k ｉ is the integral gain, k ｄ is the derivative gain.

[0038] Assuming that a system in which the input value of the descriptive model P is PID-controlled by the controller C is model G, the transfer function G ｙｒ (s) of model G is expressed as in Equation (8) using the transfer functions P(s) and C(s).

[0039] From Equation (8), the inverse transfer function 1 / G ｙｒ (s) of the transfer function G ｙｒ (= G ｙｒ (s) －１) can be expressed as shown in equation (9).

[0040] From equation (6), the inverse transfer function of the reference model M's transfer function M(s) is 1 / M(s) (= M(s) －１ ) can be expressed as shown in equation (10).

[0041] When equations (9) and (10) are expanded using the Maclaurin series, we obtain equations (11) and (12).

[0042] When equation (11) and equation (12) are matched, the coefficients of the linear function of s become equal, and the coefficients of the quadratic function of s become equal. Therefore, equations (13) and (14) hold. By rearranging equations (13) and (14), we obtain equations (15) and (16). As mentioned above, the time constant Tm of the normative model M, the time constant τ of the descriptive model P, and the gain k are known values, so the proportional gain k ｐ and integral gain k ｉ The value of is determined. Also, in first-order lag system model matching, the differential gain k ｄ is 0 (k ｄ = 0).

[0043] Proportional gain k ｐ and integral gain k ｉ Based on the value of , a discount rate γ is calculated to bring the actual loss function value closer to the normative model's loss function value. The actual loss function value obtained in a certain number of trials (number of steps or episodes) and the loss function value of the normative model at convergence are used as input values, and a proportional gain k ｐ and integral gain k ｉWhen input is given to a function based on this, the discount rate γ is output. The function can be expressed as follows, for example: Error × Proportional Gain + Cumulative Error × Integral Gain = Discount Rate γ Here, the error is the difference between the actual loss function value and the loss function value at convergence of the normative model. The cumulative error value is the cumulative error value in each trial so far. This function is based on a known formula used in submodel matching methods for first-order lag systems. This allows the discount rate γ to be changed for each trial.

[0044] The reinforcement learning device 1 performs a second reinforcement learning (ST5) in which it determines the policy to increase the discounted cumulative reward using a discount rate set in accordance with the number of trials and the value of the loss function (parameter). As shown in Figure 10, when a fixed value is used for the discount rate, the loss function decreases in the later stages of learning and converges to a value close to 0. On the other hand, in the reinforcement learning method according to this embodiment, the discount rate is set so that the value of the loss function follows a normative model M in which the value of the loss function is maintained constant without decreasing even in the later stages of learning. As a result, the discount rate is set to increase in the later stages of learning so that the value of the loss function does not decrease. In the later stages of learning, the average cumulative reward increases because the value of the loss function is maintained at a relatively high value.

[0045] The descriptive model P(s) used to approximate the relationship between the number of trials and the parameters may be a second-order lag system, as shown by equation (17). In this case, the normative model M(s) is shown by equation (18). Here, K is the gain, ζ is the damping ratio, ω is the natural angular frequency, and s is the Laplace operator. The subscript m indicates the parameters of the normative model.

[0046] Transfer function G ｙｒ (s) is expressed by equation (19).

[0047] 1 / G ｙｒ The Maclaurin series expansions of (s) and 1 / M(s) are given by equations (20) and (21).

[0048] Proportional gain kｐ , integral gain k ｉ , and differential gain k ｄ is 1 / G ｙｒ This is determined by matching the linear, quadratic, and cubic terms of the Laplace operator s in the Maclaurin series expansion of (s) and 1 / M(s). Equations (22), (23), and (24) represent comparisons of the linear, quadratic, and cubic terms, respectively. According to equations (22), (23), and (24), the proportional gain k ｐ , integral gain k ｉ , and differential gain k ｄ This is determined as shown in equations (25), (26), and (27).

[0049] Next, we will explain an example of applying the above reinforcement learning method to the well-known mountain climbing problem (Mountain Car). As shown in Figure 11, the mountain climbing problem involves a car located in a valley, with the goal of reaching the top of the hill on the right. The learning objective is to find the optimal action to reach the goal in the fewest steps. Because the car's propulsion is weak, simply accelerating will not allow it to climb the hill. Therefore, the car needs to move to the left, gain momentum, and then climb the hill on the right.

[0050] The mountain climbing problem is set up so that the state includes the car's position and speed. Agent 5 can choose from the following actions: accelerate to the left, accelerate to the right, or do nothing. The rewards include a negative reward given for each step and a positive reward given when the car reaches the goal. The termination conditions are that the car reaches the goal, or that the goal is not reached even after the maximum number of steps has elapsed. The learning algorithm used was DDPG. DDPG has four networks: Critic, Actor, Target Critic, and Target Actor. The loss function of DDPG is given by equation (17). Here, L is the value of the loss function, B is the set of sampled experiences, and s ｉ Current state, s ｉ＋１The following state, a ｉ Actions in the current state, r ｉ θ is the reward, γ is the discount rate, and θ is the discount rate. μ＾ (^ is placed above μ) is a parameter of the target actor, θ Ｑ＾ (^ is placed above Q) is the parameter of the target critic, θ Ｑ μ(s) is the parameter of the critic. ｉ＋１ |θ μ＾ ) is the action chosen by the target actor from the following states, Q(s ｉ＋１ ,μ(s ｉ＋１ |θ μ＾ )|θ Ｑ＾ ) is the Q value (value function) of the next state output by the target critic, Q(s ｉ , a ｉ |θ Ｑ This is the Q-value (value function) of the current state output by the Critic. The Critic tries to minimize the value of the loss function.

[0051] First, the reinforcement learning device 1 performs a first reinforcement learning process in which it sets a fixed value for the discount rate and determines a policy to increase the cumulative discount reward, and obtains a graph R1 showing the relationship between the number of trials and the value of the loss function (ST1), as shown in Figure 12. In this example, the fixed value is 0.999. The number of trials is expressed as the number of steps.

[0052] Next, the reinforcement learning device 1 approximates graph R1, which shows the relationship between the number of steps and the value of the loss function in Figure 12, with the transfer function P(s) of a first-order lag system (see graph P in Figure 12, equation 5, ST2). From the values ​​of graph P in Figure 12, the gain k and time constant τ are determined. In this example, the gain k is 0.173 and the time constant τ is 171.344.

[0053] Next, the reinforcement learning device 1 sets the transfer function M(s) of the normative model M for the relationship between the value of the loss relation and the number of steps (see Equation 6, ST3). Here, the gain K of the normative model M is set. ｍ The gain k of the descriptive model P may be multiplied by a predetermined coefficient. Also, the time constant T of the normative model M. ｍ This may be obtained by multiplying the time constant τ of the descriptive model P by a predetermined coefficient. In this example, Kｍ = 0.7 × k, T ｍ = 1.4 × τ. The graph of the normative model is represented as graph M in Figure 12.

[0054] Next, the reinforcement learning device 1 sets a discount rate corresponding to the number of steps (ST4) so ​​that the value of the loss function output from the descriptive model P approaches the value of the loss function output from the normative model M. The inverse transfer function G is based on equations (7) to (16) described above. ｙｒ (s) －１ The inverse transfer function G is determined. In this example, ｙｒ (s) －１ proportional gain k ｐ The integral gain is 4.13, k ｉ This becomes 0.024.

[0055] Next, the reinforcement learning device 1 performs a second reinforcement learning (ST5) in which it determines a policy to increase the discounted cumulative reward using a discount rate set in accordance with the number of steps and the value of the loss function of the normative model M. Here, the discount rate is the inverse transfer function G described above. ｙｒ (s) －１ This is obtained by inputting the number of steps of the normative model M and the value of the loss function corresponding to the number of steps.

[0056] Figures 13(A) to (C) show graph G when a dynamic discount rate is applied using the reinforcement learning method according to this embodiment, graph R1 when the discount rate is fixed at 0.999, and graph R2 when the discount rate is fixed at 0.9. As shown in Figure 13(C), in the reinforcement learning method according to this embodiment, the initial value of the discount rate is set to 0.5 as an example. In the reinforcement learning method according to this embodiment, by setting the discount rate relatively low, learning becomes faster and the average cumulative reward increases relatively early. When the value of the loss function is lower than the value of the loss function determined by the normative model M, the discount rate increases. As a result, in the early stages of learning, the discount rate increases from 0.5 to approximately 0.999 as the number of steps increases. As shown in Figures 13(B) and (C), when the value of the loss function becomes larger than the value of the loss function determined by the normative model M, the discount rate decreases.

[0057] As shown in Figure 13(A), in the reinforcement learning method according to this embodiment, learning proceeds with roughly the same efficiency as when the discount rate is set to a relatively low fixed value (R2). Furthermore, in the reinforcement learning method according to this embodiment, an average cumulative reward roughly the same as when the discount rate is set to a relatively close fixed value (R2) can be obtained in the later stages of learning. In addition, since fluctuations in the value of the loss function are suppressed, the average cumulative reward also shows a stable value.

[0058] Next, an embodiment in which the above-described reinforcement learning method is applied to the learning of a trained model implemented in the control device 32 of the drive unit 31 will be described. The drive unit 31 is used as a power source for an electric vehicle.

[0059] As shown in Figure 14, the drive unit 31 has an electric motor 34 and a two-speed transmission 35. The electric motor 34 may be a permanent magnet synchronous motor or an induction motor. The two-speed transmission 35 has a transmission input shaft 38 connected to a motor output shaft 37 which is the output shaft of the electric motor 34, a sun gear 39 coupled to the transmission input shaft 38, a carrier 42 that rotatably supports a plurality of planetary gears 41 that mesh with the sun gear 39, a ring gear 43 that meshes with the plurality of planetary gears 41, a case 46 that rotatably supports the ring gear 43 in one direction via a one-way clutch 44, a transmission output shaft 47 coupled to the carrier 42, a clutch 48 that connects and disconnects the transmission input shaft 38 and the carrier 42, and a control device 32 that controls the electric motor 34 and the clutch 48. The two-speed transmission 35 includes a so-called planetary gear mechanism.

[0060] The transmission input shaft 38 is rotatably supported in the case 46. The outer end of the transmission input shaft 38 protrudes from the case 46 and is coaxially connected to the motor output shaft 37. The transmission input shaft 38 rotates integrally with the motor output shaft 37. The sun gear 39 is a spur gear. The sun gear 39 is coupled to the inner end of the transmission input shaft 38 and rotates integrally with the transmission input shaft 38. The carrier 42 is coaxially positioned with the transmission input shaft 38 and is rotatably supported on the transmission input shaft 38. The carrier 42 may be rotatably supported in the case 46.

[0061] The one-way clutch 44 allows the ring gear 43 to rotate in the forward direction relative to the case 46, and prohibits rotation in the reverse direction. That is, the ring gear 43 can only rotate in the forward direction relative to the case 46. Here, the forward direction of rotation of the ring gear 43 coincides with the forward direction of rotation of the electric motor 34.

[0062] The clutch 48 has a drive plate 48A provided on the transmission input shaft 38 and a driven plate 48B provided on the carrier 42. One of the drive plate 48A and the driven plate 48B is formed to be displaceable in the axial direction. That is, the drive plate 48A and the driven plate 48B can move toward and away from each other. By the contact between the drive plate 48A and the driven plate 48B, the rotational force of the transmission input shaft 38 is transmitted to the carrier 42 via the drive plate 48A and the driven plate 48B. In this embodiment, the drive plate 48A is provided to be displaceable relative to the driven plate 48B. The clutch 48 has a clutch drive unit 48C that presses the drive plate 48A against the driven plate 48B. The clutch drive unit 48C may be a hydraulic device or an electric motor. The force that the clutch drive unit 48C applies to the drive plate 48A is called the clutch force. The clutch force changes over time, for example, as shown in Figure 15. When the clutch 48 begins to engage, the clutch force increases proportionally to time after a predetermined period of time has elapsed, and then becomes constant. In this embodiment, the clutch 48 is engaged in 0.5 seconds.

[0063] The transmission output shaft 47 is arranged coaxially with the transmission input shaft 38. The transmission output shaft 47 protrudes outward from the carrier 42 to the case 46. Preferably, the transmission output shaft 47 is rotatably supported by the case 46.

[0064] The two-speed transmission 35 switches between first and second gear by engaging and disengaging the clutch 48. In first gear, the clutch 48 is disengaged, and the transmission input shaft 38 and the carrier 42 rotate independently of each other. At this time, the rotation of the ring gear 43 is restricted by the one-way clutch 44 and does not rotate relative to the case 46. As a result, the speed ratio (output / input) for first gear is Zs / (Zs+Zr), where Zs is the number of teeth on the sun gear 39 and Zr is the number of teeth on the ring gear 43.

[0065] When the clutch 48 engages, the two-speed transmission 35 changes from first gear to second gear. In second gear, because the clutch 48 is engaged, the sun gear 39 and carrier 42 rotate together. As a result, the multiple planetary gears 41 do not rotate on their own. Consequently, the sun gear 39, carrier 42, and ring gear 43 rotate together. This results in a speed ratio (output / input) of 1 in second gear.

[0066] Figure 16 shows the rotational speeds of the sun gear 39, carrier 42, and ring gear 43 when the two-speed transmission 35 changes from first gear to second gear.

[0067] The control device 32 is a computer having a processor and memory. The configuration of the computer constituting the control device 32 may be the same as that of the computer constituting the reinforcement learning device 1. The control device 32 functions as a control unit 51 and a learning unit 52 when the processor executes a program. The control unit 51 controls the electric motor 34 and the clutch 48. The control unit 51 controls the clutch 48 at a predetermined timing to shift the two-speed transmission 35 from the first speed to the second speed. In detail, the control unit 51 controls the clutch drive unit 48C of the clutch 48 to generate clutch force (see Figure 15). At the same time, the control unit 51 controls the electric motor 34 to suppress shifting shock and clutch judder. That is, the control unit 51 controls the torque of the electric motor 34 to achieve seamless shifting of the two-speed transmission 35. At this time, the control unit 51 controls the torque of the electric motor 34 taking into account the aging deterioration of the clutch 48.

[0068] It has been shown that clutch judder occurs when the μ-ν characteristic of the friction surface shows a negative slope with respect to the sliding velocity. The coefficient of friction is modeled by the following equations (29), (30), and (31). Here, μ ０ μ represents the static friction coefficient, μ represents the sliding friction coefficient, and ν represents the relative velocity between the drive plate 48A and the driven plate 48B. Also, k １ = 0.85, k ２ = 0.08, k ３ =0.0004 is a hyperparameter, and i indicates the number of clutch engagements of the clutch 48. Static friction coefficient μ ０ By adjusting the value of the sliding friction coefficient μ, it is possible to reproduce various friction characteristics. Figure 17 shows the friction coefficient generated based on equation (29). From equations (30) and (31), the static friction coefficient μ ０ Furthermore, it can be seen that the sliding friction coefficient μ changes depending on the number of times the clutch 48 is engaged i. That is, the static friction coefficient μ changes due to the aging deterioration of the clutch 48. ０ And the coefficient of sliding friction μ changes.

[0069] The control unit 51 obtains the target motor torque of the electric motor 34 by inputting the state variables of the two-speed transmission 35 into a learned model, and controls the electric motor 34 based on the target motor torque. The state variables of the two-speed transmission 35 may include at least one of the following: the output torque of the two-speed transmission 35 (transmission output torque), clutch force, clutch engagement signal, rotational speed of the sun gear 39, rotational speed of the carrier 42, rotational speed of the ring gear 43, rotational speed of the drive plate 48A of the clutch 48, and rotational speed of the driven plate 48B of the clutch 48. In this embodiment, the state variables of the two-speed transmission 35 include all of the following: transmission output torque, clutch force, clutch engagement signal, rotational speed of the sun gear 39, rotational speed of the carrier 42, rotational speed of the ring gear 43, rotational speed of the drive plate 48A, and rotational speed of the driven plate 48B. These state variables may be normalized to values ​​between -1 and 1.

[0070] The transmission output torque, clutch force, rotational speed of the sun gear 39, the carrier 42, the ring gear 43, the drive plate 48A, and the driven plate 48B are preferably obtained by a sensor 55 provided on the two-speed transmission 35. The sensor 55 may include a rotational speed sensor such as a rotary encoder, a torque sensor, and a load sensor. The sun gear 39 and the drive plate 48A are provided on the transmission input shaft 38, but twisting occurs in the transmission input shaft 38, which may cause a difference in the rotational speed of the sun gear 39 and the drive plate 48A. Similarly, a difference may occur in the rotational speed of the carrier 42 and the driven plate 48B.

[0071] The clutch engagement signal represents the clutch 48 as 1 when it is engaged and as 0 when it is disengaged. The clutch engagement signal is preferably generated by the control unit 51. For example, the control unit 51 may generate a clutch engagement signal of 1 when the absolute value of the difference between the rotational speed of the drive plate 48A and the rotational speed of the driven plate 48B is less than or equal to a predetermined determination value, indicating that the clutch 48 is engaged, and generate a clutch engagement signal of 0 when the absolute value of the difference between the rotational speed of the drive plate 48A and the rotational speed of the driven plate 48B is greater than the determination value, indicating that the clutch 48 is disengaged.

[0072] The control unit 51 may use the graph in Figure 15 to obtain the clutch force based on the elapsed time since the clutch 48 started to engage.

[0073] The control unit 51 inputs the state variables of the two-speed transmission 35 into a learned model at predetermined time intervals to obtain the target torque of the electric motor 34. The time interval may be, for example, 0.005 [s]. The control unit 51 controls the electric motor 34 so that the output torque of the electric motor 34 approaches the target torque. The control unit 51 may, for example, set the current value supplied to the electric motor 34 based on the target torque of the electric motor 34.

[0074] The trained model is trained by the learning unit 52 based on the reinforcement learning method described above. The learning unit 52 acquires the state quantity of the two-speed transmission 35 as the state and the target torque of the electric motor 34 as the action. Then, the learning unit 52 determines the reward for each state based on the state quantity of the two-speed transmission 35. The reward may be set based on, for example, the difference between the torque of the transmission output shaft 47 (output torque) and its target torque, or the connection state of the clutch 48 (connection signal). The learning unit 52 stores the acquired data in memory as a set of "state, action, reward, next state". The learning unit 52 extracts samples from memory and updates the actor network and the critic network based on the DDPG algorithm.

[0075] In the initial stages of learning, a simulator that simulates the drive unit 31 may be used instead of the actual drive unit 31.

[0076] The details of the reinforcement learning method are described below. Reinforcement learning uses the DDPG algorithm. The learning process is described in Algorithm 1 in Figure 18. Lines 20-24 implement the updating of the discount rate using PID control. Lines 20-22 calculate the adjustment amount based on the loss function, and line 23 applies it to the discount rate. In subsequent steps, line 16 updates the critic and line 17 updates the actor.

[0077] In this embodiment, the gear ratio for the first gear is set to 4.45, and the target output torque is set to 30 [Nm]. The clutch force is set to reach its maximum value 0.35 [s] after the start of engagement, as shown in Figure 15.

[0078] The input layer consists of state variables for the two-speed transmission 35. In this embodiment, the state variables for the two-speed transmission 35 include the transmission output torque, clutch force, clutch engagement signal, rotational speed of the sun gear 39, rotational speed of the carrier 42, rotational speed of the ring gear 43, rotational speed of the drive plate 48A, and rotational speed of the driven plate 48B. Each state variable is normalized to a value between -1 and 1.

[0079] The hyperparameters for the DDPG algorithm are shown in Table 1.

[0080] The total simulation time (duration per episode) is set to 0.5 seconds, and the time step is set to 0.005 seconds. One episode is defined as repeating the simulation until the maximum number of steps (100 steps) is reached. Training was considered complete when the number of episodes reached 8000. The discount rate is updated every 100 steps. The actor network is updated once every two steps.

[0081] The reward is expressed by the following equations (32) to (35). Here, we ｔ r1 is the difference between the transmission output torque and its target value, and C is the clutch engagement signal. Equation (33) is a reward function based on the difference between the transmission output torque and its target value, where the smaller the difference, the greater the reward. Equation (34) is a reward function based on the difference between the transmission output torque and its target value, where the reward decreases in proportion to the magnitude of the difference when the difference between the transmission output torque and its target value exceeds a threshold. The rewards in equations (33) and (34) are the rewards given at each step. Equation (35) is a reward based on the clutch engagement state, and is given only at the end of an episode. If the clutch is engaged at the end of an episode (C=1), the reward is 0. On the other hand, if the clutch is not engaged (C=0), a penalty of -20 is given, prompting the agent to engage the clutch before the end of one episode. Equation (32) shows that the reward R is the sum of the individual rewards r1 to r3.

[0082] As described above, first, the value of the loss function obtained when training with a fixed discount rate is approximated as a quadratic system. Figure 19 shows the value of the loss function during training and the result of approximating it with a quadratic system. The parameters of the approximated quadratic system model are K = 0.184, ω = 112.09, and ζ = 0.135.

[0083] Next, a normative model is determined. The normative model is designed to have a low gain, a damping ratio greater than 1 to suppress overshoot, and a high natural angular frequency to avoid an excessively slow response. The graph shown in Figure 20 shows the parameters of the normative model as K ｍ = 0.95 × K, ζ ｍ = 4.0 × ζ, ω ｍ This is obtained by setting = 2.0 × ω. Using equations (25), (26), and (27), the proportional gain k ｐ = 4.0, integral gain k ｉ = 0.0166, differential gain k ｄ The result is determined to be 263.4.

[0084] Referring to Figures 21 to 23, the learning method for the control device 32 of the drive unit 31 (example) and the learning method according to the comparative example are compared. The only difference between the comparative example and the example is that the discount rate is fixed at 0.99.

[0085] As shown in Figure 21, in both the example and the comparative example, the discount rate gradually increases from a low initial value during the initial learning phase, and the value of the loss function also increases accordingly to track the target value. When the number of episodes is around 3000, the reward temporarily decreases in both the example and the comparative example, but in the example, the period of reward decline is shortened by adjusting the discount rate, and a higher reward is achieved more quickly.

[0086] In the later stages of learning, the value of the loss function decreases as learning progresses. However, in the example, the discount rate is increased to take into account long-term rewards, resulting in rewards equivalent to or greater than those in the comparative example. From these findings, it can be confirmed that the dynamic adjustment of the discount rate in the example enables the stabilization of learning and the acquisition of strategies to achieve higher rewards.

[0087] Figure 22 shows the gear shifting behavior of the drive unit 31 by the control device 32 using a trained model that has completed 8,000 episodes of training using the comparative example's training method. Figure 23 shows the gear shifting behavior of the drive unit 31 by the control device 32 using a trained model that has completed 8,000 episodes of training using the embodiment's training method. In the embodiment (Figure 23), it can be confirmed that the fluctuation in output torque is smaller, i.e., the shift shock is smaller, compared to the comparative example (Figure 22). Therefore, it can be concluded that the method according to the embodiment achieves stable learning of a control strategy that suppresses shift shock.

[0088] The embodiments are not limited to the above configuration and can be broadly modified. For example, the descriptive model P may use the transfer function of a third-order lag system instead of the transfer functions of first-order and second-order lag systems. In this case, the normative model should also use the transfer function of a third-order lag system to match the descriptive model.

[0089] The reinforcement learning method and reinforcement learning apparatus described above can create trained models used, for example, in vehicle control or robot control. Vehicle control includes, for example, autonomous driving control, such as lane changes, follow control, merging control, and gear shift control.

[0090] The reinforcement learning method according to this embodiment is effective in environments where the optimal policy depends on the discount rate. In other words, the reinforcement learning method according to this embodiment is effective in environments where there exists a number of steps that make the derivative of the discounted cumulative reward zero. On the other hand, the reinforcement learning method according to this embodiment is not effective in environments where the sign of the derivative of the discounted cumulative reward does not change, that is, in environments where the derivative is not zero. In this environment, the discounted cumulative reward is monotonically increasing or monotonically decreasing, and the optimal policy does not depend on the discount rate.

[0091] The above embodiments may also be described as follows:

[0092] One embodiment is a computer-based reinforcement learning method, which performs a first reinforcement learning process in which a fixed value is set for the discount rate and a policy is determined to increase the cumulative discount reward; information is obtained showing the relationship between the number of trials and parameters corresponding to the learning progress; based on the information, the relationship between the number of trials and the parameters is approximated as a descriptive model P; a normative model M is set for the relationship between the number of trials and the parameters, where the normative model M is represented by a transfer function of a first-order lag system; the discount rate corresponding to the number of trials is set so that the parameters output from the descriptive model P approach the parameters output from the normative model M; and a second reinforcement learning process is performed in which a policy is determined to increase the cumulative discount reward using the discount rate set in accordance with the number of trials and the parameters.

[0093] According to this embodiment, a reinforcement learning method can be provided that can increase the learning speed and the discounted cumulative reward.

[0094] In the above embodiment, the discount rate corresponding to the number of trials may be set by performing model matching based on PID control so that the parameters output from the descriptive model P approach the parameters output from the normative model M.

[0095] In this embodiment, the discount rate is set so that the parameter corresponding to the progress of learning approaches the target value determined by the normative model. In other words, the progress of learning can be controlled by controlling the discount rate.

[0096] In the above embodiment, the parameter may be the value of the loss function.

[0097] According to this embodiment, the value of the loss function can be controlled to a target value by controlling the discount rate.

[0098] In the above embodiment, the descriptive model P may be represented by the transfer function of a first-order lag system. Alternatively, the descriptive model may be represented by the transfer function of a second-order lag system.

[0099] According to this embodiment, the value of the loss function can be approximated by a simple transfer function.

[0100] Another embodiment is a reinforcement learning device 1 that performs first reinforcement learning to determine a policy to increase the discounted cumulative reward while fixing the discount rate, acquires information showing the relationship between the number of trials and parameters corresponding to the learning progress, approximates the relationship between the number of trials and the parameters as a descriptive model P based on the information, sets a normative model M for the relationship between the number of trials and the parameters, where the normative model M is represented by a transfer function of a first-order lag system, sets the discount rate corresponding to the number of trials so that the parameters output from the descriptive model P approach the parameters output from the normative model M, and performs second reinforcement learning to determine a policy to increase the discounted cumulative reward using the discount rate set corresponding to the number of trials.

[0101] According to this embodiment, a reinforcement learning device can be provided that can increase the learning speed and increase the discounted cumulative reward.

[0102] Another embodiment is a control program for causing a computer to execute a reinforcement learning method, which causes the computer to execute a first reinforcement learning program that fixes the discount rate and determines a policy that increases the cumulative discount reward, to acquire information showing the relationship between the number of trials and parameters corresponding to the learning progress, to approximate the relationship between the number of trials and the parameters as a descriptive model P based on the information, to set a normative model M for the relationship between the number of trials and the parameters, where the normative model M is represented by a transfer function of a first-order lag system, to set the discount rate corresponding to the number of trials such that the parameters output from the descriptive model P approach the parameters output from the normative model M, and to execute a second reinforcement learning program that uses the discount rate set corresponding to the number of trials to determine a policy that increases the cumulative discount reward.

[0103] According to this embodiment, a control program can be provided that can increase the learning speed and increase the discounted cumulative reward.

[0104] Another embodiment is a method for manufacturing a trained model executed by a computer, which involves performing first reinforcement learning to determine a policy such that the discounted cumulative reward increases while fixing the discount rate, obtaining information showing the relationship between the number of trials and parameters corresponding to the learning progress, approximating the relationship between the number of trials and the parameters as a descriptive model P based on the information, setting a normative model M for the relationship between the number of trials and the parameters, where the normative model M is represented by a transfer function of a first-order lag system, setting the discount rate corresponding to the number of trials so that the parameters output from the descriptive model P approach the parameters output from the normative model M, and performing second reinforcement learning to determine a policy such that the discounted cumulative reward increases using the discount rate set corresponding to the number of trials.

[0105] According to this embodiment, it is possible to provide a method for manufacturing a trained model that can increase the learning speed and increase the discounted cumulative reward.

[0106] Another embodiment is a reinforcement learning method for a learned model implemented in a control device 32 of a drive unit 31 having an electric motor 34 and a two-speed transmission 35, wherein the two-speed transmission 35 includes a transmission input shaft 38 connected to the output shaft (motor output shaft 37) of the electric motor 34, a sun gear 39 coupled to the transmission input shaft 38, a carrier 42 that rotatably supports a plurality of planetary gears 41 that mesh with the sun gear 39, a ring gear 43 that meshes with the plurality of planetary gears 41, a case 46 that rotatably supports the ring gear 43 in one direction via a one-way clutch 44, a transmission output shaft 47 coupled to the carrier 42, and a clutch that connects and disconnects the transmission input shaft 38 and the carrier 42, and the control device 32 inputs state quantities of the two-speed transmission 35 to the learned model, thereby enabling the electric motor 3 4 A target motor torque is obtained, the electric motor 34 is controlled based on the target motor torque, and a first reinforcement learning is performed in which a policy is determined to increase the discounted cumulative reward by setting a fixed value for the discount rate. Information is obtained showing the relationship between the number of trials and parameters corresponding to the learning progress, and based on the information, the relationship between the number of trials and the parameters is approximated as a descriptive model, and a normative model is set for the relationship between the number of trials and the parameters, where the normative model is represented by a transfer function of a second-order lag system, and the discount rate corresponding to the number of trials is set so that the parameters output from the descriptive model approach the parameters output from the normative model, and a second reinforcement learning is performed in which a policy is determined to increase the discounted cumulative reward using the discount rate set in accordance with the number of trials and the parameters.

[0107] According to this embodiment, a reinforcement learning method can be provided that can increase the learning speed and the discounted cumulative reward.

[0108] 1: Reinforcement learning device 2: Processor 3: Memory 5: Agent 6: Environment 7: State observation unit 8: Action decision unit 9: Learning unit 11: State generation unit 12: Reward generation unit 13: Transition model 15: DQN model 31: Drive unit 32: Control device 34: Electric motor 35: Two-speed transmission 37: Motor output shaft 38: Transmission input shaft 39: Sun gear 41: Planetary gear 42: Carrier 43: Ring gear 44: One-way clutch 46: Case 47: Transmission output shaft 48: Clutch 51: Control unit 52: Learning unit 55: Sensor

Claims

1. A reinforcement learning method performed by a computer, comprising: performing a first reinforcement learning in which a fixed value is set for the discount rate and a policy is determined to increase the discounted cumulative reward; obtaining information showing the relationship between the number of trials and parameters corresponding to the learning progress; approximating the relationship between the number of trials and the parameters as a descriptive model based on the information; setting a normative model for the relationship between the number of trials and the parameters, where the normative model is represented by a transfer function of a first-order lag system; setting the discount rate corresponding to the number of trials so that the parameters output from the descriptive model approach the parameters output from the normative model; and performing a second reinforcement learning in which a policy is determined to increase the discounted cumulative reward using the discount rate set in accordance with the number of trials and the parameters.

2. The reinforcement learning method according to claim 1, wherein the discount rate corresponding to the number of trials is set by performing model matching based on PID control so that the parameters output from the descriptive model approach the parameters output from the normative model.

3. The reinforcement learning method according to claim 1 or 2, wherein the parameter is the value of the loss function.

4. The reinforcement learning method according to claim 1 or 2, wherein the descriptive model is represented by the transfer function of a first-order lag system.

5. The reinforcement learning method according to claim 1 or 2, wherein the descriptive model is represented by the transfer function of a second-order lag system.

6. A reinforcement learning device that performs a first reinforcement learning process in which it fixes the discount rate and determines a policy to increase the discounted cumulative reward, obtains information showing the relationship between the number of trials and parameters corresponding to the learning progress, approximates the relationship between the number of trials and the parameters as a descriptive model based on the information, sets a normative model for the relationship between the number of trials and the parameters, where the normative model is represented by a transfer function of a first-order lag system, sets the discount rate corresponding to the number of trials so that the parameters output from the descriptive model approach the parameters output from the normative model, and performs a second reinforcement learning process in which it uses the discount rate set corresponding to the number of trials to determine a policy to increase the discounted cumulative reward.

7. A control program for causing a computer to execute a reinforcement learning method, the program causing the computer to execute a first reinforcement learning program in which it determines a policy to increase the discounted cumulative reward while fixing the discount rate, to obtain information showing the relationship between the number of trials and parameters corresponding to the learning progress, to approximate the relationship between the number of trials and the parameters as a descriptive model based on the information, to set a normative model for the relationship between the number of trials and the parameters, where the normative model is represented by a transfer function of a first-order lag system, to set the discount rate corresponding to the number of trials such that the parameters output from the descriptive model approach the parameters output from the normative model, and to execute a second reinforcement learning program in which it determines a policy to increase the discounted cumulative reward using the discount rate set corresponding to the number of trials.

8. A method for manufacturing a trained model executed by a computer, comprising: performing first reinforcement learning to determine a policy such that the discounted cumulative reward increases while fixing the discount rate; obtaining information showing the relationship between the number of trials and parameters corresponding to the learning progress; approximating the relationship between the number of trials and the parameters as a descriptive model based on the information; setting a normative model for the relationship between the number of trials and the parameters, where the normative model is represented by a transfer function of a first-order lag system; setting the discount rate corresponding to the number of trials such that the parameters output from the descriptive model approach the parameters output from the normative model; and performing second reinforcement learning to determine a policy such that the discounted cumulative reward increases using the discount rate set in accordance with the number of trials.

9. A reinforcement learning method for a trained model implemented in a control device of a drive unit having an electric motor and a two-speed transmission, wherein the two-speed transmission includes: a transmission input shaft connected to the output shaft of the electric motor; a sun gear coupled to the transmission input shaft; a carrier rotatably supporting a plurality of planetary gears that mesh with the sun gear; a ring gear that meshes with the plurality of planetary gears; a case that supports the ring gear rotatably in one direction via a one-way clutch; a transmission output shaft coupled to the carrier; and a clutch that connects and disconnects the transmission input shaft and the carrier, wherein the control device obtains a target motor torque of the electric motor by inputting state quantities of the two-speed transmission to the trained model, controls the electric motor based on the target motor torque, performs first reinforcement learning which determines a policy to increase the discounted cumulative reward by setting a fixed value for the discount rate, obtains information showing the relationship between the number of trials and parameters corresponding to the learning progress, and approximates the relationship between the number of trials and the parameters as a descriptive model based on the information, A reinforcement learning method that performs a second reinforcement learning, which involves setting a normative model for the relationship between the number of trials and the parameters, where the normative model is represented by a transfer function of a second-order lag system, setting a discount rate corresponding to the number of trials so that the parameters output from the descriptive model approach the parameters output from the normative model, and determining a policy to increase the discounted cumulative reward using the discount rate set in accordance with the number of trials and the parameters.

Images