Reinforcement learning-based control method for differential steering drive system of wall‑climbing cleaning vehicle

Abstract

The present invention relates to the technical field of hull cleaning vehicle control methods. Disclosed is a reinforcement learning-based control method for a differential steering drive system of a wall‑climbing cleaning vehicle. The method comprises the following steps: model establishment: modeling a differential steering drive control process of a wall-climbing cleaning vehicle as a Markov decision process; completing the definition of a state space and an action space; constructing a state transition model of a differential steering drive system; acquiring current state data, and generating a steering wheel speed and steering time allocation strategy by means of the model; executing a steering action strategy, and updating the state data when an action is executed; returning a reward on the basis of an actual execution effect to evaluate whether the action strategy is good or not; performing model optimization by means of an experience replay buffer; and resetting a scene, continuing training until the model converges, and finally obtaining a trained differential steering drive system model. The present invention reduces the complexity of path planning and control algorithms, meets the requirement of maximizing coverage cleaning, and simplifies path planning and control algorithms.

Description

A Reinforcement Learning-Based Control Method for Differential Steering Drive System of a Crawling Washing Truck Technical Field

[0001] This invention belongs to the technical field of control methods for hull cleaning trolleys, and more specifically, it relates to a control method for a differential steering drive system of a hull cleaning trolley based on reinforcement learning. Background Technology

[0002] The inner walls of bulk carrier cargo holds are often subject to rust and fouling, making cleaning difficult and requiring strict inspection standards. This is especially true when loading grain cargo after transporting oily cargo, in which case shipping companies must invest significant resources in cleaning to avoid delays. The outer walls of the hull also face biofouling problems, which not only affect speed and increase fuel consumption but also serve as a major carrier for the transfer of aquatic species.

[0003] Currently, hull cleaning is mostly done manually or with the assistance of semi-automated equipment. This is not only time-consuming, costly, and labor-intensive, but also poses safety risks. Therefore, cargo hold cleaning robots and underwater cleaning robots are gradually being adopted to complete the cleaning tasks. Cleaning robots are usually based on a cart structure and are fixed to the hull surface using magnetic adsorption. They clean using cleaning brushes or jet disks. However, these devices usually require professional operators, and their autonomous navigation capabilities and intelligence levels still need improvement. This not only makes it difficult for the cleaning cart to effectively cover the entire cleaning area due to the complex shape of the hull's interior and exterior surfaces, resulting in omissions and repeated cleaning, but also requires a high level of operator skill, affecting the cleaning effect and efficiency. Summary of the Invention

[0004] This invention addresses the technical problems existing in the prior art by providing a control method for the differential steering drive system of a climbing cleaning truck based on reinforcement learning.

[0005] To solve the above-mentioned technical problems, the present invention includes the following steps:

[0006] S1. Model establishment: The differential steering drive control process of the wall-climbing cleaning vehicle is modeled as a Markov decision process.

[0007] S2, complete the definition of state space and action space;

[0008] S3. Construct a state transition model for the differential steering drive system;

[0009] S4. Obtain current state data and generate steering wheel speed and steering time allocation strategies through the model;

[0010] S5. Execute the steering action strategy and update the state data after the action is executed;

[0011] S6. Return rewards based on actual performance to evaluate the effectiveness of the action strategy;

[0012] S7. Optimize the model using the experience replay pool;

[0013] S8, the wall-climbing cleaning vehicle continues to move forward, and the preset cleaning distance is maintained;

[0014] S9. Repeat S4-S8. After the training session ends, reset the scene and continue training until the model converges. Finally, obtain the trained differential steering drive system model.

[0015] Preferably, in S1, a Markov decision process for the differential steering drive of the wall-climbing cleaning vehicle is represented by a tuple [S,A,R,P], and an initial action strategy π is given, where:

[0016] S represents the state space, that is, the state information of the environment in which the agent is located at time t;

[0017] A represents the action space, that is, the environmental state at time t is S. t At that time, the agent selects an action based on policy π, where policy π refers to the mapping from state to action;

[0018] R represents the reward, i.e., the environmental state at time t is S. t At that time, the intelligent agent takes action A. t Enter the next state S t+ The reward obtained at time 1;

[0019] P represents the state transition probability, indicating that the agent is given the previous state S at time t. t ∈S and action A t Given that ∈ A, the probability of random variables s´ and r occurring.

[0020] Preferably, S2 includes:

[0021] S2.1. Construct a global coordinate system with the initial climbing position of the trolley as the origin;

[0022] S2.2 The current status data of the wall-climbing cleaning vehicle includes: the coordinates (x,y,z) of the wall-climbing cleaning vehicle in the global coordinate system, the velocity v of the wall-climbing cleaning vehicle in the geodetic coordinate system, the roll angle Roll, pitch angle Pitch, yaw angle Yaw, and the preset lateral turning distance d.

[0023] S2.3, The action output is: Action = [leftS, rightS, turnT];

[0024] Where leftS represents the speed of the left wheel when the wall-climbing cleaning vehicle turns, in radians per second; rightS represents the speed of the right wheel when the wall-climbing cleaning vehicle turns, in radians per second; and turnT represents the turning time of the wall-climbing cleaning vehicle, in seconds.

[0025] Preferably, the minimum, maximum and step values ​​of the left and right wheel speeds and turning times of the wall-climbing cleaning vehicle are set, and the continuous values ​​are converted into discrete points. At this time, the motion space can be understood as a three-dimensional discrete point motion space, and each point in the three-dimensional motion space corresponds to different left and right wheel speeds and turning times.

[0026] Preferably, in S3, a control model based on the Deep Q-Network (DQN) framework is constructed. The DQN framework includes:

[0027] The policy network estimates the Q-value of each action in a given state. The action-value function Q(s,a) is used to evaluate the expected reward after performing action a in state s.

[0028] The target network, which has the same structural parameters as the online network, generates the target Q-value.

[0029] Preferably, in S4, the state data is transmitted as input to the policy network, and the policy network outputs the expected reward value obtained when taking different actions in the current state.

[0030] Preferably, in S5, an ε-greedy algorithm is introduced, including:

[0031] S5.1. Randomly select an action with probability ϵ, namely, a random steering wheel speed and steering time;

[0032] S5.2 Select the steering wheel speed and steering time strategy with the highest Q value based on probability 1-ϵ as the action output of the current strategy;

[0033] S5.3, Execute the action and update the state data S t+ 1.

[0034] Preferably, S6 includes:

[0035] Reward R(s,a) = R(str) + R(dev) + R(ec)

[0036] Wherein, R(str) is the straight-line travel penalty, which is used to measure whether the car can travel in a predetermined straight line after turning;

[0037] The judgment indicator is the change in cleaning height or depth z after turning and traveling a set cleaning distance;

[0038] R(dev) is the lateral distance penalty, used to measure whether the car has laterally shifted a set distance relative to its position before turning after turning;

[0039] The height / depth of the car before turning is z, the height / depth of the car after turning is z´, and the position offset before and after turning is z´- z;

[0040] By determining the roll angle, pitch angle, and yaw angle of the wall-climbing cleaning vehicle relative to the global coordinate system before and after turning, two coordinate systems corresponding to Euler angles are determined. The projections of the two coordinate systems onto the corresponding plane are calculated, and the projection vectors are normalized. The angle θ between the two vectors, which is the angle between the vehicle body and the turning wall before and after turning, is solved.

[0041] The judgment index is the difference between the positional offset z´- z before and after steering and the preset lateral steering distance d, under the action of the angle θ between the vehicle body and the steering wall before and after steering.

[0042] R(ec) is the energy loss penalty, which is used to measure whether the car slips during the turning process and generates unnecessary energy consumption.

[0043] The criterion for judgment is the lateral slip angle, which is the angular difference between the actual driving direction and the direction of the tire.

[0044] Lateral slip angle is defined as:

[0045]

[0046] Among them, v lateral The lateral velocity is the equivalent velocity of the wall-climbing cleaning vehicle in the geodetic coordinate system.

[0047] vlongitudinal is the longitudinal velocity, calculated using acceleration and angular velocity data provided by the inertial unit.

[0048] Preferably, S7 includes:

[0049] S7.1 Perform data sampling, and transfer the interactive data [S,A,R,S] t+1 The data are stored in the experience replay pool. Once the experience replay pool has accumulated enough samples, the data in the experience replay pool is randomly selected for training.

[0050] S7.2. Introduce priority experience replay to improve sample utilization. Calculate the sampling probability of each sample in the experience replay pool. The method for calculating the sampling probability is as follows:

[0051]

[0052] Among them, D j Let D be the priority of the j-th sample. k Let α represent the priority of the k-th sample, and α be the degree of priority.

[0053] S7.3. Predict the currently extracted sample using the current value network, calculate the loss between the current value network and the target value network, and use gradient descent to minimize the loss to update the parameters of the current value network. After a certain number of iterations, synchronize the parameters of the current value network to the target value network.

[0054] The value function update formula is:

[0055] θ t +1 ←θ t +α[R +γmaxQ(s´,a;w) -Q(s,a;w)] ∇Q(s,a;w)

[0056] Where α is the learning rate, γ is the reward deduction factor, and w is the parameter of the corresponding network.

[0057] Preferably, S9 includes:

[0058] As training progresses, the cumulative reward for each training round gradually increases, indicating that the model policy is constantly improving and eventually tends to a stable value, indicating that the model's performance in the current training environment has reached a balance point.

[0059] The loss function used to measure the mean square error between the current value network and the target value network should gradually decrease, reflecting the convergence of the model in the optimization process. When the loss stabilizes at a certain value, it indicates that the model's predictions are becoming consistent with the target Q value.

[0060] When both metrics show a stable state, it can be considered that the current training has converged and the model has reached a relatively ideal performance.

[0061] Compared with the prior art, the present invention has the following beneficial effects:

[0062] This invention employs reinforcement learning to optimize the steering speed and steering time strategy of a four-wheeled wall-climbing cleaning vehicle. It can autonomously adapt to task requirements by performing lateral displacement with a small turning radius, transforming the path planning problem of "where to go and how to get there" into "how to adjust to the desired posture to maximize the cleaning coverage area".

[0063] This invention constructs a state transition model for a differential steering drive system, collects state data of the wall-climbing vehicle, and transmits it to the model to generate corresponding steering action strategies. Simultaneously, it optimizes the model and evaluates the strategies, reducing the complexity of path planning and control algorithms. The reinforcement learning algorithm enables the vehicle to continuously learn and adjust its behavior strategies through interaction with the environment. By continuously trying and failing during the interaction between the agent and the environment, it learns how to take actions in the environment to maximize expected returns, finding the optimal wall-climbing path and actions. By adjusting to the optimal posture to meet the requirement of maximizing the cleaning area, this invention effectively improves the intelligence level of the wall-climbing cleaning vehicle, enhancing its cleaning efficiency and effectiveness. Attached Figure Description

[0064] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0065] Figure 1 is a schematic diagram of the training process of an embodiment of the control method of the present invention;

[0066] Figure 2 is a schematic diagram of the algorithm framework of an embodiment of the control method of the present invention. Detailed Implementation

[0067] To make the technical problems, technical solutions, and beneficial effects to be solved by this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this application.

[0068] Please refer to Figure 1. This embodiment of the invention provides a control method for a differential steering drive system of a climbing cleaning truck based on reinforcement learning, including the following steps:

[0069] S1. Model establishment: The differential steering drive control process of the wall-climbing cleaning vehicle is modeled as a Markov decision process.

[0070] S2, complete the definition of state space and action space;

[0071] S3. Construct a state transition model for the differential steering drive system;

[0072] S4. Obtain current state data and generate steering wheel speed and steering time allocation strategies through the model;

[0073] S5. Execute the steering action strategy and update the state data after the action is executed;

[0074] S6. Return rewards based on actual performance to evaluate the effectiveness of the action strategy;

[0075] S7. Optimize the model using the experience replay pool;

[0076] S8, the wall-climbing cleaning vehicle continues to move forward, and the preset cleaning distance is maintained;

[0077] S9. Repeat S4-S8. After the training session ends, reset the scene and continue training until the model converges.

[0078] This invention employs reinforcement learning to optimize the steering speed and steering time strategy of a four-wheeled wall-climbing cleaning vehicle. It can autonomously adapt to task requirements by performing lateral displacement with a small turning radius, transforming the path planning problem of "where to go and how to get there" into "how to adjust to the desired posture to maximize the cleaning coverage area".

[0079] This invention constructs a state transition model for a differential steering drive system, collects state data of the wall-climbing vehicle, and transmits it to the model to generate corresponding steering action strategies. Simultaneously, it optimizes the model and evaluates the strategies, reducing the complexity of path planning and control algorithms. The reinforcement learning algorithm enables the vehicle to continuously learn and adjust its behavior strategies through interaction with the environment. By continuously trying and failing during the interaction between the agent and the environment, it learns how to take actions in the environment to maximize expected returns, finding the optimal wall-climbing path and actions. By adjusting to the optimal posture to meet the requirement of maximizing the cleaning area, this invention effectively improves the intelligence level of the wall-climbing cleaning vehicle, enhancing its cleaning efficiency and effectiveness.

[0080] Specifically, this invention provides a reinforcement learning-based control method for a differential steering drive system of a wall-climbing cleaning vehicle. By adjusting to the optimal posture to maximize the cleaning area, the method aims to enable the wall-climbing vehicle to cover more areas and improve cleaning efficiency per unit time. Assuming the current cleaning method involves lateral cleaning at different heights or depths through steering, this invention is also applicable to longitudinal cleaning. Based on this, the control method includes the following steps:

[0081] S1. Model Establishment: The differential steering drive control process of the wall-climbing cleaning vehicle is modeled as a Markov decision process.

[0082] Specifically, a Markov decision process for the differential steering drive of the wall-climbing cleaning vehicle is represented by a tuple [S,A,R,P], given an initial action policy π, where:

[0083] S represents the state space, that is, the state information of the environment in which the agent is located at time t;

[0084] A represents the action space, that is, the environmental state at time t is S. tAt that time, the agent selects an action based on policy π, where policy π refers to the mapping from state to action;

[0085] R represents the reward, i.e., the environmental state at time t is S. t At that time, the intelligent agent takes action A. t Enter the next state S t+ The reward obtained at time 1;

[0086] P represents the state transition probability, indicating that the agent is given the previous state S at time t. t ∈S and action A t Given that ∈ A, the probability of random variables s´ and r occurring.

[0087] S2, complete the definition of state space and action space.

[0088] Specifically, it includes:

[0089] S2.1 First, construct a global coordinate system with the initial climbing position of the trolley as the origin;

[0090] S2.2. Based on this, the current status data of the wall-climbing cleaning vehicle includes: the coordinates (x, y, z) of the wall-climbing cleaning vehicle in the global coordinate system, the velocity v of the wall-climbing cleaning vehicle in the geodetic coordinate system, the roll angle Roll, pitch angle Pitch, yaw angle Yaw, and the preset lateral turning distance d.

[0091] S2.3, The action output is: Action = [leftS, rightS, turnT];

[0092] Where leftS represents the speed of the left wheel when the wall-climbing cleaning vehicle turns, in radians per second;

[0093] rightS represents the right wheel speed of the wall-climbing cleaning vehicle when it turns, in radians per second;

[0094] turnT indicates the turning time of the wall-climbing cleaning vehicle, in seconds.

[0095] Furthermore, to simplify the computation, minimum, maximum, and step values ​​of the left and right wheel speeds and turning times of the wall-climbing cleaning vehicle are set, transforming continuous values ​​into discrete points. At this point, the motion space can be understood as a three-dimensional discrete point motion space, where each point corresponds to a different left and right wheel speed and turning time.

[0096] S3. Construct a state transition model for the differential steering drive system.

[0097] A control model based on the Deep Q-Network (DQN) framework is constructed, as shown in Figure 2. The DQN framework includes the following two key components:

[0098] 1. Policy Network (Q Network): Estimates the Q-value of each action in a given state. The action-value function Q(s,a) is used to evaluate the expected reward after performing action a in state s.

[0099] 2. Target Network: A target network with the same structural parameters as the online network generates the target Q value.

[0100] S4. Obtain current state data and generate steering wheel speed and steering time allocation strategies through the model.

[0101] The state data is transmitted as input to the policy network, which outputs the expected reward value obtained when taking different actions in the current state.

[0102] S5. Execute the steering action strategy and update the state data after the action is executed.

[0103] Introducing the ϵ-greedy algorithm, including:

[0104] S5.1. Randomly select an action with probability ϵ, namely, a random steering wheel speed and steering time;

[0105] S5.2 Select the steering wheel speed and steering time strategy with the highest Q value based on probability 1-ϵ as the action output of the current strategy;

[0106] S5.3, Execute the action and update the state data S t+ 1.

[0107] S6. Return rewards based on actual performance to evaluate the effectiveness of the action strategy.

[0108] Specifically, it includes:

[0109] Reward R(s,a) = R(str) + R(dev) + R(ec)

[0110] R(str) is the straight-line travel penalty, which is used to measure whether the car can travel in a predetermined straight line after turning.

[0111] The judgment indicator is the change in cleaning height or depth z after turning and driving a set cleaning distance.

[0112] R(dev) is the lateral distance penalty, used to measure whether the car has laterally shifted a set distance relative to its position before turning after turning.

[0113] The height / depth of the car before turning is z, the height / depth of the car after turning is z´, and the position offset before and after turning is z´- z.

[0114] Furthermore, in this embodiment, since the hull has a certain curvature, two coordinate systems corresponding to Euler angles are determined by the roll angle, pitch angle and heading angle of the wall-climbing cleaning vehicle relative to the global coordinate system before and after turning. The projections of the two coordinate systems on the corresponding plane are calculated, and the projection vectors are normalized to solve the angle θ between the two vectors between the vehicle body and the turning wall before and after turning.

[0115] The judgment index is the difference between the positional offset z´- z before and after steering and the preset lateral steering distance d, under the action of the angle θ between the vehicle body and the steering wall before and after steering.

[0116] R(ec) is the energy loss penalty, used to measure whether the car slips during steering and generates unnecessary energy consumption.

[0117] The criterion for judgment is the lateral slip angle, which is the angular difference between the actual driving direction and the direction of the tire.

[0118] Lateral slip angle is defined as:

[0119]

[0120] Among them, v lateral The lateral velocity is the equivalent velocity of the wall-climbing cleaning vehicle in the geodetic coordinate system.

[0121] vlongitudinal is the longitudinal velocity, calculated using acceleration and angular velocity data provided by the inertial unit.

[0122] S7. Optimize the model using the experience replay pool.

[0123] As shown in Figure 2, it specifically includes:

[0124] S7.1 First, data sampling is performed, and the interactive data [S,A,R,S] is collected. t+1 The data are stored in the experience replay pool. Once enough samples have accumulated in the experience replay pool, data is randomly selected from the experience replay pool for training.

[0125] S7.2 Simultaneously, priority experience replay is introduced to improve sample utilization. The sampling probability of each sample in the experience replay pool is calculated. The method for calculating the sampling probability is as follows:

[0126]

[0127] Among them, D j Let D be the priority of the j-th sample. k Let α be the priority of the k-th sample, and α be the priority level; when α is 0, it is uniform sampling.

[0128] S7.3. Predict the currently extracted sample using the current value (policy) network, calculate the loss between the current value network and the target value network, and use gradient descent to minimize the loss to update the parameters of the current value network. After a certain number of iterations, synchronize the parameters of the current value network to the target value network.

[0129] The value function update formula is:

[0130] θ t +1 ←θ t +α[R +γmaxQ(s´,a;w) -Q(s,a;w)] ∇Q(s,a;w)

[0131] Where α is the learning rate, γ is the reward deduction factor, and w is the parameter of the corresponding network.

[0132] S8, the wall-climbing cleaning vehicle continues to move forward, and the preset cleaning distance is maintained.

[0133] S9. Repeat S4-S8. After the training session ends, reset the scene and continue training until the model converges. Finally, obtain the trained differential steering drive system model.

[0134] Specifically, it includes:

[0135] As training progresses, the cumulative reward for each training round gradually increases, indicating that the model policy is constantly improving and eventually tends to a stable value, indicating that the model's performance in the current training environment has reached a balance point.

[0136] The loss function used to measure the mean squared error between the current value network and the target value network should gradually decrease, reflecting the convergence of the model during the optimization process. When the loss stabilizes at a certain value, it indicates that the model's predictions are becoming consistent with the target Q value.

[0137] When both of the above metrics show a stable state, it can be considered that the current training has converged and the model has reached a relatively ideal performance.

[0138] This invention proposes a reinforcement learning-based control method for the differential steering drive system of a wall-climbing cleaning vehicle to address the technical problem of existing ship hull cleaning equipment, which struggles to construct accurate dynamic models and design effective path planning algorithms. The method mainly includes the following steps: Modeling process: The differential steering drive control process of the wall-climbing cleaning vehicle is modeled as a Markov decision process, defining the state space and action space, and constructing a state transition model for the differential steering drive system; Strategy generation: The state data of the wall-climbing cleaning vehicle is collected and transmitted to the model to generate corresponding steering action strategies. Simultaneously, a reward value is returned based on the strategy execution result to evaluate the strategy's effectiveness; Model optimization: The state, action, and reward data are stored in an experience replay pool. The model is trained and improved using optimization algorithms to ultimately obtain a differential steering drive system model that outputs the optimal strategy.

[0139] This invention employs reinforcement learning to optimize the steering speed and timing strategies of a four-wheeled wall-climbing cleaning vehicle. This achieves optimized control that autonomously adapts to task requirements, enabling small-radius lateral displacement. The path planning problem of "where to go and how to get there" is transformed into adjusting the vehicle's posture to maximize cleaning coverage, simplifying the path planning and control algorithms. It also effectively addresses the impact of gravity, speed interference, and surface material on system performance. Furthermore, this invention has low dependence on sensor accuracy, considers slippage during training, improves the stability of the four-wheeled wall-climbing cleaning vehicle during steering, reduces energy consumption, and minimizes wear on the magnetic wheel anti-slip sleeves.

[0140] This invention constructs a state transition model for a differential steering drive system, collects state data of the wall-climbing vehicle, and transmits it to the model to generate corresponding steering action strategies. Simultaneously, it optimizes the model and evaluates the strategies, reducing the complexity of path planning and control algorithms. The reinforcement learning algorithm enables the vehicle to continuously learn and adjust its behavior strategies through interaction with the environment. It learns how to take actions in the environment to maximize expected returns through trial and error during the interaction between the agent and the environment, finding the optimal wall-climbing path and actions. It can adapt to different surface and gravity conditions, cope with speed disturbances or changes in surface friction, and adjust to the optimal posture to meet the requirement of maximizing the cleaning area. This effectively improves the intelligence level of the wall-climbing cleaning vehicle, enhancing its cleaning efficiency and effect. It is suitable for efficient and intelligent cleaning of the inner and outer walls of ship cabins.

[0141] In the description of this invention, it should be understood that terms such as “length”, “width”, “upper”, “lower”, “front”, “rear”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inner”, and “outer” indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0142] Furthermore, in the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0143] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A control method for a differential steering drive system of a climbing cleaning vehicle based on reinforcement learning, characterized in that, Includes the following steps: S1. Model establishment: The differential steering drive control process of the wall-climbing cleaning vehicle is modeled as a Markov decision process. S2, complete the definition of state space and action space; S3. Construct a state transition model for the differential steering drive system; S4. Obtain current state data and generate steering wheel speed and steering time allocation strategies through the model; S5. Execute the steering action strategy and update the state data after the action is executed; S6. Return rewards based on actual performance to evaluate the effectiveness of the action strategy; S7. Optimize the model using the experience replay pool; S8, the wall-climbing cleaning vehicle continues to move forward, and the preset cleaning distance is maintained; S9. Repeat S4-S8. After the training session ends, reset the scene and continue training until the model converges. Finally, obtain the trained differential steering drive system model.

2. The control method of claim 1, wherein, In S1, a Markov decision process for the differential steering drive of the wall-climbing cleaning vehicle is represented by a tuple [S,A,R,P], and an initial action policy π is given, where: S represents the state space, that is, the state information of the environment in which the agent is located at time t; A represents the action space, that is, at time t, the agent selects an action according to the policy π, and the policy π refers to the mapping from the state to the action; t at time t, the agent selects an action according to the policy π, and the policy π refers to the mapping from the state to the action; R represents a reward, i.e. at time t, the environment state is S t , the agent takes action A t , enters the next state S t+ 1, the reward obtained at time t; P represents a state transition probability, representing the given predecessor state S of the agent at time t t ∈S and action A t ∈A, the probability of occurrence of random variables s´ and r.

3. The control method of claim 1, wherein S2 include: S2.

1. Construct a global coordinate system with the initial climbing position of the trolley as the origin; S2.2 The current status data of the wall-climbing cleaning vehicle includes: the coordinates (x,y,z) of the wall-climbing cleaning vehicle in the global coordinate system, the velocity v of the wall-climbing cleaning vehicle in the geodetic coordinate system, the roll angle Roll, pitch angle Pitch, yaw angle Yaw, and the preset lateral turning distance d. S2.3, The action output is: Action = [leftS, rightS, turnT]; Where leftS represents the speed of the left wheel of the wall-climbing cleaning vehicle when turning, in radians per second; rightS represents the speed of the right wheel of the wall-climbing cleaning vehicle when turning, in radians per second; and turnT represents the turning time of the wall-climbing cleaning vehicle, in seconds.

4. The control method of claim 3, wherein, Set the minimum, maximum, and step size of the left and right wheel speeds and turning times of the wall-climbing cleaning vehicle, and convert the continuous values ​​into discrete points. At this time, the motion space can be understood as a three-dimensional discrete point motion space, where each point corresponds to a different left and right wheel speed and turning time.

5. The control method of claim 1, wherein, In S3, a control model based on the Deep Q-Network (DQN) framework is constructed. The DQN framework includes: The policy network estimates the Q-value of each action in a given state. The action-value function Q(s,a) is used to evaluate the expected reward after performing action a in state s. The target network, which has the same structural parameters as the online network, generates the target Q-value.

6. The control method of claim 1, wherein, In S4, the state data is transmitted as input to the policy network, and the policy network outputs the expected reward value obtained when taking different actions in the current state.

7. The control method of claim 1, wherein, In S5, the ϵ-greedy algorithm is introduced, including: S5.

1. Randomly select an action with probability ϵ, namely, random steering wheel speed and steering time; S5.2 Select the steering wheel speed and steering time strategy with the highest Q value based on probability 1-ϵ as the action output of the current strategy; S5.3, perform action and update state data S t+ 1.

8. The control method of the differential steering driving system of the reinforced learning-based climbing cleaning vehicle according to claim 1, wherein S6 include: Reward R(s,a) = R(str) + R(dev) + R(ec) Where R(str) is the straight-line travel penalty, which is used to measure whether the car can travel in a predetermined straight line after turning; The judgment indicator is the change in cleaning height or depth z after turning and traveling a set cleaning distance; R(dev) is the lateral distance penalty, used to measure whether the car has laterally shifted a set distance relative to its position before turning after turning; The height / depth of the car before turning is z, the height / depth of the car after turning is z´, and the position offset before and after turning is z´- z; By determining the roll angle, pitch angle, and yaw angle of the wall-climbing cleaning vehicle relative to the global coordinate system before and after turning, two coordinate systems corresponding to Euler angles are determined. The projections of the two coordinate systems onto the corresponding plane are calculated, and the projection vectors are normalized. The angle θ between the two vectors, which is the angle between the vehicle body and the turning wall before and after turning, is solved. The judgment index is the difference between the positional offset z´- z before and after steering and the preset lateral steering distance d, under the action of the angle θ between the vehicle body and the steering wall before and after steering. R(ec) is the energy loss penalty, which is used to measure whether the car slips during the turning process and generates unnecessary energy consumption. The criterion for judgment is the lateral slip angle, which is the angular difference between the actual driving direction and the direction of the tire. Lateral slip angle is defined as: ; wherein v lateral is the lateral velocity, equivalent to the velocity of the wall cleaning vehicle in the earth coordinate system; vlongitudinal is the longitudinal velocity, calculated using acceleration and angular velocity data provided by the inertial unit.

9. The control method of claim 1, wherein, S7 includes: S7.1, data sampling is performed, and interaction data [S, A, R, S t+1 ] is stored in an experience replay pool. When the experience replay pool accumulates enough samples, data in the experience replay pool is randomly drawn for training. S7.

2. Introduce priority experience replay to improve sample utilization. Calculate the sampling probability of each sample in the experience replay pool. The method for calculating the sampling probability is as follows: ; wherein D j is the priority of the jth sample, D k is the priority of the kth sample, and a is the degree of priority. S7.

3. Predict the currently extracted sample using the current value network, calculate the loss between the current value network and the target value network, and use gradient descent to minimize the loss to update the parameters of the current value network. After a certain number of iterations, synchronize the parameters of the current value network to the target value network. The value function update formula is: θ t +1 ← θ t + α[R + γmaxQ(s, a ; w) - Q(s, a; w)] ∇Q(s, a; w) Where α is the learning rate, γ is the reward deduction factor, and w is the parameter of the corresponding network.

10. The control method of claim 1, wherein, S9 includes: As training progresses, the cumulative reward for each training round gradually increases, indicating that the model policy is constantly improving and eventually tends to a stable value, indicating that the model's performance in the current training environment has reached a balance point. The loss function used to measure the mean square error between the current value network and the target value network should gradually decrease, reflecting the convergence of the model in the optimization process. When the loss stabilizes at a certain value, it indicates that the model's predictions are becoming consistent with the target Q value. When both metrics show a stable state, it can be considered that the current training has converged and the model has reached a relatively ideal performance.

Images