Scheduling method and device for automatic charging system, storage medium, and computer program product

Abstract

A scheduling method and device for an automatic charging system, a storage medium, and a computer program product. The scheduling method for an automatic charging system comprises: a construction step: constructing a policy value function and an action value function for each of a charging pile and a robotic arm, and constructing an identical reward function for the charging pile and the robotic arm; a modeling step: using a neural network to model the respective constructed action value functions and policy value functions of the charging pile and the robotic arm, so as to establish an action value network and a policy network; a training step: using a temporal difference algorithm to train the established action value network of the charging pile and the robotic arm, and training the established policy network of the charging pile and the robotic arm with the objective of maximizing cumulative reward; and a parameter updating step: upon completing the training step, updating neural network parameters, so that policies of the charging pile and the robotic arm are optimal scheduling policies.

Description

Automatic charging system scheduling methods, scheduling devices, storage media, and computer program products Technical Field

[0001] This invention relates to the field of automatic charging of electric vehicles, and in particular to a scheduling method and device for an automatic charging system, a storage medium, and a computer program product. Background Technology

[0002] With the rapid growth of the electric vehicle market, the demand for charging infrastructure is also increasing. Currently, most charging stations use fixed charging piles. In this model, electric vehicles may not leave in time after charging and continue to occupy the vacant charging piles, resulting in low utilization of the charging piles. This can lead to serious charging wait problems during peak hours, affecting user experience.

[0003] To address the aforementioned issues, charging parking lots have begun to explore the introduction of automation technologies. Existing automation technologies (for example, see Chinese patent applications CN119142178A and CN112776624A) utilize automated charging robots that carry mobile charging stations to autonomously move to the electric vehicles requiring charging, inserting the charging gun into the vehicles to complete the automatic charging process. This removes the strong binding relationship between charging stations and parking spaces, transforming the "car-to-charging-station" process into a "charging-to-car" process.

[0004] However, the above technologies do not take into account the collision and congestion problems that may occur when the parking lot becomes larger and the number of automatic charging robots increases, nor do they take into account how to efficiently dispatch a large number of mobile automatic charging robots to minimize the charging waiting time.

[0005] To address these challenges, a corresponding scheduling algorithm needs to be designed for specific parking lots. The scheduling problem of automated charging robots is a complex multi-objective optimization problem. First, these robots operate in a dynamically changing environment, with charging demands varying with time and location. Second, the scheduling algorithm must consider not only charging waiting time but also robot energy consumption. Furthermore, when multiple charging robots are present, their coordination is crucial. Traditional scheduling algorithms struggle to handle these difficulties.

[0006] In recent years, reinforcement learning has demonstrated outstanding performance in multiple fields, including gaming, robot control, and autonomous driving. By learning optimal policies through the interaction between an agent and its environment to maximize long-term rewards, reinforcement learning offers a potential solution to the aforementioned problems. Summary of the Invention

[0007] This invention addresses the problems of low utilization rate of charging piles, long charging waiting time, and inflexible scheduling mechanism in existing electric vehicle charging parking lots. By introducing a scheduling method based on multi-agent reinforcement learning, the scheduling process of the automatic charging system with suspended track in the parking lot is optimized, thereby improving the charging efficiency of the parking lot and enhancing the user experience.

[0008] Therefore, this invention is made in view of the above-mentioned technical background. It applies reinforcement learning to the system scheduling problem of automatic charging parking lots with suspended tracks. Specifically, it uses a multi-agent reinforcement learning algorithm to enable the charging piles and robotic arms to cooperate closely and coordinate with each other in a fully cooperative relationship, learn their own optimal strategies to maximize the common reward, and use reinforcement learning with a safety control module to restrict the behavior of the scheduling system to conform to physical constraints.

[0009] According to a first aspect of the present invention, a scheduling method for an automatic charging system is provided. The automatic charging system includes a charging pile and a robotic arm. The robotic arm performs plug-and-unplug operations on a charging gun on the charging pile to charge an electric vehicle. The scheduling method includes the following steps: a construction step, in which a policy value function and an action value function are constructed for the charging pile and the robotic arm respectively, and the same reward function is constructed for the charging pile and the robotic arm; a modeling step, in which the action value function and policy value function of the charging pile and the robotic arm constructed in the construction step are modeled using a neural network to establish an action value network and a policy network; a training step, in which the action value network of the charging pile and the robotic arm established in the modeling step is trained using a temporal difference algorithm, and the policy network of the charging pile and the robotic arm established in the modeling step is trained with the goal of maximizing the cumulative reward; and a parameter update step, in which the parameters of the neural network are updated after the training step is completed, so that the policies of the charging pile and the robotic arm are the optimal scheduling policies.

[0010] As a preferred embodiment, according to the scheduling method of the automatic charging system of the present invention, the objective function of the strategy value function for the construction of the charging pile and the robotic arm in the construction step is defined as follows:

[0011] Where π C* and π J* Let π represent the strategy value functions for the charging station and the robotic arm, respectively. C (a C |s) and π J (a J The objective function of |s), a C ,a J These represent the actions of the charging station and the robotic arm, respectively, while 's' represents the overall environmental state of the automatic charging system. and Let be the expected function.

[0012] r(s,a C ,a J H(π) is the reward function for the charging station and the robotic arm. C (a C |s)) and H(π J (a J |s)) are respectively π C (a C |s) and π J (a J The entropy of |s) represents the degree of randomness of the charging pile strategy and the robotic arm strategy in state s, and α is the regularization coefficient.

[0013] As a preferred embodiment, according to the scheduling method of the automatic charging system of the present invention, the reward functions of the charging pile and the robotic arm are defined as follows:

[0014] Among them, t i (s,a C ,a J This indicates that in state s, the charging station performs action a. C And the robotic arm performs action a J The waiting time for the i-th charging request; q C (a C ) indicates that the charging pile has performed action a. C The total energy consumption, and q J (a J ) indicates that the robotic arm performs the action a. J Total energy consumption; w1, w2, w3, and w4 are all weighting coefficients.

[0015] As a preferred embodiment, according to the scheduling method of the automatic charging system of the present invention, in the construction step, based on the objective function of the constructed strategy value function, action value functions are constructed for the charging pile and the robotic arm respectively as follows:

[0016] Among them, Q C Q J V represents the motion value function for the charging pile and the robotic arm, respectively. c V J The state-value functions for the charging pile and the robotic arm are respectively, r(s,a) C ,a J ) is the reward function for the charging pile and the robotic arm, E is the expected function, and γ is the discount factor.

[0017] As a preferred embodiment, in the scheduling method of the automatic charging system according to the present invention, the neural network parameter ω is used in the modeling step. C and ωJ Modeling the motion value functions of the charging pile and the robotic arm separately to establish a motion value network. and Using neural network parameters θ C and θ J The policy value functions of the charging pile and the robotic arm are modeled separately to establish a policy network. and

[0018] As a preferred embodiment, in the scheduling method of the automatic charging system according to the present invention, in the training step, the loss functions used to train the motion value networks of the charging pile and the robotic arm constructed using the neural network in the modeling step using the temporal difference algorithm are defined as follows:

[0019] Where E is the expectation function, and R is the dataset collected in the past by the policy. This represents a data sample taken from dataset R, where t represents the current time and t+1 represents the next time. and This indicates that actions are collected based on a strategy, where γ is the discount factor and α is the regularization coefficient. and Indicates the target network.

[0020] As a preferred embodiment, in the scheduling method of the automatic charging system according to the present invention, the loss functions used to train the policy networks of the charging pile and the robotic arm established by the neural network in the modeling step with the objective of maximizing the cumulative reward are defined as follows:

[0021] Where E is the expectation function, R is the dataset collected in the past by the policy, and s t ~R represents the data sample s obtained from dataset R. t , and This indicates that the charging pile and the robotic arm collect actions according to the strategy, where α is the regularization coefficient.

[0022] As a preferred embodiment, the scheduling method of the automatic charging system according to the present invention further includes a safety control step, which restricts the movements of the charging pile and the robotic arm so that the movements of the charging pile and the robotic arm satisfy the constraints.

[0023] As a preferred embodiment, in the scheduling method of the automatic charging system according to the present invention, in the training step, data samples are sampled from the dataset R, and loss functions of the motion value functions of the charging pile and the robotic arm are calculated respectively. and To train the network and update the target network.

[0024] According to a second aspect of the present invention, a scheduling device for an automatic charging system is provided. The automatic charging system includes a charging pile and a robotic arm. The robotic arm performs plug-and-unplug operations on a charging gun on the charging pile to charge an electric vehicle. The scheduling device includes: a construction unit, which constructs a policy value function and an action value function for the charging pile and the robotic arm respectively, and constructs the same reward function for the charging pile and the robotic arm; a modeling unit, which uses a neural network to model the action value function and policy value function of the charging pile and the robotic arm constructed by the construction unit to establish an action value network and a policy network; a training unit, which uses a temporal difference algorithm to train the action value network of the charging pile and the robotic arm established in the modeling step, and trains the policy network of the charging pile and the robotic arm established in the modeling step with the objective of maximizing the cumulative reward; and a parameter update unit, which updates the neural network parameters after the training unit completes the training, so that the policy of the charging pile and the robotic arm is the optimal scheduling policy.

[0025] According to a third aspect of the present invention, a non-transitory storage medium is provided, which stores a computer program that, when executed by a processor, enables the scheduling method of the automatic charging system according to the first aspect of the present invention.

[0026] According to a fourth aspect of the present invention, a computer program product is provided, comprising computer instructions that, when executed by a processor, enable the scheduling method of the automatic charging system according to a first aspect of the present invention.

[0027] The beneficial effects of this invention are as follows: The scheduling method of this invention, through the combination of a multi-agent system and reinforcement learning, transforms the scheduling problem of the suspended track charging system into a problem of finding the optimal strategy using reinforcement learning. The policy neural networks of the suspended track charging piles and the suspended track robotic arm continuously learn from the parking lot operation data, achieving intelligent scheduling of charging, significantly improving the utilization rate and charging efficiency of the charging piles, and shortening the charging waiting time. Furthermore, the scheduling method of this invention for the automatic charging system incorporates a safety control module designed according to constraints into the algorithm, thereby ensuring the rationality and safety of system operation. In addition, due to the diversity and flexibility of the reward function design, this method has high flexibility and scalability, and can adapt to suspended track automatic charging parking lots of different sizes and configurations, providing electric vehicle users with more efficient, convenient, and safe charging services. Attached Figure Description

[0028] Figure 1 illustrates a schematic diagram of an exemplary structure of an automatic charging system according to the present invention.

[0029] Figure 2 illustrates a flowchart of the charging operation process of the automatic charging system according to the present invention.

[0030] Figure 3 illustrates the overall flowchart of the scheduling method for the automatic charging system according to the present invention.

[0031] Figure 4 illustrates a flowchart of the training and execution process of the scheduling method for the automatic charging system according to the present invention. Detailed Implementation

[0032] Exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative configuration of components, numerical representations, and values ​​described in these embodiments does not limit the scope of the invention.

[0033] The scheduling method of the automatic charging system of the present invention can be implemented by the processor in the automatic charging system executing a computer program stored in the memory of the automatic charging system. Alternatively, the automatic charging system can communicate with a server, where a processor in the server executes a computer program stored on the server or in the cloud and feeds back the program execution result to the automatic charging system in real time.

[0034] In this invention, the term "unit" can refer to a software environment, a hardware environment, or a combination of both. In a software environment, the term "unit" refers to a function, application, software module, feature, routine, set of instructions, or program that can be executed by a programmable processor (such as a microprocessor, central processing unit (CPU), or specially designed programmable device) or controller. Memory contains instructions or programs that, when executed by the CPU, cause the CPU to perform operations corresponding to the unit or function. In a hardware environment, the term "unit" refers to a hardware element, circuit, component, physical structure, system, module, or subsystem. According to a particular embodiment, the term "unit" can include mechanical, optical, or electrical components, or any combination thereof. The term "unit" can include active (e.g., transistors) or passive (e.g., capacitors) components. The term "unit" can include a semiconductor device having a substrate and other material layers having various conductivity concentrations. It can include a CPU or programmable processor that can execute programs stored in memory to perform specified functions. The term "unit" can include logic elements (e.g., AND, OR) implemented by transistor circuitry or any other switching circuitry. In the context of a combination of software and hardware environments, the term "unit" or "circuit" refers to any combination of software and hardware environments as described above. Additionally, the terms "element," "component," "part," or "device" may also refer to a "circuit" integrated with or not integrated with packaging material.

[0035] The system architecture of the automatic charging system of the present invention will be described below with reference to the accompanying drawings.

[0036] The automatic charging system of this invention is designed with a network of main tracks and branch tracks, thereby ensuring that intelligent components can move efficiently and safely to any designated parking space. The main track is closed and runs through the entire parking lot, while the branch tracks extending from the main track lead to each parking space, allowing intelligent components to stay next to the parking space without affecting the passage of the main track.

[0037] The intelligent components of the automatic charging system of this invention include a charging pile and a robotic arm, both of which are capable of autonomous movement on the main track and branch tracks. The charging pile can move to a specific parking space according to scheduling instructions to provide power replenishment for electric vehicles; the robotic arm is responsible for inserting and removing the charging gun from the charging pile, and also has the ability to move to any parking space to ensure automation and seamless connection of the charging process.

[0038] An exemplary structure of the automatic charging system of the present invention is shown in Figure 1. Figure 1 illustrates the shape of the track of the automatic charging system and its positional relationship with parking spaces in a parking lot. The automatic charging system of the present invention includes a main track 1 and branch tracks 2, as well as charging piles and robotic arms that can move autonomously on these two tracks. As shown in Figure 1, the main track 1 forms a closed loop and is configured to cover the entire parking lot. Branch tracks 2 extend from the main track 1, and charging spaces 3 are distributed on both sides of each branch track. The branch tracks 2 allow charging piles and robotic arms to stop and operate at the parking spaces without affecting the unobstructed flow of the main track 1.

[0039] Although the main track 1 shown in Figure 1 is a closed loop, the present invention is not limited thereto, and the shape of the main track 1 can also be other closed shapes such as circles or ellipses.

[0040] Furthermore, although the main track and branch tracks of the automatic charging system of the present invention are preferably suspended in the following description to save space and facilitate passage and operation, the present invention is not limited thereto. Depending on the height and layout of the parking lot, the automatic charging system of the present invention can also be applied to ground-mounted tracks with tracks laid on the ground.

[0041] The charging operation process of the automatic charging system of the present invention will be described in detail below.

[0042] Figure 2 illustrates a flowchart of the charging operation process of the automatic charging system according to the present invention. As shown in Figure 2, firstly, in step S101, the electric vehicle enters a parking space in the automatic charging system. After the electric vehicle comes to a stop, it sends a charging request to the automatic charging system in step S102. In step S103, the automatic charging system selects a suitable suspended track charging pile and a suspended track robotic arm. Specifically, after receiving the charging request from the electric vehicle, the automatic charging system selects the most suitable suspended track charging pile and suspended track robotic arm to undertake the charging task of the parking space according to the scheduling method described later.

[0043] In steps S104A and S104B, the selected suspended track charging pile and suspended track robotic arm are moved to the designated parking space, respectively. It should be understood that the present invention does not impose a specific order on the arrival of the suspended track charging pile and the suspended track robotic arm at the designated parking space; they can arrive sequentially or simultaneously.

[0044] In step S105, after both the suspended track charging pile and the suspended track robotic arm have moved along the track to the corresponding branch track of the designated parking space, the suspended track robotic arm will automatically grab the charging gun on the suspended track charging pile and insert it into the electric vehicle. In step S106, it is determined whether the insertion and docking of the charging gun into the electric vehicle by the suspended track robotic arm is complete. If it is determined in step S106 that the insertion and docking is not complete (No in step S106), the process returns to step S105 to continue the insertion operation of the charging gun into the electric vehicle. If it is determined in step S106 that the insertion and docking is complete (Yes in step S106), the process proceeds to steps S107A and S107B.

[0045] In step S107A, the suspended track charging station begins charging the electric vehicle, while in step S107B, the suspended track robotic arm can leave to perform the task of plugging and unplugging charging guns at other parking spaces. Alternatively, the suspended track robotic arm can remain in place waiting for charging to finish, without leaving to perform the task of plugging and unplugging charging guns at other parking spaces. It should be understood that the charging operation of the suspended track charging station and the departure of the suspended track robotic arm can be performed in parallel or sequentially; this invention does not impose any particular limitation on this.

[0046] Next, in step S108, it is determined whether the charging of the electric vehicle by the suspended track charging pile is complete. If it is determined in step S108 that charging is not complete (no in step S108), the process returns to step S107A to continue charging. If it is determined in step S108 that charging is complete (yes in step S108), the process proceeds to step S109. In step S109, the automatic charging system uses a scheduling method to reselect a suitable suspended track robotic arm to arrive at the current parking space. Then, in step S120, after the selected suspended track robotic arm moves along the track to the current parking space, it removes the charging gun and puts it back on the charging pile. Thus, the charging operation is completed, and the charging pile and robotic arm leave the current parking space to perform other charging tasks.

[0047] As an alternative, in step S107B, the suspended track robotic arm can remain in place and wait until charging is complete, then perform the operation of unplugging the charging gun and placing it back on the charging pile in step S120, without leaving to perform the task of plugging and unplugging the charging gun at other parking spaces. In this case, the operation of reselecting the suspended track robotic arm in step S109 will be omitted.

[0048] The scheduling method used in step S103 for selecting a suitable charging pile and robotic arm will be described in detail below.

[0049] To further leverage the advantages of the entire automatic charging system and improve the utilization rate and charging efficiency of parking lot charging piles, this invention provides a scheduling method based on multi-agent reinforcement learning for selecting appropriate charging piles and robotic arms for operation in response to charging requests, thereby achieving intelligent scheduling of charging.

[0050] Reinforcement learning typically refers to an agent learning an optimal policy in a Markov process to maximize cumulative reward by interacting with the environment, performing actions, and receiving feedback (usually rewards or punishments). The concepts involved include a set of states, a set of actions, and a reward.

[0051] In this invention, the state set is defined as including the location and state of each charging pile, the location and state of each robotic arm, and the state of each parking space; the action set includes three actions of the charging pile and the robotic arm, namely, going to the parking space, working, and resting; the reward is the weighted average of the waiting time of all charging requests, the mobile energy consumption of all robots, and the waiting time of vehicles that queued earlier.

[0052] The suspended track charging station and the suspended track robotic arm involved in this invention interact independently with the environment within the shared environment of the automatically charging system created above. They utilize environmental feedback rewards to improve their strategies and obtain higher cumulative rewards. The suspended track charging station and the suspended track robotic arm share the same goal and are in a fully cooperative relationship, therefore receiving the same rewards and having the same reward function. They continuously improve their respective strategies using the feedback rewards.

[0053] In this invention, the scheduling method of the automatic charging system mainly includes a function construction step, a network modeling step, a network training step, and a parameter update step. The scheduling method of the automatic charging system of this invention will be described below with reference to FIG3.

[0054] As shown in Figure 3, firstly, in the function construction step S201, a strategy value function and an action value function are constructed for the charging pile and the robotic arm, respectively. The constructed strategy value function and action value function will be described in detail below.

[0055] Next, in the network modeling step S202, neural networks are used to model the motion value functions and policy value functions of the charging pile and the robotic arm constructed in step S201, in order to establish the motion value network and the policy network. Specifically, two neural networks Q are used to model the motion value functions of the suspended track charging pile and the suspended track robotic arm, respectively, and two neural networks π are used to model the policy value functions of the suspended track charging pile and the suspended track robotic arm, respectively.

[0056] Then, in the network training step S203, the temporal difference algorithm is used to train the action value network of the charging pile and the robotic arm established in step S202, and the policy network of the charging pile and the robotic arm established in step S202 is trained with the goal of maximizing the cumulative reward.

[0057] After the network training is completed in step S203, the neural network parameters are updated in step S204 to make the scheduling strategies of the charging pile and the robotic arm optimal. Specifically, in step S204, the strategies of the suspended track charging pile and the suspended track robotic arm are gradually optimized through soft updates of the neural network parameters.

[0058] The following is a detailed description of each step in the scheduling method of this invention.

[0059] In the automatic charging system of this invention, the suspended track charging piles and suspended track robotic arms are two sets of agents in reinforcement learning, each using different strategies. Their states consist of their position and the progress of the task they are undertaking. The state set s∈S of the entire environment consists of the states of all suspended track charging piles and suspended track robotic arms, and the states of all parking spaces. The actions a of the suspended track charging piles and suspended track robotic arms are... C ,a J ∈A, where action set A includes moving to a specific parking space, performing tasks, and resting. The suspended track charging station and the suspended track robotic arm are two sets of agents that cooperate fully and share the same goal, therefore having the same reward function r(s,a). C ,a J The reward function is preferably represented by the following formula (1):

[0060] Among them, t i (s, a) C a J This indicates that in the current state s, the suspended track charging station performs action a. C Furthermore, when the suspended track robotic arm performs action aJ, the i-th charging request requires a waiting time; q C (a C ) indicates that the suspended track charging station is performing action a. C Total energy consumption; q J (a J ) indicates that the suspended track robotic arm performs the action a. J Total energy consumption; w1, w2, w3, and w4 are all weighting coefficients.

[0061] As a preferred embodiment, the reward function in this invention also considers the waiting time of all charging requests and the system energy consumption. A fourth term is added to the formula to prevent the waiting time of a single request from being too long. The waiting time in the reward function can be calculated based on the moving speed and relative position of the suspended track charging pile and the suspended track robotic arm, and the energy consumption can be calculated based on the unit energy consumption of the suspended track charging pile and the suspended track robotic arm moving a certain distance.

[0062] To maximize cumulative reward, both the suspended track charging station and the suspended track robotic arm aim to learn a strategy for selecting actions from their current states. In this invention, the strategies (strategy functions) for the suspended track charging station and the suspended track robotic arm are represented by π. C (a C |s),π J (a J |s) represents the state S to action a. C and a J The mapping.

[0063] Based on the aforementioned reward function design, for the suspended track charging pile and the suspended track robotic arm, the objective function of the multi-agent reinforcement learning of this invention can be defined as follows: Formula (2) and Formula (3) respectively.

[0064] The objective function above is π C* and π J* This represents the goal to be achieved by training the policy functions of the charging station and the robotic arm, i.e., the optimal policy. and Let H(X) be the expectation function, which represents the average of the values ​​within the square brackets ([]) following it. H(X) is the entropy of X, representing a measure of the randomness of a random variable. In this multi-agent reinforcement learning, H(π) is used. C (a C |s)) and H(π J (a J |s)) represents the degree of randomness of the suspended track charging pile strategy and the suspended track robotic arm strategy in state s. α is the regularization coefficient, used to control the importance of entropy.

[0065] The scheduling method of this invention uses the SAC (Solution-Based Algorithm) offline policy algorithm to train the policies of the suspended track charging station and the suspended track robotic arm. The SAC algorithm employs the idea of ​​maximum entropy reinforcement learning. Therefore, the goal of the multi-agent reinforcement learning in this invention, by introducing entropy and regularization coefficients, is not only to maximize the cumulative reward but also to make the policy more random and exploratory, thereby accelerating the policy learning of the suspended track charging station and the suspended track robotic arm and reducing the possibility of the policy getting trapped in poor local optima.

[0066] Based on the objective function designed above, the policy becomes more random and more exploratory. Therefore, the Bellman equation in reinforcement learning in this invention becomes a soft form. Specifically, the Bellman equation for the action-value function constructed in this invention is shown in Equations 4 and 5 below:

[0067] Among them, Q C Q J These are the motion value functions for the suspended track charging pile and the suspended track robotic arm, respectively. γ is the expected value, which has the same meaning as the expected value function above, and is used to represent the average value of the contents within the square brackets ([]); γ is the discount factor, which is a hyperparameter set according to the situation (artificially set), mainly used to balance immediate rewards and future rewards to avoid infinite rewards; V c V JLet be the state value functions for the suspended track charging station and the suspended track robotic arm, respectively. These functions measure the expected reward that the suspended track charging station and the suspended track robotic arm can obtain from a certain state when adopting the current strategy. The state value functions are expressed as follows:

[0068] To leverage the powerful high-dimensional data learning and nonlinear function approximation capabilities of neural networks, this invention uses two neural networks Q to model the action value functions of the suspended track charging pile and the suspended track robotic arm constructed above, and uses two neural networks π to model the strategy value functions of the suspended track charging pile and the suspended track robotic arm, respectively.

[0069] Specifically, the motion value functions (with parameter ω) of the suspended track charging pile and the suspended track robotic arm are calculated using neural networks Q and π. C ,ω J ) and policy function (parameter θ) C ,θ J To represent (model) and establish the action value network and policy network, the action value network is specifically represented as follows: and Policy network is represented as and These can also be called trained networks, used for subsequent training steps.

[0070] In the scheduling method of the automatic charging system of the present invention, the loss function of the motion value network used to train the motion value functions of the charging pile and the robotic arm modeled by the time-series difference algorithm is defined as follows: Formula 6 and Formula 7:

[0071] Where E is the expectation function, which represents the average value of the contents within square brackets ([]); R is the dataset collected in the past by the strategy; This represents a data sample taken from dataset R, where t represents the current time (also known as the current time step), and t+1 represents the next time (also known as the next time step after the current time step). and This indicates that actions are collected according to the strategy, meaning the actions to be taken by the charging pile and robotic arm in the next moment are determined based on the strategy; similarly, γ is the discount factor, which is a manually set hyperparameter; α is the regularization coefficient mentioned above. Because SAC is a discrete policy algorithm, to make training more stable, this invention preferably uses a target network. and Used to represent the trained network and The goal is to achieve the following: The network parameters in this invention can be updated using a soft update method. Since the soft update method only updates a portion or a certain proportion of the network parameters, it is a slow update, which is more conducive to the stability of neural network training.

[0072] Furthermore, in the scheduling method of the automatic charging system of the present invention, the loss functions of the strategies used to train the policy networks of the charging pile and the robotic arm modeled by the neural network with the goal of maximizing the cumulative reward are defined as shown in Equations 8 and 9 below, respectively:

[0073] Where E is the expectation function, which, as above, represents the average of the contents within the brackets ([]), R is the dataset collected in the past by the strategy, and s t ~R represents a data sample (state s) taken from dataset R. t ), and This indicates that the charging pile and the robotic arm collect actions according to the strategy, and α is the regularization coefficient mentioned above, just like above.

[0074] On the other hand, as a preferred embodiment, the scheduling method of the present invention further includes a safety control step to restrict the movements of the suspended track charging pile and the suspended track robotic arm, ensuring that constraints are met. Specifically, the scheduling method of the present invention additionally inserts a safety control module (unit) to restrict the movements of the suspended track charging pile and the suspended track robotic arm to meet constraints. In the present invention, the main constraints are, for example, that the suspended track charging pile and the suspended track robotic arm moving on the main track must not collide, and that the suspended track charging pile and the suspended track robotic arm must not be interrupted to perform other tasks while they are charging or performing plugging / unplugging operations. Specifically, the safety control module checks the action reward function r(s,aC,aJ) in real time. If a violation of the above constraints occurs, the current reward function r(s,aJ) is adjusted accordingly. C ,a J The policy can be modified, for example, by changing it to a very small negative value to impose a large penalty on the action, so that the policy will avoid violating the constraints in subsequent learning.

[0075] Furthermore, it should be understood that in this invention, the safety control module can be implemented in hardware within the automatic charging system, or in software by method steps executed by the processor in the automatic charging system.

[0076] The specific training and execution process of the scheduling method of the automatic charging system in this invention will be described in detail below.

[0077] The training and execution process of the scheduling algorithm based on multi-agent reinforcement learning for the automatic charging system of the present invention is described with reference to FIG4.

[0078] As shown in Figure 4, the automatic charging system is first initialized in step S301. Specifically, the automatic charging system is initialized in the pre-arranged site using random network parameters ω. C ,ω J Initialize the motion value network of the suspended track charging station and the suspended track robotic arm. and And using random network parameters θ C ,θ J Initialize the policy network for the suspended track charging station and the suspended track robotic arm. and

[0079] Next, in step S302, the target network of the action value network is initialized. Specifically, the same parameter ω is copied. C ,ω J to ω C- ω J- Initialize the target network of the action value network respectively. and

[0080] In step S303, the initial state s1 of the current system environment is obtained. In step S304, the time step t is set to 1. Then, in step S305, it is determined whether t is less than or equal to the set time threshold T. If it is determined in step S305 that t is less than or equal to T (yes in step S305), the process proceeds to step S306. In step S306, the suspended track charging pile and the suspended track robotic arm select their respective actions according to the current strategy. and Next, in step S307, the suspended track charging station and the suspended track robotic arm perform the action selected in step S306 and receive a reward. t and change the environmental state to s t+1 .

[0081] Then, in step S308, the safety control module checks the constraints. Specifically, the actions and environmental states of the suspended track charging pile and the suspended track robotic arm are transmitted to the safety control module to check for constraint violations. If a constraint is violated, the obtained reward is corrected, and a penalty is imposed. Then, in step S309, the collected data samples are... Add the dataset to the data replay pool R (dataset).

[0082] Next, in step S310, data is sampled from the data playback pool R, and the loss function of the action value function is calculated respectively. and The network is trained, and the loss function of each policy is calculated. and The network is then trained and the target network is updated. Preferably, in this invention, network training refers to using the data continuously collected during the above process (which is stored in R), calculating the loss function mentioned above, and then using conventional methods for training neural networks, such as gradient descent, for training.

[0083] After the processing in step S310 is completed, the time step t is incremented by 1, that is, t = t + 1 is set, and the processing returns to step S305. It is determined whether the time step t is less than or equal to the time threshold T. Within the time step t from 1 to T, steps S306 to S310 are executed repeatedly until t is greater than T.

[0084] If it is determined in step S305 that t is greater than T (step S305 is negative), the process proceeds to step S311. In step S311, it is determined whether e is less than E, where e represents the cumulative reward obtained within the current time period T, and E represents the expected (satisfactory) cumulative reward, for example, E is the score set by the system (e.g., subjectively given by a person). If it is determined in step S311 that e is less than E (step S311 is positive), it means the cumulative reward score is not high enough, that is, the cumulative reward obtained within the current time period T has not met the system's expectation, then the process returns to step S303 and starts executing the process again until a satisfactory scheduling strategy for the suspended track charging pile and suspended track robotic arm is obtained. If it is determined in step S311 that e is greater than or equal to E (step S311 is negative), it means the cumulative reward obtained within the current time period T has met the system's expectation, then the process ends. A satisfactory scheduling strategy for the suspended track charging pile and suspended track robotic arm is obtained through training.

[0085] Finally, by softly updating the neural network parameters, the scheduling strategy for the suspended track charging station and the suspended track robotic arm is gradually optimized. The soft-updating method used in this invention is more conducive to the stability of neural network training.

[0086] The above process allows for the pre-training of a scheduling strategy in the current suspended track automatic charging parking lot. Furthermore, data can be continuously collected during subsequent actual operation to train the scheduling strategy in real time, thereby dynamically responding to different situations, reducing the waiting time for charging requests, and improving the charging efficiency of the parking lot.

[0087] In summary, the scheduling method of the automatic charging system of this invention, through the combination of multi-agent systems and reinforcement learning, transforms the scheduling problem of the suspended track charging system into a problem of finding the optimal strategy using reinforcement learning. The policy neural networks of the suspended track charging piles and the suspended track robotic arm continuously learn from the parking lot operation data, achieving intelligent scheduling of charging, significantly improving the utilization rate and charging efficiency of charging piles, and shortening charging waiting time. Furthermore, the scheduling method of the automatic charging system of this invention incorporates a safety control module designed according to constraints into the algorithm, thereby ensuring the rationality and safety of system operation. In addition, due to the diversity and flexibility of the reward function design, this method has high flexibility and scalability, and can adapt to suspended track automatic charging parking lots of different sizes and configurations, providing electric vehicle users with more efficient, convenient, and safe charging services.

[0088] Other implementation methods

[0089] Embodiments of the invention can also be implemented by a computer that reads and executes computer-executable instructions (e.g., one or more programs) recorded on a storage medium (also more fully referred to as a "non-transitory computer-readable storage medium") to perform one or more functions in the above embodiments, and / or includes one or more circuits (e.g., application-specific integrated circuits (ASICs)) for performing one or more functions in the above embodiments. Furthermore, embodiments of the invention can be implemented using a method by which the computer of the system or device, for example, reads and executes the computer-executable instructions from the storage medium to perform one or more functions in the above embodiments, and / or controls the one or more circuits to perform one or more functions in the above embodiments. The computer may include one or more processors (e.g., central processing unit (CPU), microprocessor unit (MPU)) and may include separate computers or a network of separate processors to read and execute the computer-executable instructions. The computer-executable instructions may be provided to the computer, for example, from a network or the storage medium. The storage medium may include, for example, a hard disk, random access memory (RAM), read-only memory (ROM), the memory of a distributed computing system, or an optical disc (such as a compact disc (CD), a digital versatile optical disc (DVD), or a Blu-ray disc (BD)). TM One or more of the following: flash memory devices and memory cards.

[0090] While the present invention has been described above with reference to exemplary embodiments, these embodiments are only for illustrating the technical concept and features of the present invention and should not be construed as limiting the scope of protection of the present invention. Any equivalent variations or modifications made in accordance with the spirit and essence of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A scheduling method for an automatic charging system, characterized in that, The automatic charging system includes a charging pile and a robotic arm. The robotic arm performs plugging and unplugging operations on the charging gun on the charging pile to enable the charging gun to charge the electric vehicle. The scheduling method includes the following steps: In the construction step (S201), the policy value function and action value function are constructed for the charging pile and the robotic arm respectively, and the same reward function is constructed for the charging pile and the robotic arm. In the modeling step (S202), a neural network is used to model the action value function and policy value function of the charging pile and robotic arm constructed in the construction step, so as to establish the action value network and policy network. In the training step (S203), the action value network of the charging pile and the robotic arm established in the modeling step is trained using the temporal difference algorithm, and the policy network of the charging pile and the robotic arm established in the modeling step is trained with the goal of maximizing the cumulative reward. The parameter update step (S204) updates the neural network parameters after the training step is completed, so that the strategies of the charging pile and the robotic arm are the optimal scheduling strategies.

2. The scheduling method for the automatic charging system according to claim 1, characterized in that, The objective function of the strategy value function for building charging piles and robotic arms in the construction step is defined as follows: Where π C* and π J* Let π represent the strategy value functions for the charging station and the robotic arm, respectively. C (a C |s) and π J (a J The objective function of |s), a C ,a J These represent the actions of the charging station and the robotic arm, respectively, while 's' represents the overall environmental state of the automatic charging system. and Let be the expected function. r(s,a C a J ) represents the reward function for the charging station and the robotic arm. H(π C (a C |s)) and H(π J (a J |s)) are respectively π C (a C |s) and π J (a J The entropy of |s) represents the degree of randomness of the charging pile strategy and the robotic arm strategy in state s. α is the regularization coefficient.

3. The scheduling method for the automatic charging system according to claim 2, characterized in that, The reward functions for the charging pile and the robotic arm are defined as follows: Among them, t i (s,a C ,a J This indicates that in state s, the charging station performs action a. C And the robotic arm performs action a J The waiting time for the i-th charging request; q C (a C ) indicates that the charging pile has performed action a. C The total energy consumption, and q J (a J ) indicates that the robotic arm performs the action a. J Total energy consumption; w1, w2, w3, and w4 are all weighting coefficients.

4. The scheduling method for the automatic charging system according to claim 2, characterized in that, In the construction step, based on the objective function of the constructed strategy value function, action value functions are constructed for the charging pile and the robotic arm respectively as follows: Among them, Q C Q J V represents the motion value function for the charging pile and the robotic arm, respectively. c V J The state-value functions for the charging pile and the robotic arm are respectively, r(s,a) C ,a J ) is the reward function for the charging pile and the robotic arm, E is the expected function, and γ is the discount factor.

5. The scheduling method for the automatic charging system according to claim 1, characterized in that, In the modeling step, the neural network parameter ω is used. C and ω J Modeling the motion value functions of the charging pile and the robotic arm separately to establish a motion value network. and Using neural network parameters θ C and θ J The policy value functions of the charging pile and the robotic arm are modeled separately to establish a policy network. and 6. The scheduling method for the automatic charging system according to claim 5, characterized in that, In the training step, the loss functions used to train the motion value networks of the charging pile and the robotic arm constructed using the neural network in the modeling step using the temporal difference algorithm are defined as follows: Where E is the expectation function, and R is the dataset collected in the past by the policy. This represents a data sample taken from dataset R, where t represents the current time and t+1 represents the next time. and This indicates that actions are collected based on a strategy, where γ is the discount factor and α is the regularization coefficient. and Indicates the target network.

7. The scheduling method for the automatic charging system according to claim 5, characterized in that, In the training step, the loss functions used to train the policy networks of the charging pile and the robotic arm established using the neural network in the modeling step, with the objective of maximizing cumulative reward, are defined as follows: Where E is the expectation function, R is the dataset collected in the past by the policy, and s t ~R represents the data sample s obtained from dataset R. t , and This indicates that the charging pile and the robotic arm collect actions according to the strategy, where α is the regularization coefficient.

8. The scheduling method for the automatic charging system according to claim 1, characterized in that, The scheduling method further includes: Safety control measures restrict the movements of the charging pile and the robotic arm to ensure that their movements meet the constraints.

9. The scheduling method for the automatic charging system according to claim 6, characterized in that, In the training step, data samples are sampled from dataset R, and loss functions are calculated for the motion value functions of the charging pile and the robotic arm, respectively. and To train the network and update the target network.

10. A scheduling device for an automatic charging system, characterized in that, The automatic charging system includes a charging pile and a robotic arm. The robotic arm performs plugging and unplugging operations on the charging gun on the charging pile to enable the charging gun to charge the electric vehicle. The scheduling device includes: The construction unit constructs its own policy value function and action value function for the charging pile and the robotic arm respectively, and constructs the same reward function for the charging pile and the robotic arm. The modeling unit uses neural networks to model the motion value function and policy value function of the charging pile and robotic arm constructed by the building unit, so as to establish the motion value network and policy network. The training unit uses the temporal difference algorithm to train the action value network of the charging pile and the robotic arm established in the modeling step, and trains the policy network of the charging pile and the robotic arm established in the modeling step with the goal of maximizing the cumulative reward. The parameter update unit updates the neural network parameters after the training unit completes the training, so that the strategy of the charging pile and the robotic arm is the optimal scheduling strategy.

11. A non-transitory storage medium storing a computer program that, when executed by a processor, enables the scheduling method of the automatic charging system according to any one of claims 1-9.

12. A computer program product comprising computer instructions that, when executed by a processor, enable the scheduling method of the automatic charging system according to any one of claims 1-9.

Images