Network defense method and apparatus, device and medium

Abstract

The present application relates to the technical field of network security, and in particular, to a network defense method and apparatus, a device and a medium, which are used for effectively coping with constantly-changing network threats and improving the network defense effect. The method comprises: constructing a deep reinforcement learning (DRL) model, setting a state space and an action space of the DRL model, and, on the basis of a key defense indicator of a network system, constructing a reward function; using a soft actor-critic (SAC) algorithm to train the DRL model; and acquiring current network state information of the network system, inputting the current network state information into the trained DRL model to obtain an optimal network defense measure, and executing the optimal network defense measure. In this way, by using the deep reinforcement learning technology, a protection mechanism can be dynamically adjusted to cope with constantly-changing network threats. In addition, when a potential threat is detected, effective network defense measures are automatically executed, such that the system can switch from passive monitoring to active defense.

Description

A network defense method, device, equipment and medium

[0001] Cross-reference to related applications

[0002] This application claims priority to Chinese Patent Application No. 202411639918.0, filed on November 18, 2024, with the State Intellectual Property Office of the People's Republic of China, entitled "A Network Defense Method, Apparatus, Device and Medium", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of network security technology, and in particular to a network defense method, apparatus, device, and medium. Background Technology

[0004] In today's era of rapid digitalization and internetization, cybersecurity has become a key global concern. With technological advancements, cyberattack methods are constantly evolving and upgrading, posing serious challenges to network defense. These methods include malware, phishing attacks, distributed denial-of-service (DDoS) attacks, and zero-day vulnerabilities.

[0005] In existing technologies, firewalls are commonly used for network defense. Specifically, they control the flow of data packets by setting access control lists (ACLs) and network communication rules, blocking unauthorized access and potential network threats, and dividing data flows into allowed and prohibited ones, thus providing a physical and logical line of defense.

[0006] However, the above methods are static defense methods. Static rules cannot flexibly respond to constantly changing network threats, resulting in poor network defense effectiveness. Moreover, managing and updating rules is relatively complex and it is difficult to achieve dynamic adaptation. Summary of the Invention

[0007] This application provides a network defense method, apparatus, device, and medium to dynamically adjust the protection mechanism, respond to constantly changing network threats, and improve the effectiveness of network defense.

[0008] The specific technical solutions provided in this application are as follows:

[0009] Firstly, a network defense method is provided, the method including:

[0010] Construct a deep reinforcement learning (DRL) model, define the state space and action space of the DRL model, and construct a reward function based on the key defense indicators of the network system;

[0011] The Flexible Action-Evaluation (SAC) algorithm was used to train the DRL model;

[0012] Obtain the current network state information of the network system, input the current network state information into the trained DRL model, obtain the optimal network defense measures, and execute the optimal network defense measures.

[0013] In one possible embodiment, each state in the state space represents comprehensive information about a network situation of the network system, including but not limited to: network traffic characteristics, system security level, and system alarms.

[0014] Each action in the action space corresponds to a network defense measure.

[0015] In one possible embodiment, key defense metrics include: attack detection rate, attack false positive rate, network system performance, and security recovery time.

[0016] Based on the key defense indicators of the network system, the reward function is constructed as follows:

[0017] A reward function is constructed based on attack detection rate, attack false positive rate, network system performance, security recovery time, and the weights of each indicator.

[0018] Using the above method, the reward function incorporates key defense metrics, which are then weighted to ensure that the agent can achieve the best balance between accuracy, efficiency, and stability when optimizing defense strategies.

[0019] In one possible embodiment, the SAC algorithm is used to train the DRL model, including:

[0020] Initialize the parameters of the state value function, the target network, the action value function, and the policy network;

[0021] Establish an experience replay buffer. A tuple in the experience replay buffer includes: current state, current action, current reward, and next time step state.

[0022] Based on the experience replay buffer, each parameter is iteratively updated until the training termination condition is met, thus obtaining the trained DRL model.

[0023] The above methods can improve the accuracy of the DRL model, thereby enhancing the effectiveness of network defense.

[0024] In one possible embodiment, establishing an experience playback buffer includes:

[0025] The policy network samples a current action in the current state;

[0026] Perform the current action to obtain the current reward and the state at the next moment;

[0027] Store the current state, current action, current reward, and next state as a tuple in the experience replay buffer.

[0028] The above methods enable intelligent agents to adjust their behavior in real time in complex network environments and store observed state transitions.

[0029] In one possible implementation, the parameters are iteratively updated based on the experience replay buffer, including:

[0030] Randomly draw sample data from the experience replay buffer;

[0031] Calculate the target state value, and update the parameters of the state value function by minimizing the loss function of the state value function using gradient descent.

[0032] Calculate the target Q-value, and update the parameters of the action value function by minimizing the loss function of the action value function using gradient descent.

[0033] The parameters of the policy network are updated by minimizing the loss function of the policy network and using gradient descent.

[0034] The parameters of the target network are updated based on the parameters of the state value function and the parameter update rate of the target network.

[0035] By updating the parameters using the above methods, the accuracy of the state value function, action value function, policy network, and target network is ensured, in order to cope with constantly changing network threats.

[0036] Secondly, this application provides a network defense device, the device comprising:

[0037] The building module is used to construct a deep reinforcement learning (DRL) model, define the state space and action space of the DRL model, and construct a reward function based on the key defense metrics of the network system.

[0038] The training module is used to train the DRL model using the Flexible Action-Evaluation (SAC) algorithm.

[0039] The determination module is used to obtain the current network state information of the network system, input the current network state information into the trained DRL model, obtain the optimal network defense measures, and execute the optimal network defense measures.

[0040] In one possible embodiment, each state in the state space represents comprehensive information about a network situation of the network system, including but not limited to: network traffic characteristics, system security level, and system alarms.

[0041] Each action in the action space corresponds to a network defense measure.

[0042] In one possible embodiment, key defense metrics include: attack detection rate, attack false positive rate, network system performance, and security recovery time.

[0043] When constructing the reward function based on the key defense indicators of the network system, the construction module is also used for:

[0044] A reward function is constructed based on attack detection rate, attack false positive rate, network system performance, security recovery time, and the weights of each indicator.

[0045] In one possible embodiment, when training the DRL model using the SAC algorithm, the training module is further configured to:

[0046] Initialize the parameters of the state value function, the target network, the action value function, and the policy network;

[0047] Establish an experience replay buffer. A tuple in the experience replay buffer includes: current state, current action, current reward, and next time step state.

[0048] Based on the experience replay buffer, each parameter is iteratively updated until the training termination condition is met, thus obtaining the trained DRL model.

[0049] In one possible embodiment, when establishing the experience replay buffer, the training module is also used for:

[0050] The policy network samples a current action in the current state;

[0051] Perform the current action to obtain the current reward and the state at the next moment;

[0052] Store the current state, current action, current reward, and next state as a tuple in the experience replay buffer.

[0053] In one possible embodiment, when iteratively updating the parameters based on the experience replay buffer, the training module is also used to:

[0054] Randomly draw sample data from the experience replay buffer;

[0055] Calculate the target state value, and update the parameters of the state value function by minimizing the loss function of the state value function using gradient descent.

[0056] Calculate the target Q-value, and update the parameters of the action value function by minimizing the loss function of the action value function using gradient descent.

[0057] The parameters of the policy network are updated by minimizing the loss function of the policy network and using gradient descent.

[0058] The parameters of the target network are updated based on the parameters of the state value function and the parameter update rate of the target network.

[0059] Thirdly, this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method described in any one of the first aspects above.

[0060] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any one of the first aspects above.

[0061] Fifthly, this application provides a computer program product comprising: computer program code, which, when run on a computer, causes the computer to perform the method described in any one of the first aspects.

[0062] In this embodiment, a deep reinforcement learning (DRL) model is constructed, defining its state and action spaces. A reward function is built based on key defense metrics of the network system. The DRL model is then trained using the Flexible Action-Evaluation (SAC) algorithm. Finally, the current network state information is obtained and input into the trained DRL model to obtain and execute the optimal network defense measures. This utilizes deep reinforcement learning technology to learn and update defense strategies in real time, dynamically adjusting protection mechanisms to address evolving network threats. Through interactive learning, the system continuously optimizes rule generation and adjustment, enhancing adaptability and response speed. Furthermore, automated processing and analysis of security data reduces reliance on manual operations, optimizes incident response processes, significantly reduces management burden, and lowers operating costs. Additionally, when a potential threat is detected, effective network defense measures are automatically executed, not only issuing warnings but also proactively responding, enabling the system to shift from passive monitoring to proactive defense and improving its defense capabilities. Attached Figure Description

[0063] Figure 1 is a schematic diagram of the application scenario in the embodiments of this application;

[0064] Figure 2 is a flowchart illustrating the implementation of a network defense method provided in an embodiment of this application;

[0065] Figure 3 is a schematic diagram of the DRL model in the embodiment of this application;

[0066] Figure 4 is a schematic diagram of the model application in the embodiments of this application;

[0067] Figure 5 is a schematic diagram of a network defense device provided in an embodiment of this application;

[0068] Figure 6 is a schematic diagram of the structure of the electronic device in an embodiment of this application. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Unless otherwise specified, the embodiments and features in the embodiments of this application can be arbitrarily combined with each other. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0070] The terms "first" and "second" in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the term "comprising" and any variations thereof are intended to cover non-exclusive protection. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. The term "multiple" in this application can mean at least two, for example, two, three, or more, and the embodiments of this application do not impose limitations.

[0071] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These embodiments should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description. It should be noted that in the embodiments of this application, certain existing industry solutions such as software, components, and models may be mentioned. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solutions of this application, and do not imply that the applicant has already used or necessarily used such solutions.

[0072] The acquisition, transmission, storage, and use of data in this application all comply with the requirements of relevant national laws and regulations.

[0073] To facilitate understanding of the technical solutions provided in the embodiments of this application, some key terms used in the embodiments of this application will be explained below:

[0074] Distributed Denial of Service (DDoS) attack: is an attack launched by multiple malicious devices, which aims to overwhelm a target network or service with a large number of requests or traffic, causing it to become overloaded and eventually paralyzed, thereby denying service requests from legitimate users.

[0075] The Internet of Things (IoT) is a network of numerous devices and objects that connect and exchange data through embedded sensors, software, and other technologies to enable intelligent interaction and management.

[0076] Access Control Lists (ACLs) are sets of rules in a network device that specify whether data flows are allowed or denied through that device. They enforce security controls and traffic management policies by matching the source, destination, and other protocol-specific information of data packets.

[0077] Deep Reinforcement Learning (DRL) is a technique that combines deep learning and reinforcement learning. It uses deep neural networks to process high-dimensional perceptual inputs and learns decision-making strategies that maximize cumulative rewards through interaction with the environment.

[0078] Soft Actor-Critic (SAC) Algorithm: The SAC algorithm is a deep learning-based reinforcement learning method that aims to achieve a balance between exploration and exploitation while maintaining high policy entropy (i.e., randomness). The SAC algorithm is suitable for tasks with continuous action spaces and has proven to perform well on various benchmarks.

[0079] Before introducing the network defense method provided in the embodiments of this application, for ease of understanding, the technical background of the embodiments of this application will be described in detail below.

[0080] In today's rapidly developing digital and internet-based era, cybersecurity has become a key global concern. With technological advancements, cyberattack methods are constantly evolving and upgrading, posing serious challenges to network defense. These methods include malware, phishing attacks, DDoS attacks, and zero-day vulnerabilities. Furthermore, the widespread application of cloud computing and the Internet of Things (IoT) further increases the complexity of network defense, demanding higher levels of real-time response capabilities and intelligence from network defense methods.

[0081] In existing technologies, the following network defense methods are commonly used:

[0082] Method 1: Firewall-based network defense method.

[0083] By setting ACLs and network communication rules to control the flow of data packets, unauthorized access and potential network threats can be blocked, and data flows can be divided into allowed and prohibited ones, providing a physical and logical line of defense. However, static rules may not adapt to rapidly changing attack patterns, resulting in poor network defense effectiveness, and managing and updating rules can be complex, making dynamic adaptation difficult.

[0084] Method 2: Network defense method based on Intrusion Detection System (IDS).

[0085] IDS systems identify abnormal behavior patterns by analyzing network traffic. However, they can only provide warnings and do not have the ability to prevent attacks, which limits their application in dynamic defense.

[0086] Method 3: Network defense method based on endpoint protection.

[0087] Malware and viruses can be prevented by deploying antivirus software, anti-malware, and other security tools at each network endpoint. However, this requires installing and updating protection software separately at each endpoint, resulting in high management costs and limited protection against new and unknown viruses.

[0088] Method 4: Network defense method based on security information and event management (SIEM).

[0089] By collecting and analyzing logs and event information from across the network, SIEM provides real-time security alerts and incident responses. SIEM helps organizations centrally manage their security alerts and improves the accuracy of incident detection by correlating data from different sources. While SIEM systems offer comprehensive security insights, their configuration and maintenance are relatively complex, requiring specialized security teams to effectively manage and parse large amounts of data. They can also incur significant performance overhead, all of which can limit their ability to quickly adapt to new threats.

[0090] Method 5: Network defense method based on zero-trust architecture.

[0091] Zero Trust architecture is an emerging cybersecurity concept whose core principle is "never trust, always verify." Under this architecture, all access requests, whether internal or external, must undergo strict authentication and authorization. Zero Trust architecture ensures that only fully authenticated and authorized users and devices can access resources through fine-grained access control and authentication mechanisms. While Zero Trust architecture can significantly improve network security, implementing it requires a large-scale restructuring of existing networks and security policies, which can involve complex technical challenges and high costs. Furthermore, the strict controls of Zero Trust architecture can impact user experience and productivity, especially in environments requiring frequent access to multiple systems and applications.

[0092] In the context of rapid digital development, cybersecurity faces increasing challenges. Although existing network defense methods provide multi-layered protection, they are no longer effective in dealing with increasingly complex and automated network attack methods.

[0093] In view of this, this application provides a network defense method, apparatus, device, and medium. A deep reinforcement learning (DRL) model is constructed, its state space and action space are defined, and a reward function is built based on key defense indicators of the network system. The DRL model is then trained using the SAC algorithm. Finally, the current network state information of the network system is obtained and input into the trained DRL model to obtain and execute the optimal network defense measures. This utilizes deep reinforcement learning technology to learn and update defense strategies in real time, dynamically adjusting protection mechanisms to cope with constantly changing network threats. Through interactive learning, the system can continuously optimize rule generation and adjustment, thereby enhancing adaptability and response speed. Furthermore, automated processing and analysis of security data reduces reliance on manual operation, optimizes event response processes, significantly reduces management burden, and lowers operating costs. In addition, when a potential threat is detected, effective network defense measures are automatically executed, not only issuing warnings of anomalies but also proactively responding, enabling the system to shift from passive monitoring to proactive defense and improving defense capabilities.

[0094] The preferred embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0095] Referring to Figure 1, this is a schematic diagram of an application scenario in an embodiment of this application. It includes a network defense system 110 and an external system 120. The network defense system 110 includes a monitoring and recording layer 1101, a first intelligent decision support layer 1102, an interface layer 1103, a second intelligent decision support layer 1104, an intelligent defense engine layer 1105, a policy and management layer 1106, and a defense execution layer 1107. The external system 120 includes, but is not limited to, wide area network (WAN), cloud protection, and endpoint protection.

[0096] The monitoring and logging layer 1101 is used for logging and anomaly handling, and transmits the network data collected, such as network logs and anomaly data, to the first intelligent decision support layer.

[0097] The first intelligent decision support layer 1102 includes a DRL model, which is used to analyze network data, identify threats, and formulate strategies.

[0098] Interface layer 1103 includes: data interface, permission interface and linkage interface, etc.

[0099] The second intelligent decision support layer 1104 includes a DRL model and is used to execute optimal network defense measures.

[0100] The intelligent defense engine layer 1105 includes: a security prediction module, a security monitoring module, and a security defense module.

[0101] The strategy and management layer 1106 includes: strategy management, linked device management, access control, and effect display.

[0102] Defense execution layer 1107 includes: application blocking, network blocking, terminal blocking, etc.

[0103] Referring to Figure 2, which is a flowchart of an implementation method for a network defense method provided in this application, the specific implementation process of the method is as follows:

[0104] Step 20: Construct a deep reinforcement learning (DRL) model, define the state space and action space of the DRL model, and construct a reward function based on the key defense metrics of the network system.

[0105] In this embodiment, each state in the state space represents comprehensive information about a network situation, including network traffic characteristics, system security level, and system alarms. This comprehensive information is not limited in this respect. Each action in the action space corresponds to a network defense measure.

[0106] Network traffic characteristics include, but are not limited to, traffic volume, traffic type, traffic source and destination, traffic behavior patterns, and protocol information.

[0107] Network defense measures include, but are not limited to: adjusting security configurations and activating defense mechanisms. Adjusting security configurations includes, but is not limited to, adjusting firewall rules (e.g., modifying ACLs to block or allow communication from specific IP addresses or ports) and updating security policies (e.g., adding defenses against SQL injection or cross-site scripting attacks). Activating defense mechanisms includes, but is not limited to: activating IDS (e.g., enabling or adjusting IDS sensitivity and alert settings) and starting isolation procedures (e.g., isolating detected malicious activities or infected systems).

[0108] Referring to Figure 3, which is a schematic diagram of the DRL model in this embodiment, the DRL model includes an agent and an environment. The agent, as the main decision-making entity, is responsible for analyzing network state information obtained from the environment and taking corresponding actions to maintain network security. The environment represents the network system operated by the agent, including various network devices, services, data flows, and potential security threats. The agent receives states from the environment, selects and executes actions based on the states, and receives rewards and new states from the environment. Through continuous interaction, the agent can accurately assess the security of various network behaviors and adjust and optimize its response strategies.

[0109] In this system, state provides the agent with comprehensive information about the network situation, including but not limited to network traffic characteristics, system security level, and system alarms. State helps the agent understand the real-time security status of the environment and forms the basis of its decision-making process. Action is the network defense measure taken by the agent based on the state. Reward is a quantitative feedback on the effectiveness of the agent's actions.

[0110] Additionally, it is worth noting that, in order to facilitate processing by intelligent agents, the information constituting the state (such as network traffic characteristics, system security levels, and system alarms) is encoded into numerical or category labels.

[0111] In this embodiment of the application, when performing environment modeling, real-time data is collected from network devices (such as routers, switches, servers, etc.) and security systems (such as firewalls and intrusion detection systems, etc.), features (such as network traffic statistics, abnormal behavior indicators and security alarms, etc.) are identified and extracted, and these features are used to build an environment model to ensure that the environment model can reflect the real-time status of the network system and potential security threats.

[0112] In this application embodiment, key defense indicators include: attack detection rate, attack false alarm rate, network system performance, and security recovery time.

[0113] Among them, the attack detection rate index represents the proportion of real attacks that are successfully detected; the attack false positive rate index represents the proportion of normal behavior that is incorrectly marked as an attack; the attack false positive rate index represents the impact of network defense measures on network performance; and the security recovery time index represents the time required for the network system to recover from an attack to a normal state.

[0114] In this embodiment of the application, a reward function is constructed based on the attack detection rate index, the attack false positive rate index, the network system performance index, the security recovery time index, and the weights corresponding to each index.

[0115] Specifically, the reward function can be expressed as: R(s) t ,a t )=α×TPR(s t ,a t )-β×FPR(s t ,a t )-γ×Impact(s t ,a t )-δ×Recovery(s t ,a t )

[0116] Wherein, α is the attack detection rate weight, used to reward the agent's ability to detect real attacks. Increasing the value of α can incentivize the agent to focus more on accurately identifying and preventing real attacks. β is the false alarm rate weight, used to penalize the agent for mistakenly identifying normal behavior as an attack. Increasing the value of β helps reduce interference with normal network activities. γ is the network system performance weight, used to penalize network performance degradation caused by network defense measures. Adjusting the value of γ can help find a balance between network security and network performance. δ is the security recovery time weight, used to incentivize the agent to take network defense measures that can restore the network to its pre-attack normal state as quickly as possible. TPR(s) t ,a t ) is the attack detection rate metric, FPR(s) t ,a t Impact(s) is an attack false positive rate metric. t ,a t Recovery(s) is a network system performance indicator. t ,a t () is an indicator of safe recovery time.

[0117] In this way, the learning process of the agent is guided by a reward function that incorporates key defense metrics. These metrics are adjusted by weights to ensure that the agent can achieve the best balance between accuracy, efficiency, and stability when optimizing defense strategies.

[0118] Step 21: Train the DRL model using the Flexible Action-Evaluation (SAC) algorithm.

[0119] Network defense can be viewed as a continuous control task. The SAC algorithm's advantages in handling complex tasks with continuous action spaces meet the requirements for application in dynamic network defense scenarios. By integrating the maximum entropy principle and policy learning, the SAC algorithm achieves high efficiency, stability, and wide applicability in continuous control tasks, especially demonstrating its unique value and advantages in dynamic and constantly changing network defense scenarios. Training DRL models using the SAC algorithm can improve network defense effectiveness.

[0120] Optionally, in this application embodiment, a possible implementation for training the DRL model using the SAC algorithm is provided, specifically by performing the following operations:

[0121] Step 210: Initialize the parameters of the state value function, the target network, the action value function, and the policy network.

[0122] The state value function represents the expected cumulative reward that can be obtained by taking a specific strategy in a specific state, and the action value function represents the expected value of the long-term cumulative reward that the agent can obtain after taking a specific action in a specific state. In this embodiment, there are two action value functions. The target network is a copy of the state value function. The parameters of the target network are not updated in real time during training, but are copied from the state value function intermittently. This can provide a relatively stable target state value and reduce variance and instability during training.

[0123] In this embodiment, the parameters ψ of the state value function are initialized, the parameters ψ' of the target network are initialized, the parameters θ1 and θ2 of the two action value functions are initialized, and the parameters φ of the policy network are initialized.

[0124] Step 211: Establish an experience replay buffer.

[0125] A tuple in the experience replay buffer includes: current state, current action, current reward, and next state.

[0126] Optionally, in this application embodiment, a possible implementation for establishing an experience replay buffer is provided, specifically by performing the following operations:

[0127] Step 2110: The policy network samples a current action in the current state.

[0128] In this embodiment of the application, the policy network π φ (a t |s t In the current state s t Downsample a current action a t .

[0129] Step 2111: Execute the current action to obtain the current reward and the state at the next moment.

[0130] In this embodiment of the application, the current action a is executed. t The current reward r(s) is obtained from observation. t a t ) and the next time step state s t+1 .

[0131] Step 2112: Store the current state, current action, current reward, and next moment state as a tuple in the experience replay buffer.

[0132] In this embodiment of the application, the current state s t Current action a t Current reward r(s) t a t), and the next state s t+1 As a tuple (s t a t ,r(s t a t ), s t+1 Stored in the experience playback buffer D.

[0133] This enables agents to adjust their behavior in real time in complex network environments and store observed state transitions.

[0134] Step 212: Based on the experience replay buffer, iteratively update each parameter until the training termination condition is met, and obtain the trained DRL model.

[0135] In this embodiment, sample data collection and parameter optimization are repeated continuously until the training termination condition is met.

[0136] Optionally, in this embodiment of the application, a possible implementation is provided for iteratively updating each parameter based on the experience replay buffer, specifically by performing the following operations:

[0137] Step 2120: Randomly draw sample data from the experience replay buffer.

[0138] Step 2121: Calculate the target state value by minimizing the loss function of the state value function and updating the parameters of the state value function using gradient descent.

[0139] The target state value is the value of the target state value function, which is also the expected value of the target network.

[0140] In this embodiment, the target state value is obtained by subtracting the log probability of the policy from the action value function. Based on the difference between the estimated state value function and the target state value, the loss function of the state value function is obtained. The parameters of the state value function are updated using gradient descent.

[0141] In this embodiment of the application, the target state value is represented as:

[0142] Among them, Q θ It is the action-value function, used to estimate the expected reward under a given state and action, logπ. φ (a t |s t ) is the policy network that takes action a. t The natural logarithmic probability is used to introduce an entropy reward to encourage the policy network to explore more actions, which can help balance exploration.

[0143] In this embodiment of the application, the loss function for the state value function with respect to parameter ψ is defined as:

[0144] in, This represents the expected operation, sampled into the experience playback buffer D, where s t This represents the state randomly sampled from the experience replay buffer. The expectation operation ensures that the loss is evaluated across the entire dataset, making the learning process more stable and generalizable.

[0145] in, It is the mean squared error term, used to quantify the current state value function V. ψ and the target value function V target The difference between them, the squared term guarantees the non-negativity of the loss, and emphasizes the importance of larger errors. V ψ (s t ) is an estimate of the current state value function, calculated using the parameter ψ, representing the state s. t The expected long-term returns. V target (s t ) is the estimate of the target state value function, i.e., the target state value. It is typically calculated using a stable target network to help stabilize the training process and reduce variance in parameter updates. 1 / 2 is a constant factor used to simplify subsequent gradient calculations, resulting in a more concise expression for the gradient calculation.

[0146] In this way, by optimizing the loss function, the parameters of the state value function can be adjusted, which can better predict the true value of each state, thereby improving the overall performance of the policy.

[0147] In this embodiment, gradient descent is used to update the parameters of the state value function. The update process is expressed as: ψ←ψ-η□ ψ J V (ψ).

[0148] Where ψ is the parameter of the state value function, η is the learning rate, and □ ψ J V (ψ) is the loss function J V (ψ) is the gradient of ψ.

[0149] In this way, ψ is adjusted according to the gradient of the loss function to improve the accuracy of the state value function.

[0150] Step 2122: Calculate the target Q value and update the parameters of the action value function by minimizing the loss function of the action value function using gradient descent.

[0151] In this embodiment, the target Q value is obtained based on the target state value function of the next state and the reward function of the current state. The loss function of the action value function is obtained based on the difference between the predicted Q value and the target Q value. The parameters of the action value function are updated using gradient descent.

[0152] In this embodiment of the application, the target Q value is represented as:

[0153] Where r(s) t ,a t ) is in state s t Next, execute action a t The immediate reward received. γ is the discount factor used to discount the present value of future rewards. V ψ' (s t+1 ) is in the next state s t+1 The target value function is typically evaluated using the target network parameters ψ' to provide a stable learning objective.

[0154] In this embodiment of the application, the loss function for the action value function with respect to parameter θ is defined as:

[0155] in, This indicates that the expected calculation is performed on the samples in the experience playback buffer D. This is the mean squared error term, used to quantify the difference between the predicted Q-value and the target Q-value. Q θ (s t ,a t To estimate the state s using the current parameter θ t Take action a t The predicted Q value, The target Q value.

[0156] In this embodiment, gradient descent is used to update the parameters θ1 and θ2 of the action value function. The update process is expressed as follows:

[0157] Where, θ i Let be the parameters of the value function for the i-th action, and η be the learning rate, which controls the step size for parameter updates. The loss function J Q (θ) with respect to parameter θ i The gradient.

[0158] Step 2123: Update the parameters of the policy network by minimizing the loss function of the policy network using gradient descent.

[0159] In this embodiment of the application, the loss function of the policy network is defined as:

[0160] in, This represents the state s used in the experience playback buffer D. t Calculate the expected value, D KL KL divergence is a metric that measures the difference between two probability distributions and is used to quantify the impact of policy network π. φ The difference between the generated probability distribution and the target probability distribution. π φ (·|s t ) is a policy network defined by parameter φ in a given state s t The probability distribution of the next pair of actions. exp(Q) θ (s t ,·)-logZ θ (s t The value function Q based on the current action is defined. θ The adjusted target strategy is achieved by exponentializing the Q-value and subtracting a normalization term. θ (s t ,·) represents the state s t The Q-value for all possible actions is given by logZ. θ (s t ) is the logarithm of the normalization constant, ensuring that the sum of the Q values ​​after the exponential transformation is 1, thus forming a valid probability distribution.

[0161] In this embodiment, the parameter φ of the strategy network is updated using gradient descent. The update process is expressed as: φ←φ-η□φJ π (φ).

[0162] Where φ represents the parameters of the policy network, η is the learning rate, and the learning rate is used to control the step size of parameter updates. π (φ) is the loss function J π (φ) is the gradient of the parameter φ.

[0163] In this way, by minimizing the difference between the policy and the target policy, defense decisions are optimized, defense policies are learned and updated in real time, and protection mechanisms are dynamically adjusted to cope with ever-changing network threats.

[0164] Step 2124: Update the parameters of the target network based on the parameters of the state value function and the parameter update rate of the target network.

[0165] In this embodiment of the application, the update process of the target network parameter ψ' is expressed as: ψ'←τψ+(1-τ)ψ'.

[0166] Where ψ' represents the parameters of the target network, τ is the soft update coefficient, a decimal close to 0 and greater than 0, which controls the speed of the target network parameter update. A smaller τ value can ensure that the target network parameters are updated more slowly, which helps the learning process to proceed smoothly, and ψ represents the parameters of the state value function.

[0167] In this way, the parameters of the target network are updated periodically to ensure the continuous optimization and stability of the strategy. By slowly updating the target network parameters ψ', relatively stable parameters can be used in the calculation of the target value function. A stable target value function helps to provide reliable target state values, and reliable target state values ​​guarantee the accuracy of optimizing the state value function.

[0168] Step 22: Obtain the current network state information of the network system, input the current network state information into the trained DRL model, obtain the optimal network defense measures, and execute the optimal network defense measures.

[0169] In this embodiment, the current network state information of the network system is obtained, and the current network state information and historical reward feedback are used as input parameters and input into the trained DRL model to obtain the optimal network defense measures corresponding to the current network state information, and then the optimal network defense measures are executed.

[0170] Among them, historical reward feedback is a quantitative feedback from the environment on the effectiveness of the agent's previous network defense measures.

[0171] For example, referring to Figure 4, which is a schematic diagram of model application in an embodiment of this application, the current network status information and historical reward feedback are used as input parameters and input into the DRL model. The DRL model outputs the optimal network defense measures.

[0172] Based on the same inventive concept, this application provides a network defense device. Referring to Figure 5, the device includes:

[0173] Module 501 is used to build a deep reinforcement learning (DRL) model, define the state space and action space of the DRL model, and construct a reward function based on the key defense indicators of the network system.

[0174] Training module 502 is used to train the DRL model using the Flexible Action-Evaluation (SAC) algorithm;

[0175] The determination module 503 is used to obtain the current network state information of the network system, input the current network state information into the trained DRL model, obtain the optimal network defense measures, and execute the optimal network defense measures.

[0176] In one possible embodiment, each state in the state space represents comprehensive information about a network situation of the network system, including but not limited to: network traffic characteristics, system security level, and system alarms.

[0177] Each action in the action space corresponds to a network defense measure.

[0178] In one possible embodiment, key defense metrics include: attack detection rate, attack false positive rate, network system performance, and security recovery time.

[0179] When constructing the reward function based on the key defense indicators of the network system, the construction module 501 is also used for:

[0180] A reward function is constructed based on attack detection rate, attack false positive rate, network system performance, security recovery time, and the weights of each indicator.

[0181] In one possible embodiment, when training the DRL model using the SAC algorithm, the training module 502 is further configured to:

[0182] Initialize the parameters of the state value function, the target network, the action value function, and the policy network;

[0183] Establish an experience replay buffer. A tuple in the experience replay buffer includes: current state, current action, current reward, and next time step state.

[0184] Based on the experience replay buffer, each parameter is iteratively updated until the training termination condition is met, thus obtaining the trained DRL model.

[0185] In one possible embodiment, when establishing the experience replay buffer, the training module 502 is further configured to:

[0186] The policy network samples a current action in the current state;

[0187] Perform the current action to obtain the current reward and the state at the next moment;

[0188] Store the current state, current action, current reward, and next state as a tuple in the experience replay buffer.

[0189] In one possible embodiment, when iteratively updating the parameters based on the experience replay buffer, the training module 502 is further configured to:

[0190] Randomly draw sample data from the experience replay buffer;

[0191] Calculate the target state value, and update the parameters of the state value function by minimizing the loss function of the state value function using gradient descent.

[0192] Calculate the target Q-value, and update the parameters of the action value function by minimizing the loss function of the action value function using gradient descent.

[0193] The parameters of the policy network are updated by minimizing the loss function of the policy network and using gradient descent.

[0194] The parameters of the target network are updated based on the parameters of the state value function and the parameter update rate of the target network.

[0195] Based on the above embodiments, Figure 6 shows a schematic diagram of the structure of the electronic device in the present application.

[0196] This application provides an electronic device that may include a processor 610 (Center Processing Unit, CPU), a memory 620, an input device 630, and an output device 640. The input device 630 may include a keyboard, a mouse, a touch screen, etc., and the output device 640 may include a display device, such as a liquid crystal display (LCD) or a cathode ray tube (CRT).

[0197] The memory 620 may include read-only memory (ROM) and random access memory (RAM), and provides the processor 610 with program instructions and data stored in the memory 620. In this embodiment, the memory 620 may be used to store the program of any network defense method in this embodiment.

[0198] The processor 610 executes any of the network defense methods in the embodiments of this application according to the program instructions stored in the memory 620.

[0199] Based on the above embodiments, this application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the network defense method in any of the above method embodiments.

[0200] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0201] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0202] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0203] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0204] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A cyber defense method, comprising: include: Construct a deep reinforcement learning (DRL) model, define the state space and action space of the DRL model, and construct a reward function based on the key defense indicators of the network system; The DRL model is trained using the Flexible Action-Evaluation (SAC) algorithm. The current network state information of the network system is obtained and input into the trained DRL model to obtain the optimal network defense measures, and the optimal network defense measures are executed.

2. The method of claim 1, wherein, Each state in the state space represents comprehensive information about a network situation of the network system, including but not limited to: network traffic characteristics, system security level, and system alarms; Each action in the action space corresponds to a network defense measure.

3. The method of claim 1, wherein, The key defense metrics include: attack detection rate, attack false positive rate, network system performance, and security recovery time. The reward function constructed based on the key defense indicators of the network system includes: A reward function is constructed based on the attack detection rate metric, the attack false positive rate metric, the network system performance metric, the security recovery time metric, and the weights corresponding to each metric.

4. The method of claim 1, wherein, The step of training the DRL model using the SAC algorithm includes: Initialize the parameters of the state value function, the target network, the action value function, and the policy network; An experience replay buffer is established, wherein a tuple in the experience replay buffer includes: current state, current action, current reward, and next time step state; Based on the experience replay buffer, each parameter is iteratively updated until the training termination condition is met, and the trained DRL model is obtained.

5. The method of claim 4, wherein, The establishment of the experience replay buffer includes: The policy network samples a current action in the current state; Perform the current action to obtain the current reward and the state at the next moment. The current state, the current action, the current reward, and the next state are stored as a tuple in the experience replay buffer.

6. The method of claim 1, wherein, The iterative update of each parameter based on the experience replay buffer includes: Randomly draw sample data from the experience replay buffer; Calculate the target state value, and update the parameters of the state value function by minimizing the loss function of the state value function using gradient descent. Calculate the target Q value, and update the parameters of the action value function by minimizing the loss function of the action value function using gradient descent. The parameters of the policy network are updated by minimizing the loss function of the policy network and using gradient descent. The parameters of the target network are updated based on the parameters of the state value function and the parameter update rate of the target network.

7. A network defense device characterized by comprising: include: The module is used to build a deep reinforcement learning (DRL) model, define the state space and action space of the DRL model, and construct a reward function based on the key defense indicators of the network system. The training module is used to train the DRL model using the Flexible Action-Evaluation (SAC) algorithm. The determination module is used to obtain the current network state information of the network system, input the current network state information into the trained DRL model, obtain the optimal network defense measures, and execute the optimal network defense measures.

8. The apparatus of claim 7, wherein, Each state in the state space represents comprehensive information about a network situation of the network system, including but not limited to: network traffic characteristics, system security level, and system alarms; Each action in the action space corresponds to a network defense measure.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer readable storage medium having stored thereon a computer program, characterized in that: When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images