Service fault recovery method and system for microservice platform, and storage medium

Abstract

The present invention provides a service fault recovery method and system for a microservice platform, a storage medium, and a device. The method comprises: performing real-time analysis on a service state, and collecting service state index data; determining a fault service, collecting fault state data, and acquiring structured data of service call graph information on the basis of the service state index data and the fault state data; constructing a reinforcement learning fault recovery model, inputting the structured data of the service call graph information into the reinforcement learning fault recovery model, and training the reinforcement learning fault recovery model on the basis of known historical data; and using the trained reinforcement learning fault recovery model to generate a recovery strategy of the fault service, and executing the recovery strategy to recover a fault. The service fault recovery method of the present invention improves fault recovery efficiency.

Description

Microservice platform service failure recovery methods, systems, and storage media Technical Field

[0001] This invention belongs to the field of automated operation and maintenance, and particularly relates to a method, system, storage medium and device for service failure recovery of a microservice platform. Background Technology

[0002] In modern microservice application systems, the complexity of the number of services and call relationships places higher demands on service operation and maintenance. Currently, with the development of artificial intelligence technology, various intelligent operation and maintenance models have begun to play a role in areas such as service anomaly handling and fault analysis. However, research on related technologies in the field of fault recovery is relatively lagging behind, and the current mainstream method is still to handle fault situations through human operation.

[0003] Current fault recovery methods mainly focus on two directions: one is to propose targeted recovery actions based on the specific characteristics of the current system and the defined fault scenarios to achieve fault recovery during the operation and maintenance process. For example, a recovery scheme can be used to restore the system to an effective state after a fault. This recovery scheme can be a checkpoint-based rollback strategy, which periodically creates a valid snapshot of the system state and returns the system to an effective state when a fault occurs. However, this approach is usually system-specific and may place a significant burden on system designers and operators.

[0004] Another fault recovery method involves using several specific basic recovery methods as alternatives. For different error states presented by the system, a strategy is selected based on the system's state and the constructed recovery actions. The corresponding alternative means are then chosen for recovery. This method requires rationally arranging potential repair operations based on strategies such as state operation rules. Strategies may include recursively trying actions with the lowest remaining overhead. One basic recovery method is simple rebooting. A large number of faults can be repaired using rebooting, even when the exact cause is unknown. This recovery approach can be applied at different levels and is currently used in many important systems.

[0005] However, this simple strategy may be insufficient in real-world environments due to inaccurate fault location, recurring faults, or failed repair operations. Therefore, it may lead to poor fault recovery. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a method, system, storage medium, and device for service fault recovery on a microservice platform. It constructs a reinforcement learning fault recovery model based on a deep reinforcement learning algorithm, and automatically selects and executes recovery actions based on real-time monitoring of service status indicators and fault status data, thereby achieving automatic fault recovery, improving fault recovery efficiency, and enhancing service availability.

[0007] This invention is achieved through the following technical solution:

[0008] Perform real-time analysis of service status and collect service status indicator data;

[0009] Identify the faulty service, collect fault status data, and based on the service status index data and fault status data, obtain structured data of service call graph information;

[0010] A reinforcement learning fault recovery model is constructed by inputting the structured data of the service call graph information into the reinforcement learning fault recovery model and training the reinforcement learning fault recovery model based on known historical data.

[0011] Using the trained reinforcement learning fault recovery model, a recovery strategy for the fault service is generated, and the recovery strategy is executed to recover from the fault.

[0012] Optional,

[0013] The service status metrics data include: inter-node metrics data, container metrics data, and server metrics data.

[0014] Optional,

[0015] The structured data used to obtain service call graph information based on service status indicator data and fault status data includes:

[0016] Based on the inter-node metric data, obtain the call relationship between services and construct the service call chain;

[0017] Container metric data and server metric data are aggregated into service node information, and combined with the service call chain, structured data of service call graph information is constructed.

[0018] Optional,

[0019] The process of aggregating container metric data and server metric data into service node information, and combining this with the service call chain to construct structured data for the service call graph includes:

[0020] By querying container metrics and server metrics, we can obtain the physical machine information of each service in the microservice where the faulty service is located.

[0021] Based on the physical machine information of each service, container indicator data and server indicator data are aggregated into service node information. Combined with the service call chain of the microservice project, structured data of service call graph information composed of node feature vectors and adjacency matrices is constructed.

[0022] Optional,

[0023] The construction of the reinforcement learning fault recovery model includes:

[0024] Define the state space, action space, and reward of the reinforcement learning fault recovery model.

[0025] Optional,

[0026] The step of generating a recovery strategy for the fault service using the trained reinforcement learning fault recovery model and executing the recovery strategy to recover from the fault includes:

[0027] Use system monitoring tools to obtain service status metrics data from the target system in real time;

[0028] Use anomaly detection and fault location tools to identify faulty services and obtain fault status data;

[0029] Based on service status index data and fault status data, a recovery strategy for the faulty service is generated and executed using a trained reinforcement learning fault recovery model.

[0030] The present invention also provides a microservice platform service failure recovery system for implementing the aforementioned method, the system comprising:

[0031] The data acquisition module is used to analyze the service status in real time and collect service status indicator data;

[0032] The fault information processing module is used to identify faulty services, collect fault status data, and obtain structured data of service call graph information based on the service status index data and fault status data.

[0033] The reinforcement learning module is used to construct a reinforcement learning fault recovery model. The structured data of the service call graph information is input into the reinforcement learning fault recovery model, and the reinforcement learning fault recovery model is trained based on known historical data.

[0034] The fault recovery module is used to generate a recovery strategy for the fault service using the trained reinforcement learning fault recovery model, and to execute the recovery strategy to recover the fault.

[0035] Optional,

[0036] The fault information processing module is also configured to:

[0037] Based on the inter-node metric data, obtain the call relationship between services and construct the service call chain;

[0038] Container metric data and server metric data are aggregated into service node information, and combined with the service call chain, structured data of service call graph information is constructed.

[0039] Optional,

[0040] The system is also used for:

[0041] By querying container metrics and server metrics, we can obtain the physical machine information of each service in the microservice where the faulty service is located.

[0042] Based on the physical machine information of each service, container indicator data and server indicator data are aggregated into service node information. Combined with the service call chain of the microservice project, structured data of service call graph information composed of node feature vectors and adjacency matrices is constructed.

[0043] Optional,

[0044] The fault recovery module is also configured to:

[0045] Use system monitoring tools to obtain service status metrics data from the target system in real time;

[0046] Use anomaly detection and fault location tools to identify faulty services and obtain fault status data;

[0047] Based on service status index data and fault status data, a recovery strategy for the faulty service is generated and executed using a trained reinforcement learning fault recovery model.

[0048] The present invention also provides a computer-readable storage medium storing one or more programs, which, when executed, can implement the aforementioned microservice platform service failure recovery method.

[0049] The present invention also provides a device, including a processor, a communication interface, a computer-readable storage medium, and a communication bus; wherein the processor, the communication interface, and the computer-readable storage medium communicate with each other through the communication bus;

[0050] The processor is used to execute programs stored in a computer-readable storage medium.

[0051] Compared with the prior art, the present invention has the following advantages:

[0052] 1. The microservice platform service fault recovery method proposed in this invention constructs a reinforcement learning fault recovery model based on a deep reinforcement learning algorithm. Through real-time monitoring and analysis, it obtains service status indicators and fault status information. After a fault is detected, it automatically selects and executes recovery actions to achieve automatic fault recovery, improves operation and maintenance efficiency, and thus improves service availability.

[0053] 2. The current state of the system is described by the operation and maintenance indicators and call relationship graph of the microservice system. The fault recovery problem under the microservice architecture is modeled as a fully observable Markov decision process. The strategy of selecting recovery actions is fitted by graph convolutional network to improve the availability of the microservice system.

[0054] 3. The reinforcement learning fault recovery model is trained by reinforcement learning algorithm, and the learning effect of the agent is stabilized and accelerated by priority experience replay mechanism. The model learns from historical fault scenarios and recovery actions, thereby optimizing the decision strategy of the reinforcement learning fault recovery model.

[0055] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 shows a flowchart illustrating the service failure recovery method of the microservice platform;

[0058] Figure 2 shows a schematic block diagram of the service failure recovery system of the microservice platform.

[0059] Figure 3 shows a schematic diagram of the automatic service failure recovery framework according to an embodiment of the present invention;

[0060] Figure 4 is a schematic diagram of the structure of a device according to an embodiment of the present invention. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] Referring to Figure 1, the method of the present invention includes:

[0063] S1. Perform real-time analysis of service status and collect service status indicator data;

[0064] The service status metrics data include: inter-node metrics data, container metrics data, and server metrics data.

[0065] S2. Identify the faulty service, collect fault status data, and obtain structured data of service call graph information based on the service status index data and fault status data.

[0066] Among them, based on the indicator data between nodes, the call relationship between services is obtained, and a service call chain is constructed;

[0067] The container metric data and server metric data are aggregated into service node information, and combined with the service call chain, structured data of service call graph information is constructed.

[0068] Specifically, by querying container metrics and server metrics, the physical machine information of each service in the microservice where the faulty service is located can be obtained;

[0069] Based on the physical machine information of each service, container metrics and server metrics are aggregated into service node information. Combined with the service call chain of the microservice project, structured data of service call graph information composed of node feature vectors and adjacency matrices is constructed.

[0070] S3. Construct a reinforcement learning fault recovery model. Input the structured data of the service call graph information into the reinforcement learning fault recovery model, and train the reinforcement learning fault recovery model based on known historical data.

[0071] The construction of the reinforcement learning fault recovery model includes:

[0072] Define the state space, action space, and reward of the reinforcement learning fault recovery model.

[0073] S4. Using the trained reinforcement learning fault recovery model, generate a recovery strategy for the fault service and execute the recovery strategy to recover the fault.

[0074] Among these measures, service status metrics data are obtained in real time from the target system through system monitoring tools.

[0075] Use anomaly detection and fault location tools to identify faulty services and obtain fault status data;

[0076] Based on service status index data and fault status data, a recovery strategy for the faulty service is generated and executed using a trained reinforcement learning fault recovery model.

[0077] Specifically,

[0078] 1. Conduct real-time analysis of service status and collect service status indicator data.

[0079] For the worker nodes and individual microservices on the microservice platform, tools such as monitoring and analysis, anomaly detection, and root cause analysis are used to perform real-time analysis of service status and collect and store information such as service status indicators.

[0080] In this embodiment, tools such as Prometheus and Istio can be used to perform real-time analysis of service status and collect service status indicators. Service status indicators include, but are not limited to, service call relationship related indicators and related indicators of the physical machine where the service is located.

[0081] The service status metrics data also include: inter-node metrics data, container metrics data, and server metrics data.

[0082] 2. Identify the faulty service, collect fault status data, and based on the service status index data and fault status data, obtain structured data of service call graph information.

[0083] The structured data used to obtain service call graph information based on service status indicator data and fault status data includes:

[0084] Based on the inter-node metric data, obtain the call relationship between services and construct the service call chain;

[0085] Container metric data and server metric data are aggregated into service node information, and combined with the service call chain, structured data of service call graph information is constructed.

[0086] Container metric data and server metric data are aggregated into service node information. Combined with the service call chain, structured data of service call graph information is constructed, including:

[0087] By querying container metrics and server metrics, we can obtain the physical machine information of each service in the microservice where the faulty service is located.

[0088] Based on the physical machine information of each service, container metrics and server metrics are aggregated into service node information. Combined with the service call chain of the microservice project, structured data of service call graph information composed of node feature vectors and adjacency matrices is constructed.

[0089] When a service on the platform fails, the faulty service is identified based on existing anomaly detection and fault location tools, and fault status data is collected. By processing the service status metrics and fault status data, structured data of service call graph information is obtained.

[0090] In this embodiment, anomaly detection tools include, but are not limited to: FVAE, anomaly-transformer, DCdetector, etc.; fault location tools include, but are not limited to: Dapper, Pinpoint, Microscope, AutoMap, etc.

[0091] The service status metric data and fault status data are processed to obtain structured data of service call graph information, including:

[0092] S201. Query the metrics corresponding to the faulty service in the inter-node metrics, obtain the service data packet information of all services under the microservice project where the faulty service is located, determine the starting service node and the ending service node based on the data packet information, and then obtain the call relationship between services, thereby constructing the service call chain of the microservice project.

[0093] S202. By querying container metrics and server metrics, determine the service name and its physical machine address under the corresponding namespace, and obtain the physical machine to which each service in the microservice belongs. Based on the physical machine information of each service, aggregate the container metrics and server metrics into the information of that service node, and combine it with the service call chain of the microservice project to construct structured data of service call graph information composed of the feature vectors and adjacency matrices of the nodes.

[0094] In this embodiment, by querying the TCP data transmission volume and Request transmission volume corresponding to the faulty service in the inter-node metrics, the received and sent Request and TCP packet information of all services under the microservice project can be obtained. Analyzing the obtained received and sent Request and TCP packet information yields the "destination_workload" and "source_workload" attributes. Based on these attributes, the starting and ending service nodes can be determined, thereby obtaining the call relationships between services and constructing the service call chain of the microservice project.

[0095] By querying the CPU usage time of each service in the microservice containing the faulty service, the query results for "container" and "instance" return the service name and physical machine address under the corresponding namespace, respectively, thus obtaining the physical machine to which each service belongs. Based on the physical machine information of each service, container metrics and server metrics are aggregated into the information of the service node. Combined with the service call chain of the microservice project, a structured data of service call graph information composed of node feature vectors and adjacency matrices is constructed. The feature matrix X and adjacency matrix A are shown in formula (1):

[0096] In this matrix, the number of rows m represents the number of nodes, the number of columns n represents the number of features of each node, and the adjacency matrix represents the information about the nodes' adjacency.

[0097] III. Constructing a reinforcement learning fault recovery model.

[0098] Construct a reinforcement learning-based fault recovery model, using structured data of the service call graph information as input data f. t The data is input into the reinforcement learning fault recovery model, and the model is trained using known historical data.

[0099] Constructing a reinforcement learning fault recovery model includes defining the state space, action space, and reward of the reinforcement learning fault recovery model.

[0100] 1. Define the state space.

[0101] The state space consists of the feature vectors f of m nodes in the current microservice project. t Error node e obtained based on fault status data j The action a performed in the previous step t-1 And the information r of the recovery result is used to describe it.

[0102] Define the state space S at time t. t As follows: S t =(f t ,e j ,a t-1 ,r)

[0103] In this embodiment, the feature matrix X and the adjacency matrix A are used to represent the feature vectors f of m nodes in the current microservice project. t :

[0104] These key performance indicators for operation and maintenance, which intuitively reflect the fault status, can accurately describe the state of the system and construct a state space.

[0105] 2. Define the action space.

[0106] Action space is the set of all possible actions that an agent can take in a given state. The reinforcement learning fault recovery model outputs that the agent selects a recovery action based on the current state, thereby changing the environment to enter the next state.

[0107] The design of the action space is crucial for the control and fault recovery of microservice systems. A well-designed action space can reduce the complexity of system control, thereby improving problem-solving. At the same time, the construction of the action space needs to be sufficiently complete to ensure effective problem-solving. For example, in some cases, certain actions may not be obvious, but they may be essential for solving the problem. Therefore, the action space should be constructed to provide diverse and precise control for problem-solving.

[0108] In this embodiment, the action space consists of optional recovery methods, including but not limited to: micro-reboot, service instance migration, service instance horizontal scaling (scale out), and service instance vertical scaling (scale up).

[0109] Micro-Reboot: Fine-grained restart of individual application components. It is used for well-isolated stateless components, where all important application states (including current resource limits, host machine status, etc.) are stored in a dedicated state store. When an operation is performed, the service instance is restarted directly according to the stored state.

[0110] Service instance migration (Migrate): Considering the resource utilization of different worker nodes, migrate the current service / component to a worker node in good condition.

[0111] Horizontal scaling of service instances: Proportional scaling of replicas of the current service instance outwards on other nodes while maintaining the current instance state.

[0112] Scale up service instance: Without changing the number of replicas of the current service instance, add more CPU and memory resource quotas to the service instance to vertically scale its performance.

[0113] Service instance rollback: During the continuous integration of a microservice system, service instances are also being updated and changed. The rollback operation can roll back the current replica instance to a previous deployment version.

[0114] These specific recovery methods constitute an action space, from which the agent determines which action is more conducive to improving the state.

[0115] 3. Reward design.

[0116] Reward design is a crucial component for successful agent learning. It serves as an "action guide" for the agent's actions, reflecting evaluation metrics for specific recovery actions. Depending on the agent's goals, rewards can be considered from multiple perspectives, including but not limited to: recovery result, recovery time, and recovery cost.

[0117] Recovery Result: After taking this action in a fault state, whether the environment transitions to the next state and is no longer considered a fault state is a direct measure of the recovery effect.

[0118] Recovery Time: The time cost of the steps required to restore a system from a faulty state to a normal state. The time taken for the system recovery process can directly affect the business benefits of the system.

[0119] Recovery Cost: The total cost incurred by all API functions involved in the recovery operation. Recovery operations such as starting a new instance will incur additional performance overhead costs. Different recovery operations mean different costs, which is also a factor to consider.

[0120] In this embodiment, recovery result (RR), recovery time (RT), and recovery cost (RC) are used as three criteria for evaluating an action. The reward or penalty for the action can be represented as follows: Reward action =RR+RT+RC

[0121] 4. Reinforcement learning fault recovery model training.

[0122] The structured data of the service call graph information is used as input data and fed into the reinforcement learning fault recovery model. The reinforcement learning fault recovery model is then trained by combining it with known historical data.

[0123] The reinforcement learning network consists of two graph convolutional layers with ReLU activation functions and two fully connected layers. The graph convolutional layers can effectively aggregate the information around the node to the current node, fully reflecting the calling relationship. The information of the graph structure is extracted through the two graph convolutional layers, and the extracted graph structure information data is flattened into one dimension. The data is then mapped to the action space through the two fully connected layers of the network.

[0124] By training a reinforcement learning fault recovery model using a fault recovery algorithm, the recovery strategy for selecting recovery actions is optimized, enabling the action selection of the reinforcement learning fault recovery model to find a balance between "exploration and exploitation".

[0125] In this embodiment, the fault recovery algorithm training process includes:

[0126] Initialize the learning rate α and the network parameters θ.

[0127] Initialize the training batch B for each training step, the number of updates U for each training batch, and the size N for each batch;

[0128] Initialize the Boltamann policy temperature τ; # Used to adjust the randomness of the policy during decision-making.

[0129] The empirical memory is initialized with the largest batch size K, and ε and η are initialized; #ε and η are exploration parameters used to control the smoothness and intensity of priority calculation.

[0130] for m = 1, ..., MAX STEPS do: #Perform multiple steps of exploratory training in each experiment

[0131] Collect and store h experiences using the current strategy (s) i ,a i ,r i ,s′ i ,terminal); #The algorithm collects a certain amount (h) of experience by interacting with the environment, which is then used for training.

[0132] Where, r i For state s i Next, take action a i The immediate reward obtained afterward is calculated based on the reward function; s′ i For state s i Next, take action a i The new state to which the user transitions is determined based on the state transition function and the previous state and action; a i For state s′ i Next, the action selected based on the current strategy.

[0133] When executing a policy, the algorithm updates the state and reward based on the environment's response.

[0134] for b = 1 to B do:

[0135] Extract the real experience of batch b from the experience replay memory;

[0136] for u = 1, ..., U do: # The number of parameter updates performed in each training batch. Each update adjusts the network parameters based on the samples in the batch.

[0137] for i = 1, ..., N do: # Iterate through the samples in each training batch, process each sample and update the network parameters.

[0138] #Calculate the target Q value for each sample:

[0139] Where, when s′ i At termination, δ s ′ i =0, otherwise 1;

[0140] #Calculate the absolute TD error for each sample:

[0141] end for;

[0142] #Calculate the loss using MSE:

[0143] #Update network parameters:

[0144] Calculate the priority of each sample:

[0145] Experience replay memory is updated based on the new priority;

[0146] end for

[0147] end for

[0148] Attenuation τ

[0149] end for

[0150] In this embodiment, by training a reinforcement learning fault recovery model through a fault recovery algorithm, the recovery strategy can be optimized, thereby enabling better selection of recovery actions.

[0151] Fourth, use the trained reinforcement learning fault recovery model to generate and execute recovery strategies.

[0152] Deploy and configure reinforcement learning fault recovery models in the service platform to achieve rapid fault recovery.

[0153] In this embodiment, service status indicator data is obtained in real time from the target system through system monitoring tools, and faulty services are identified and fault status data is obtained using anomaly detection and fault location tools. Based on the service status indicator data and fault status data, a recovery strategy is automatically generated using a trained reinforcement learning fault recovery model. According to the generated recovery strategy, specific recovery actions are selected and executed to achieve automatic recovery of service faults.

[0154] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0155] Referring to Figure 2, the structure of a microservice platform service fault recovery system for implementing the above method is shown, including a data acquisition module, a fault information processing module, a reinforcement learning module, and a fault recovery module.

[0156] The data acquisition module is used to perform real-time analysis of service status and collect service status indicator data.

[0157] The fault information processing module is used to determine the faulty service, collect fault status data, and obtain structured data of service call graph information based on the service status index data and fault status data.

[0158] The reinforcement learning module is used to construct a reinforcement learning fault recovery model, input the structured data of the service call graph information into the reinforcement learning fault recovery model, and train the reinforcement learning fault recovery model based on known historical data.

[0159] The fault recovery module is used to generate a recovery strategy for the fault service using the trained reinforcement learning fault recovery model, and execute the recovery strategy to recover the fault.

[0160] Figure 3 is a schematic diagram of the automatic service failure recovery framework according to an embodiment of the present invention. The framework obtains indicator information from the target system; performs anomaly detection on the target system, locates the fault based on the anomaly list, and collects error status information; utilizes a reinforcement learning fault recovery model to generate a recovery strategy for the faulty service, and selects recovery actions to execute based on the recovery strategy to perform error recovery. During the anomaly check, service anomaly logs are recorded, including anomaly information and recovery status, for use in optimizing the generation of the recovery strategy.

[0161] Furthermore, embodiments of the present invention also provide a microservice platform service failure recovery device, comprising:

[0162] The data acquisition module performs real-time analysis of service status and collects service status indicator data, including: inter-node indicator data, container indicator data, and server indicator data.

[0163] The fault information processing module identifies the faulty service, collects fault status data, and obtains structured data of service call graph information based on the service status indicator data and fault status data. Specifically, based on the inter-node indicator data, it obtains the call relationship between services and constructs a service call chain. It aggregates container indicator data and server indicator data into service node information and combines it with the service call chain to construct structured data of service call graph information.

[0164] The reinforcement learning module constructs a reinforcement learning fault recovery model by inputting structured data of the service call graph information into the reinforcement learning fault recovery model and training the model based on known historical data. The construction of the reinforcement learning fault recovery model includes defining the state space, action space, and reward of the model.

[0165] The fault recovery module utilizes the trained reinforcement learning fault recovery model to generate a recovery strategy for the faulty service and executes the recovery strategy to recover from the fault. Specifically, it acquires service status indicator data in real time from the target system using system monitoring tools; identifies the faulty service and acquires fault status data using anomaly detection and fault location tools; and generates and executes the recovery strategy for the faulty service based on the service status indicator data and the fault status data using the trained reinforcement learning fault recovery model.

[0166] Based on the same inventive concept, the present invention also provides a computer-readable storage medium storing one or more programs, which, when executed, can implement the aforementioned microservice platform service failure recovery method.

[0167] As shown in Figure 4, this embodiment of the invention also provides a device, including a processor, a communication interface, a memory, and a communication bus. The processor, communication interface, and memory communicate with each other via the communication bus.

[0168] The memory is a computer-readable storage medium used to store one or more programs.

[0169] The processor is configured to execute a program stored in a computer-readable storage medium.

[0170] The computer-readable storage medium may be included in the device / apparatus described in the above embodiments; or it may exist independently and not assembled into the device / apparatus.

[0171] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A microservice platform service failure recovery method, characterized in that, include: Perform real-time analysis of service status and collect service status indicator data; Identify the faulty service, collect fault status data, and based on the service status index data and fault status data, obtain structured data of service call graph information; A reinforcement learning fault recovery model is constructed by inputting the structured data of the service call graph information into the reinforcement learning fault recovery model and training the reinforcement learning fault recovery model based on known historical data. Using the trained reinforcement learning fault recovery model, a recovery strategy for the fault service is generated, and the recovery strategy is executed to recover from the fault.

2. The method of claim 1, wherein, The service status metrics data include: inter-node metrics data, container metrics data, and server metrics data.

3. The method of claim 1, wherein, The structured data used to obtain service call graph information based on service status indicator data and fault status data includes: Based on the inter-node metric data, obtain the call relationship between services and construct the service call chain; Container metric data and server metric data are aggregated into service node information, and combined with the service call chain, structured data of service call graph information is constructed.

4. The method of claim 3, wherein, The process of aggregating container metric data and server metric data into service node information, and combining this with the service call chain to construct structured data for the service call graph includes: By querying container metrics and server metrics, we can obtain the physical machine information of each service in the microservice where the faulty service is located. Based on the physical machine information of each service, container indicator data and server indicator data are aggregated into service node information. Combined with the service call chain of the microservice project, structured data of service call graph information composed of node feature vectors and adjacency matrices is constructed.

5. The method according to any one of claims 1 to 4, characterized in that, The construction of the reinforcement learning fault recovery model includes: Define the state space, action space, and reward of the reinforcement learning fault recovery model.

6. The method according to any one of claims 1 to 4, characterized in that, The step of generating a recovery strategy for the fault service using the trained reinforcement learning fault recovery model and executing the recovery strategy to recover from the fault includes: Use system monitoring tools to obtain service status metrics data from the target system in real time; Use anomaly detection and fault location tools to identify faulty services and obtain fault status data; Based on service status index data and fault status data, a recovery strategy for the faulty service is generated and executed using a trained reinforcement learning fault recovery model.

7. A microservices platform service failure recovery system, characterized in that, The system includes: The data acquisition module is used to analyze the service status in real time and collect service status indicator data; The fault information processing module is used to identify faulty services, collect fault status data, and obtain structured data of service call graph information based on the service status index data and fault status data. The reinforcement learning module is used to construct a reinforcement learning fault recovery model. The structured data of the service call graph information is input into the reinforcement learning fault recovery model, and the reinforcement learning fault recovery model is trained based on known historical data. The fault recovery module is used to generate a recovery strategy for the fault service using the trained reinforcement learning fault recovery model, and to execute the recovery strategy to recover the fault.

8. The system of claim 7, wherein, The fault information processing module is also configured to: Based on the inter-node metric data, obtain the call relationship between services and construct the service call chain; Container metric data and server metric data are aggregated into service node information, and combined with the service call chain, structured data of service call graph information is constructed.

9. The system of claim 8, wherein, The system is also used for: By querying container metrics and server metrics, we can obtain the physical machine information of each service in the microservice where the faulty service is located. Based on the physical machine information of each service, container indicator data and server indicator data are aggregated into service node information. Combined with the service call chain of the microservice project, structured data of service call graph information composed of node feature vectors and adjacency matrices is constructed.

10. The system according to any one of claims 7 to 9, characterized in that, The fault recovery module is also configured to: Use system monitoring tools to obtain service status metrics data from the target system in real time; Use anomaly detection and fault location tools to identify faulty services and obtain fault status data; Based on service status index data and fault status data, a recovery strategy for the faulty service is generated and executed using a trained reinforcement learning fault recovery model.

11. A computer-readable storage medium storing one or more programs, characterized in that, When one or more of these programs are executed, the microservice platform service failure recovery method described in any one of claims 1-6 can be implemented. 12.An electronic device comprising a processor, a communication interface, the computer readable storage medium of claim 11, and a communication bus; wherein, The processor, communication interface, and computer-readable storage medium communicate electronically with each other via a communication bus; characterized in that, The processor is used to execute programs stored in a computer-readable storage medium.

Images