Multi-cloud service dependency analysis method and apparatus, and computer device

Abstract

A multi-cloud service dependency analysis method, comprising: acquiring service discovery information and service invocation data of a plurality of cloud environments (S200); on the basis of the service discovery information and the service invocation data, constructing a service dependency graph (S400); on the basis of the service dependency graph, determining the importance of each service, and on the basis of the importance, performing screening to obtain Top N target services (S600); and identifying dependent services and strongly connected components of the Top N target services (S800).

Description

Multi-cloud service depends on analytical methods, devices, and computer equipment.

[0001] Cross-reference to related applications

[0002] This application claims priority to Chinese patent application No. 2024117004287, filed on November 26, 2024, entitled "Multi-cloud service dependency analysis method, apparatus and computer device", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of cloud computing technology, and in particular to a method, apparatus, computer device, computer-readable storage medium, and computer program product for multi-cloud service dependency analysis. Background Technology

[0004] With the widespread adoption of cloud computing and the rise of multi-cloud strategies, enterprises are increasingly inclined to deploy their businesses on different cloud platforms to fully leverage the advantages of various cloud service providers, improve business flexibility, and enhance disaster recovery capabilities. However, the complexity of multi-cloud environments also brings unprecedented challenges, especially in service dependency management.

[0005] Existing service dependency analysis methods are mostly designed for microservices and often rely on manual configuration and analysis, making it difficult to adapt to rapidly changing microservice architectures and dynamic service instances, resulting in low efficiency in dependency analysis. Summary of the Invention

[0006] According to various embodiments of this application, an efficient multi-cloud service dependency analysis method, apparatus, computer device, computer-readable storage medium, and computer program product are provided.

[0007] In a first aspect, this application provides a multi-cloud service dependency analysis method, including:

[0008] Obtain service discovery information and service call data from multiple cloud environments;

[0009] Construct a service dependency graph based on service discovery information and service call data;

[0010] Based on the service dependency graph, the importance of each service is determined, and the Top N target services are selected based on their importance.

[0011] Identify the dependent services and strongly connected components of the Top N target services.

[0012] As an embodiment of the multi-cloud service dependency analysis method described in this application, the method further includes:

[0013] Integrate the service discovery information, service call data, dependent services and strongly connected components of the target service within a preset time period to obtain the load time series data of the target service.

[0014] Using the load time series data of each target service as input, the trained load prediction model is invoked to obtain the load prediction data of each service;

[0015] The load prediction model is obtained by training a long short-term memory network based on the historical load time series data of each service.

[0016] As an embodiment of the multi-cloud service dependency analysis method described in this application, the method further includes:

[0017] Receive the service dependency result display message sent by the client. The service dependency result display message carries the service identifier.

[0018] Obtain the service dependency graph and load prediction data corresponding to the service identifier;

[0019] The service dependency graph is displayed in the form of an interactive diagram using pre-set visualization tools;

[0020] Use the preset charting tools to display the load forecast data in chart form.

[0021] As an embodiment of the multi-cloud service dependency analysis method described in this application, a service dependency graph is constructed based on service discovery information and service call data, including:

[0022] Based on service discovery information and service call data, construct an edge set of service call relationships;

[0023] Create an initial service dependency graph based on the pre-defined graph database;

[0024] Based on the edge set, nodes and edges of the initial service dependency graph are created to obtain the service dependency graph; each edge represents a service call, and the nodes connected at both ends of the edge are the calling service provider and the called service provider, respectively.

[0025] As an embodiment of the multi-cloud service dependency analysis method described in this application, after constructing a service dependency graph based on service discovery information and service call data, the method further includes:

[0026] Monitor cluster events, log streams, and alarm data in different cloud environments to obtain service monitoring data;

[0027] The service dependency graph is dynamically adjusted and updated based on service monitoring data.

[0028] As an embodiment of the multi-cloud service dependency analysis method described in this application, identifying the dependent services and strongly connected components of the Top N target services includes:

[0029] For each target service, the shortest path algorithm is used to find the shortest dependency path from the target service to the remaining target services;

[0030] Based on the shortest dependency path of each service, the dependent services of the Top N target services are obtained;

[0031] For each target service, a depth-first search is used to traverse the graph, recording the access order of each target service during the traversal and the lowest index during backtracking;

[0032] Strongly connected components are identified based on the access order of each target service and the lowest index during backtracking.

[0033] As an embodiment of the multi-cloud service dependency analysis method described in this application, obtaining service discovery information from multiple cloud environments includes:

[0034] Trigger the service discovery mechanism, call the preset data acquisition interface, traverse the nodes and service objects under the specified namespace, and obtain the metadata of the nodes and service objects;

[0035] Query instance information for different cloud services;

[0036] By integrating the metadata of nodes and service objects, as well as instance information of different cloud services, service discovery information for multiple cloud environments can be obtained.

[0037] In a second aspect, this application also provides a multi-cloud service dependency analysis apparatus. The apparatus includes:

[0038] The data acquisition module is used to acquire service discovery information and service call data from multiple cloud environments;

[0039] The dependency graph building module is used to build a service dependency graph based on service discovery information and service call data.

[0040] The service filtering module is used to determine the importance of each service based on the service dependency graph, and to filter out the Top N target services based on the importance.

[0041] The data identification module is used to identify the dependent services and strongly connected components of the Top N target services.

[0042] As an embodiment of the multi-cloud service dependency analysis device described in this application, the device further includes: a load prediction module, used to integrate service discovery information, service call data, dependent services and strongly connected components of the target service within a preset time period to obtain load time series data of the target service; using the load time series data of each target service as input, calling a trained load prediction model to obtain load prediction data of each service; the load prediction model is obtained by training a long short-term memory network based on the historical load time series data of each service.

[0043] As an embodiment of the multi-cloud service dependency analysis device described in this application, the device further includes: a data display module, configured to receive a service dependency result display message sent by a client, the service dependency result display message carrying a service identifier; obtain a service dependency graph and load prediction data corresponding to the service identifier; display the service dependency graph in the form of an interactive graph using a preset visualization tool; and display the load prediction data in the form of a chart by calling a preset charting tool.

[0044] As an embodiment of the multi-cloud service dependency analysis device described in this application, the dependency graph construction module is further configured to construct an edge set of service call relationships based on the service discovery information and the service call data; create an initial service dependency graph according to a preset graph database; and create nodes and edges of the initial service dependency graph based on the edge set to obtain the service dependency graph; each edge represents a service call, and the nodes connected at both ends of the edge are the calling service provider and the called service provider, respectively.

[0045] As an embodiment of the multi-cloud service dependency analysis device described in this application, the device further includes: a data update module, used to monitor cluster events, log streams and alarm data in different cloud environments to obtain service monitoring data; and dynamically adjust and update the service dependency graph based on the service monitoring data.

[0046] As an embodiment of the multi-cloud service dependency analysis device described in this application, the data identification module is further configured to, for each target service, find the shortest dependency path from the target service to the remaining target services using a shortest path algorithm; based on the shortest dependency paths of each service, obtain the dependent services of the Top N target services; for each target service, record the access order of each target service and the lowest index during backtracking by traversing the graph using a depth-first search; and identify strongly connected components based on the access order and the lowest index during backtracking of each target service.

[0047] As an embodiment of the multi-cloud service dependency analysis device described in this application, the data acquisition module is further configured to trigger a service discovery mechanism, call a preset data acquisition interface, traverse nodes and service objects under a specified namespace, acquire the metadata of the nodes and service objects, query instance information of different cloud services, and integrate the metadata of the nodes and service objects, as well as the instance information of the different cloud services, to obtain service discovery information for multiple cloud environments.

[0048] In a third aspect of the embodiments of this application, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in any of the above embodiments of the multi-cloud service dependency analysis method.

[0049] In a fourth aspect of the embodiments of this application, this application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps in any of the above embodiments of the multi-cloud service dependency analysis method.

[0050] In a fifth aspect of the embodiments of this application, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps in any of the above embodiments of the multi-cloud service dependency analysis method.

[0051] Details of one or more embodiments of this application are set forth in the following drawings and description. Other features, objects, and advantages of this application will become apparent from the specification, drawings, and claims. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 is an application environment diagram of the multi-cloud service dependency analysis method according to some embodiments;

[0054] Figure 2 is a flowchart illustrating a multi-cloud service dependency analysis method according to some embodiments;

[0055] Figure 3 is a flowchart illustrating the multi-cloud service dependency analysis method according to some embodiments;

[0056] Figure 4 is a flowchart illustrating the data display steps according to some embodiments;

[0057] Figure 5 is a flowchart illustrating the service dependency graph construction steps according to some embodiments;

[0058] Figure 6 is a flowchart illustrating the multi-cloud service dependency analysis method according to some embodiments;

[0059] Figure 7 is a flowchart illustrating the steps for identifying dependent services and strongly connected components according to some embodiments;

[0060] Figure 8 is a structural block diagram of a multi-cloud service dependency analysis device according to some embodiments;

[0061] Figure 9 is a structural block diagram of a multi-cloud service dependency analysis device according to some embodiments;

[0062] Figure 10 is a diagram of the internal structure of a computer device according to some embodiments. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0064] The data lineage analysis method provided in this application embodiment can be applied to the application environment shown in Figure 1. The terminal 102 communicates with the server 104 via a network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated onto the server 104, or it can be located in the cloud or on another network server.

[0065] Specifically, the operator can send data analysis instructions to the server 104 via the terminal 102. The server 104 connects to the target cluster by executing a preset script, and then obtains service discovery information and service call data from multiple cloud environments. Based on the service discovery information and service call data, a service dependency graph is constructed. Then, based on the service dependency graph, the importance of each service is determined, and the Top N target services are selected based on the importance. Finally, the dependent services and strongly connected components of the Top N target services are identified.

[0066] The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle systems, and projection devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, and smart glasses. The server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0067] In an exemplary embodiment, as shown in FIG2, a multi-cloud service dependency analysis method is provided. Taking the application of this method to server 104 in FIG1 as an example, the method includes the following steps S200 to S800. Wherein:

[0068] S200 obtains service discovery information and service call data from multiple cloud environments.

[0069] A cloud environment can be understood as a cloud platform. Service discovery information refers to information used to locate and identify other service instances in a distributed system, typically including the service's name, tag, address (IP and port), health status, and version number. Service call data records detailed information about calls between services, including but not limited to call time, caller, callee, request parameters, response result, and execution time.

[0070] In practice, developers can create a Python script that connects the server to service registries (such as Consul, Eureka, Kubernetes Service Discovery, etc.) and logging systems (such as ELK Stack, Prometheus, Grafana, etc.) in various cloud environments. Then, through API calls or log collection tools, the latest service discovery information is periodically retrieved from the service registry, and the latest service call data is retrieved from the logging system. After retrieving the service discovery information and service call data, the raw data can be parsed into a unified format, such as JSON or CSV, and then stored in a database or data warehouse. It's understandable that the retrieval of service discovery information and service call data can be done either periodically in real-time according to a pre-defined script, or after periodic data retrieval, the data can be stored in a database or data warehouse, and the stored service discovery information and service call data can be retrieved from the database or data warehouse in response to analysis commands sent by the terminal.

[0071] S400 constructs a service dependency graph based on service discovery information and service call data.

[0072] A service dependency graph is a graphical representation that shows the dependencies between services in a system. Each node represents a service, and edges represent the calling relationships between services.

[0073] After obtaining service discovery information and service call data, the server can remove invalid or erroneous data to ensure accuracy and consistency. Subsequently, based on the service call data, the caller and callee of each service are extracted, dependencies between services are established, and a service dependency graph is constructed using a graph database (such as Neo4j) or a graph algorithm library (such as NetworkX). Each service is treated as a node, and each call is treated as an edge.

[0074] S600 determines the importance of each service based on the service dependency graph and selects the Top N target services based on their importance.

[0075] Service importance is a metric that measures the significance of a service within a system. Factors typically considered include call frequency, the number of dependent services, and service type. The Top N target services are selected based on their importance, with the top N services chosen as the primary management targets. In this embodiment, N can be 10; however, in other embodiments, N can also be 8, 12, or other values.

[0076] In practice, the server can use graph algorithms (such as PageRank, BetweennessCentrality, etc.) based on the service dependency graph to calculate the importance of each service, taking into account the in-degree, out-degree, intermediate nodes, and other characteristics of the services.

[0077] Taking the use of the PageRank algorithm to quantify the relative importance or influence of each service as an example, services with high importance may be allocated more resources for backup, or be given priority for recovery in the event of a failure, to ensure the stable operation and service quality of the overall system.

[0078] The specific formula is as follows:

[0079] Where PR(u) represents the PageRank value of node u; d is the damping factor, usually taken as 0.85; N is the total number of nodes in the graph; Bu is the set of all nodes pointing to node u, which can be understood as the dependency frequency of node u; W(v) is the weight of node v, which can be based on the request frequency, dependency, response time, etc. of the node (i.e., service); ∑w∈BvW(w) is the sum of the outgoing weights of node v, that is, the sum of the weights of all nodes that call node v.

[0080] The adjusted algorithm introduces a weight factor W(v), allowing the algorithm to adjust based on the importance or type of the service, making the PageRank value more reflective of the actual importance of the service and improving the accuracy and fairness of the algorithm.

[0081] After obtaining the importance of each service, the services can be sorted according to their calculated importance, generating a service list arranged in descending order of importance. Next, based on business requirements, the top 10 services from the sorted service list are selected as target services. Further, the top N services can be selected as target services from the sorted service list.

[0082] S800 identifies the dependent services and strongly connected components of the Top N target services.

[0083] A dependent service refers to other services that a service depends on, including directly dependent services and indirectly dependent services. A strongly connected component is an important concept in graph theory, referring to a class of subgraphs within a directed graph. Specifically, a strongly connected component is a maximal subgraph within a directed graph where every two vertices can be reached by a path. If all vertices in a subgraph are pairwise strongly connected, then this subgraph is called a strongly connected component.

[0084] In practice, the server can identify the direct and indirect dependencies of the Top N target services from the service dependency graph. Then, graph algorithms (such as Tarjan's algorithm or Kosaraju's algorithm) can be used to detect strongly connected components in the service dependency graph. Furthermore, the composition of each strongly connected component can be analyzed to identify key services and potential risk points. The identified dependency service and strongly connected component information is stored in a database or configuration file for subsequent monitoring and optimization.

[0085] The aforementioned multi-cloud service dependency analysis method comprehensively understands the service distribution and interaction across different cloud platforms by acquiring service discovery information and service call data in the multi-cloud environment. Based on this, a precise service dependency graph is constructed, clearly illustrating the interaction patterns and dependencies between services. Next, the service dependency graph quantifies the importance of each service in the system. The Top N target services, selected based on service importance, allow administrators to focus on services with the greatest impact on business operations, ensuring the stable operation of critical services. Finally, identifying the dependent services and strongly connected components of the Top N target services helps optimize inter-service dependencies, reduce coupling between services, and improve system flexibility and scalability. It also helps reduce large-scale service interruptions caused by single points of failure. This approach effectively improves the efficiency of dependency analysis across multiple cloud services.

[0086] In some embodiments, obtaining service discovery information for multiple cloud environments includes: triggering a service discovery mechanism, calling a preset data acquisition interface, traversing nodes and service objects under a specified namespace, obtaining metadata of nodes and service objects, querying instance information of different cloud services, integrating the metadata of nodes and service objects, as well as the instance information of different cloud services, to obtain service discovery information for multiple cloud environments.

[0087] Service discovery is a mechanism used to automatically detect and locate services in a distributed system. A namespace is a logical container used to organize and isolate services; it can contain multiple nodes and service objects. A service object is a service defined within a namespace; it can be a microservice or a functional module. Metadata contains detailed information describing nodes and service objects, such as name, ID, address, port, and health status.

[0088] In practice, this can be implemented by having the server execute a Python script. This script connects to the target Kubernetes cluster using a Kubernetes Python client library (such as the Kubernetes SDK). It calls the API interfaces `list_namespaced_pod` and `list_namespaced_service` to iterate through all Pods (nodes) and Service objects (service objects) under the specified namespace. For non-Kubernetes environments, this can be done by querying AWS EC2 instances, Azure VMs, or GCP Compute Engine instance information. Subsequently, the metadata of the Pods and Services is parsed, including service name, tags, IP address, port, etc., and this information is stored in an in-memory data structure or written directly to a database as service discovery information.

[0089] Furthermore, developers can configure the Prometheus server, create or modify Prometheus configuration files, and add monitoring targets for Kubernetes services, such as through ServiceMonitor or PodMonitor resources. They can integrate the Jaeger client into the service code, inserting tracing code at key locations based on the service call logic to record information such as the start, end, success, or failure status of service calls, as well as their duration. Log collection tools such as Fluentd or Logstash can also be used, configuring log parsing rules to extract service call chain information (i.e., service call data) from service logs. Finally, the server can forward service call data to a central log storage system.

[0090] In this embodiment, the service discovery mechanism is automatically triggered to collect data. By traversing the specified namespace, the metadata of nodes and service objects can be discovered, resulting in more comprehensive and detailed service discovery information. Through monitoring targets and log collection tools, more comprehensive service call data can be collected.

[0091] As shown in Figure 3, in some embodiments, after S800, the method further includes:

[0092] S900 integrates service discovery information, service call data, dependent services, and strongly connected components of the target service within a preset time period to obtain the load time series data of the target service. Using the load time series data of each target service as input, it calls the trained load prediction model to predict the load prediction data of each target service.

[0093] In this embodiment, the load prediction model is obtained by training a long short-term memory network based on the historical load time series data of each service.

[0094] Load time series data refers to data integrated based on service discovery information, service call data, dependent services, and strongly connected components. In addition to the above data, it also includes data such as service request frequency, dependency, and response time.

[0095] In practical applications, the training process for a load prediction model can be as follows: Collect historical load time-series data for the top 10 target services within a given historical time period, including service dependency analysis results, service request rates, response times, and resource utilization. Then, preprocess the historical load time-series data, including data cleaning and standardization. Next, use the preprocessed historical load time-series data as the model's training dataset. Using Python and either TensorFlow or PyTorch frameworks, train an LSTM (Long Short-Term Memory) model with the historical load time-series data of the top 10 target services as input. The output data is the daily load prediction data for each target service within a future period, such as a week. Subsequently, adjust the model parameters through cross-validation and error analysis until the prediction accuracy requirements are met, resulting in the load prediction model. It can be understood that the model can predict load data for the next 3 days, 1 month, or 2 months, depending on the specific circumstances, and is not limited here.

[0096] In practice, after identifying the dependent services and strongly connected components of the target service, the service discovery information, service call data, dependent services, and strongly connected components of the target service within a preset time period can be integrated to obtain the load time series data of the target service. Then, using the load time series data of each target service as input, a trained load prediction model is invoked to predict the load data of each target service in the coming week.

[0097] In this embodiment, by training a load prediction model to predict the future load of the target service, computing resources can be planned and allocated in advance to ensure that there are enough resources to process requests during high-load periods, avoiding service performance degradation or downtime caused by insufficient resources. It can also take timely measures to prevent system overload and improve system stability and reliability.

[0098] As shown in Figure 4, in some embodiments, after S800, the method further includes:

[0099] S940 receives a service dependency result display message sent by the client. The service dependency result display message carries the service identifier.

[0100] A service identifier can be a unique name, ID, or other unique identifier for a service. When a user wants to visually view the service dependency analysis results, they can send a service dependency result display message to the server through the client. The server parses the received message and extracts the service identifier (Service ID). The validity of the service identifier is verified to ensure the request comes from a legitimate client. Simultaneously, request logs are recorded, including request time, client IP, service identifier, and other information, for subsequent auditing and troubleshooting.

[0101] S942, obtain the service dependency graph and load prediction data corresponding to the service identifier.

[0102] Subsequently, based on the service identifier, the corresponding service dependency graph and load prediction data are retrieved from the database or data warehouse. Furthermore, the retrieved data can be formatted into a suitable display format, such as JSON or XML. To improve response speed, frequently used service dependency graphs and load prediction data can be cached in memory or a caching system (such as Redis).

[0103] S944 uses a pre-defined visualization tool to display the service dependency graph in the form of an interactive diagram.

[0104] The formatted service dependency graph data is transmitted to a front-end visualization tool, such as D3.js, ECharts, or Graphviz. The front-end visualization tool then generates an interactive graph based on the received data. The interactive graph should support features such as displaying detailed dependency information on mouse hover and expanding to more details by clicking on nodes.

[0105] S946 invokes the preset charting tool to display the load forecast data in chart form.

[0106] Next, the formatted load forecast data is transmitted to front-end charting tools such as Highcharts, Chart.js, and Apache ECharts. These tools generate charts based on the received data, allowing users to switch between viewing the forecast load for each day along a timeline. They also provide anomaly alerts, issuing warnings when forecast values ​​exceed thresholds. Common chart types include line charts, bar charts, and pie charts. Furthermore, the server can implement dynamic chart updating capabilities, such as periodically refreshing data and allowing users to select different time ranges.

[0107] In this embodiment, interactive diagrams and dynamic charts allow users to more intuitively understand service dependencies and future load conditions, and perform operations such as viewing and zooming the charts as needed.

[0108] As shown in Figure 5, in some embodiments, S400 includes:

[0109] S420 constructs an edge set of service call relationships based on service discovery information and service call data.

[0110] An edge set is a group of edges that represent the relationships between nodes. Here, the edge set represents the call relationships between services. After the server obtains service discovery information and service call data, it can remove invalid or erroneous data, such as records of failed calls or duplicate records. It iterates through all call records, extracts the names of the caller and the callee, and then uses a tuple (caller, callee) to represent an edge. All extracted edges are stored in a list or set to form an edge set.

[0111] S440 creates an initial service dependency graph based on a pre-defined graph database.

[0112] The initial service dependency graph is an empty graph used to represent dependencies between storage services.

[0113] Following the previous step, the server selects a suitable graph database, such as Neo4j, installs and configures the graph database, ensuring it functions correctly. Next, it connects to the graph database, creates a new graph instance, and defines the graph's schema, including the types of nodes and edges. Then, it creates an empty service dependency graph in the graph database, ready for subsequent addition of nodes and edges.

[0114] S460: Based on the edge set, create the nodes and edges of the initial service dependency graph to obtain the service dependency graph; each edge represents a service call, and the nodes connected at both ends of the edge are the calling service provider and the called service provider, respectively.

[0115] A node is the basic unit in a graph, representing a service. An edge is a line connecting two nodes, representing the calling relationship between services.

[0116] After creating the initial service dependency graph, the server traverses all edges in the edge set to extract the service names of the caller and the service provider. For each service name, if there is no corresponding node in the graph, a new node is created and added to the initial service dependency graph. Then, based on each edge in the edge set, corresponding edges are created and added to the graph, with each edge starting at the node of the caller's service and ending at the node of the callee's service. Next, attributes are set for the nodes and edges. For example, nodes can have attributes such as service name and service instance address, while edges can have attributes such as call count and call time. The graph structure is further optimized as needed, such as merging duplicate edges and adjusting node layout. The created service dependency graph is persisted to a graph database for subsequent querying and analysis.

[0117] In this embodiment, by constructing a service dependency graph, a comprehensive understanding of the call relationships between services can be achieved, helping administrators quickly locate problems and optimize resource allocation. Furthermore, the service dependency graph can visually display the dependencies between services, facilitating rapid identification of fault points and their impact scope, and improving troubleshooting efficiency.

[0118] As shown in Figure 6, in some embodiments, after S400, the method further includes:

[0119] S500 monitors cluster events, log streams, and alarm data in different cloud environments to obtain service monitoring data. Based on the service monitoring data, it dynamically adjusts and updates the service dependency graph.

[0120] Cluster events refer to various events occurring within the cluster, including but not limited to node joining or leaving, service startup or shutdown, and resource allocation changes. Log streams refer to log data generated during service operation, including application logs, system logs, and security logs. Alarm data refers to various alarm information triggered in the system, such as performance alarms, security alarms, and resource usage alarms. Service monitoring data is data used to monitor the status and performance of services, derived from a combination of cluster events, log streams, and alarm data.

[0121] In practice, after building the service dependency graph, the server can periodically pull the latest cluster events, log streams, and alarm data from the cluster management system, log collection system, and alarm system through API calls or log collection tools. Then, it extracts key information from the pulled data, such as event type, log level, alarm type, timestamp, etc., and automatically updates the service dependency graph based on real-time service monitoring data to reflect the current service status and dependencies.

[0122] For example, taking updating nodes and edges in a service dependency graph based on cluster events as an example, if a service stops running, the corresponding node is removed; if a service starts, the corresponding node is added. Based on the log stream, new service call relationships are identified, and corresponding edges are added or updated. Based on alert data, problematic target services or edges are marked, for example, by highlighting them in red or adding annotations. Further, a graph database (such as Neo4j) or a graph algorithm library (such as NetworkX) is used to update the service dependency graph. The updated graph is persisted to the graph database for subsequent querying and analysis.

[0123] In this embodiment, by monitoring cluster events, log streams, and alarm data in real time and updating the service dependency graph in real time, the current service status and dependency relationships can be reflected in a timely and accurate manner.

[0124] As shown in Figure 7, in some embodiments, S800 includes:

[0125] S820, for each target service, uses the shortest path algorithm to find the shortest dependency path from the target service to the remaining target services.

[0126] The shortest dependency path is the shortest path between services. A dependency path indicates how a service depends on other services.

[0127] After the server constructs the service dependency graph, the Dijkstra algorithm or the Floyd-Warshall algorithm can be used to find the shortest dependency path from each target service to the remaining target services. Taking the Dijkstra algorithm as an example, the shortest dependency path can be found as follows:

[0128] a. Initialization

[0129] Treat the target service as a node, set the shortest path estimate dist[v] of all nodes to infinity, except for the starting node dist[A] = 0; create a set of unvisited nodes.

[0130] b. Algorithm process

[0131] Select the node u (initially A) with the minimum shortest path estimate from the set of unvisited nodes;

[0132] For each neighbor v of u:

[0133] Calculate the cost of the new path from u to v: new_cost = dist[u] + weight(u,v)·P(u,v);

[0134] If new_cost < dist[v], update the shortest path estimate of v: dist[v] = new_cost;

[0135] Repeat the above steps until the set of unvisited nodes is empty.

[0136] Subsequently, the shortest path can be constructed by recording the predecessor nodes of each node. weight(u, v) represents the cost or time of a service call, and the performance evaluation metric P(u, v) represents the performance of the call, such as latency or reliability. By considering the performance evaluation metric P(u, v), the adjusted algorithm can more comprehensively evaluate the "cost" of a path, rather than just distance or weight, which enables the algorithm to adapt to different network environments and performance requirements, such as making more reasonable path selections in terms of network latency or reliability.

[0137] S840. Based on the shortest dependency paths of each target service, obtain the dependent services of the top N target services.

[0138] After obtaining the shortest dependency paths of each target service, extract the dependent services of each target service from the shortest paths.

[0139] S860. For each target service, traverse the graph through depth-first search, and record the access order of each target service and the lowest index during backtracking.

[0140] Depth-first search is a graph traversal algorithm. Specifically, it starts from a node and searches as deep as possible along a path through recursion or a stack until it cannot continue, and then backtracks.

[0141] During specific implementation, the access order and the lowest index can be initialized for each node. Subsequently, starting from each target service node, use DFS to traverse the graph. During the traversal process, record the access order of each node, and during the backtracking process, update the lowest index of each node.

[0142] S880. Based on the access order of each target service and the lowest index during backtracking, identify the strongly connected components.

[0143] After the server obtains the access order of each target service and the lowest index during backtracking, use the Tarjan algorithm, combined with the access order and the lowest index, to identify the strongly connected components in the graph. Specifically, during the DFS traversal process, use a stack to record the nodes on the current path. When the access order of a node is equal to its lowest index, pop all the nodes from the stack, and these nodes form a strongly connected component. The specific implementation steps can be as follows:

[0144] a. Initialization

[0145] We use depth-first search, maintaining disc (discovery time) and low (earliest achievable discovery time) for each node. We use a stack to maintain the nodes on the current DFS path.

[0146] b. Algorithm process

[0147] If a node is not visited, perform a DFS access to it and update disc and low.

[0148] For each neighboring node, if the neighboring node is in the stack, update the low value.

[0149] If the low value is found to be equal to the disc value, the current node is the root of a strongly connected component, and all nodes are popped from the stack to form the SCC.

[0150] Update the low value:

[0151] If the neighboring node is in the stack, low[u] = Min(low[u], disc[v]);

[0152] If the node is not visited, low[u] = Min(low[u], low[v]).

[0153] In this embodiment, by calculating the shortest dependency path from each target service to other services, the dependent services of the target service can be determined. DFS traversal can cover all nodes in the graph, ensuring no omissions. The Tarjan algorithm can accurately identify strongly connected components in the graph, providing a reliable foundation for subsequent dependency analysis.

[0154] To provide a clearer explanation of the data lineage analysis method provided in this application, a specific embodiment is described below, which includes the following steps:

[0155] S1 retrieves service discovery information and service call data from multiple cloud environments.

[0156] S2: Based on service discovery information and service call data, construct an edge set of service call relationships. According to the preset graph database, create an initial service dependency graph. Based on the edge set, create nodes and edges of the initial service dependency graph to obtain the service dependency graph.

[0157] Each edge represents a service call, and the two ends of the edge connect the calling service provider and the called service provider, respectively.

[0158] S3 determines the importance of each service based on the service dependency graph and selects the Top N target services based on their importance.

[0159] S4. For each target service, use the shortest path algorithm to find the shortest dependency path from the target service to the remaining target services. Based on the shortest dependency paths of each target service, obtain the dependency services of the Top N target services.

[0160] S5: For each target service, a depth-first search is used to traverse the graph, recording the access order of each target service and the lowest index during backtracking. Based on the access order and the lowest index during backtracking of each target service, strongly connected components are identified.

[0161] S6 integrates the service discovery information, service call data, dependent services, and strongly connected components of the target service within a preset time period to obtain the load time series data of the target service.

[0162] S7 takes the load time series data of each target service as input, calls the trained load prediction model, and predicts the load prediction data of each target service.

[0163] S8 monitors cluster events, log streams, and alarm data in different cloud environments to obtain service monitoring data, and dynamically adjusts and updates the service dependency graph based on the service monitoring data.

[0164] S9 receives a service dependency result display message sent by the client. The service dependency result display message carries the service identifier. Through the preset visualization tool, the service dependency graph is displayed in the form of an interactive graph. The preset chart tool is called to display the load prediction data in the form of a chart.

[0165] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0166] Based on the same inventive concept, this application also provides a data lineage analysis device for implementing the data lineage analysis method described above. The solution provided by this device is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data lineage analysis device embodiments provided below can be found in the limitations of the data lineage analysis method described above, and will not be repeated here.

[0167] In an exemplary embodiment, as shown in FIG8, a multi-cloud service dependency analysis device 800 is provided, including: a data acquisition module 810, a dependency graph construction module 820, a service filtering module 830, and a data identification module 840, wherein:

[0168] The data acquisition module 810 is used to acquire service discovery information and service call data from multiple cloud environments.

[0169] The dependency graph construction module 820 is used to construct a service dependency graph based on service discovery information and service call data.

[0170] The service filtering module 830 is used to determine the importance of each service based on the service dependency graph, and to filter out the Top N target services based on the importance.

[0171] The data identification module 840 is used to identify the dependent services and strongly connected components of the Top N target services.

[0172] As shown in Figure 9, in some embodiments, the device further includes a load prediction module 850, which is used to integrate the service discovery information, service call data, dependent services and strongly connected components of the target service within a preset time period to obtain the load time series data of the target service. Using the load time series data of each target service as input, the device calls the trained load prediction model to obtain the load prediction data of each service. The load prediction model is obtained by training a long short-term memory network based on the historical load time series data of each service.

[0173] As shown in Figure 9, in some embodiments, the device further includes a data display module 860, which is used to receive a service dependency result display message sent by the client. The service dependency result display message carries a service identifier. The device obtains the service dependency graph and load prediction data corresponding to the service identifier, displays the service dependency graph in the form of an interactive graph through a preset visualization tool, and calls a preset chart tool to display the load prediction data in the form of a chart.

[0174] In some embodiments, the dependency graph construction module 820 is further configured to construct an edge set of service call relationships based on service discovery information and service call data, create an initial service dependency graph according to a preset graph database, and create nodes and edges of the initial service dependency graph based on the edge set to obtain the service dependency graph; each edge represents a service call, and the nodes connected at both ends of the edge are the calling service provider and the called service provider, respectively.

[0175] As shown in Figure 9, in some embodiments, the device further includes a data update module 870, which is used to monitor cluster events, log streams and alarm data in different cloud environments, obtain service monitoring data, and dynamically adjust and update the service dependency graph based on the service monitoring data.

[0176] In some embodiments, the data identification module 840 is further configured to, for each target service, find the shortest dependency path from the target service to the remaining target services using a shortest path algorithm, obtain the dependent services of the Top N target services based on the shortest dependency paths of each service, record the access order of each target service and the lowest index during backtracking during the traversal of the graph using a depth-first search for each target service, and identify strongly connected components based on the access order and the lowest index during backtracking of each target service.

[0177] In some embodiments, the data acquisition module 810 is also used to trigger a service discovery mechanism, call a preset data acquisition interface, traverse nodes and service objects under a specified namespace, acquire the metadata of nodes and service objects, query instance information of different cloud services, integrate the metadata of nodes and service objects and the instance information of different cloud services, and obtain service discovery information for multiple cloud environments.

[0178] Each module in the aforementioned data lineage analysis device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0179] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram is shown in Figure 10. The computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is connected to the system bus via the I / O interfaces. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes non-volatile storage media and internal memory. The non-volatile storage media stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device stores SQL query statements, stored procedure data, and data lineage relationships, etc. The I / O interfaces of the computer device are used for exchanging information between the processor and external devices. The communication interface of the computer device is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a data lineage analysis method.

[0180] Those skilled in the art will understand that the structure shown in Figure 10 is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or may combine certain components, or may have different component arrangements.

[0181] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in any of the above-described embodiments of the data lineage analysis method.

[0182] In one embodiment, the processor, when executing the computer program, also implements the steps in any of the above-described embodiments of the data lineage analysis method.

[0183] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in any of the above-described data lineage analysis method embodiments.

[0184] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in any of the above embodiments of the multi-cloud service dependency analysis method.

[0185] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0186] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0187] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0188] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A multi-cloud service dependency analysis method, characterized by, The method includes: Obtain service discovery information and service call data from multiple cloud environments; Based on the service discovery information and the service call data, a service dependency graph is constructed; Based on the service dependency graph, the importance of each service is determined, and the Top N target services are selected based on their importance. Identify the dependent services and strongly connected components of the Top N target services.

2. The method according to claim 1, characterized in that, The method further includes: By integrating the service discovery information, service call data, dependent services, and strongly connected components of the target service within a preset time period, the load time series data of the target service is obtained. Using the load time series data of each target service as input, the trained load prediction model is invoked to obtain the load prediction data of each service; The load prediction model is obtained by training a long short-term memory network based on the historical load time series data of each service.

3. The method of claim 2, wherein, The method further includes: Receive a service dependency result display message sent by the client, the service dependency result display message carrying a service identifier; Obtain the service dependency graph and load prediction data corresponding to the service identifier; The service dependency graph is displayed in the form of an interactive diagram using a pre-set visualization tool; The preset charting tool is invoked to display the load forecast data in the form of a chart.

4. The method according to any one of claims 1 to 3, characterized in that, The step of constructing a service dependency graph based on the service discovery information and the service call data includes: Based on the service discovery information and the service invocation data, an edge set of service invocation relationships is constructed; Create an initial service dependency graph based on the pre-defined graph database; Based on the edge set, nodes and edges of the initial service dependency graph are created to obtain the service dependency graph; each edge represents a service call, and the nodes connected at both ends of the edge are the calling service provider and the called service provider, respectively.

5. The method of claim 4, wherein, After constructing the service dependency graph based on the service discovery information and the service call data, the method further includes: Monitor cluster events, log streams, and alarm data in different cloud environments to obtain service monitoring data; Based on the service monitoring data, the service dependency graph is dynamically adjusted and updated.

6. The method according to any one of claims 1 to 3, characterized in that, The identification of the dependent services and strongly connected components of the Top N target services includes: For each target service, the shortest dependency path from the target service to the remaining target services is found using the shortest path algorithm; Based on the shortest dependency path of each service, the dependent services of the Top N target services are obtained; For each target service, a depth-first search is used to traverse the graph, recording the access order of each target service during the traversal and the lowest index during backtracking; Strongly connected components are identified based on the access order of each target service and the lowest index during backtracking.

7. The method according to any one of claims 1 to 3, characterized in that, Obtain service discovery information from multiple cloud environments, including: Trigger the service discovery mechanism, call the preset data acquisition interface, traverse the nodes and service objects under the specified namespace, and obtain the metadata of the nodes and service objects; Query instance information for different cloud services; By integrating the metadata of the nodes and the service objects, as well as the instance information of the different cloud services, service discovery information for multiple cloud environments is obtained.

8. A multi-cloud service dependency analysis apparatus, characterized by, The device includes: The data acquisition module is used to acquire service discovery information and service call data from multiple cloud environments; The dependency graph construction module is used to construct a service dependency graph based on the service discovery information and the service call data. The service filtering module is used to determine the importance of each service based on the service dependency graph, and to filter out the Top N target services based on the importance. The data identification module is used to identify the dependent services and strongly connected components of the Top N target services.

9. The apparatus according to claim 8, characterized in that, The device further includes: The load prediction module is used to integrate service discovery information, service call data, dependent services and strongly connected components of the target service within a preset time period to obtain the load time series data of the target service; using the load time series data of each target service as input, it calls the trained load prediction model to obtain the load prediction data of each service; the load prediction model is obtained by training a long short-term memory network based on the historical load time series data of each service.

10. The apparatus according to claim 9, characterized in that, The device further includes: The data display module is used to receive a service dependency result display message sent by the client, the service dependency result display message carrying a service identifier; obtain the service dependency graph and load prediction data corresponding to the service identifier; display the service dependency graph in the form of an interactive graph using a preset visualization tool; and call a preset chart tool to display the load prediction data in the form of a chart.

11. The method according to any one of claims 8 to 10, characterized in that, The dependency graph construction module is also used to construct an edge set of service call relationships based on the service discovery information and the service call data; and to create an initial service dependency graph according to a preset graph database. Based on the edge set, nodes and edges of the initial service dependency graph are created to obtain the service dependency graph; each edge represents a service call, and the nodes connected at both ends of the edge are the calling service provider and the called service provider, respectively.

12. The apparatus according to claim 11, characterized in that, The device further includes: The data update module is used to monitor cluster events, log streams, and alarm data in different cloud environments to obtain service monitoring data; based on the service monitoring data, the service dependency graph is dynamically adjusted and updated.

13. The method according to any one of claims 8 to 10, characterized in that, The data identification module is further configured to, for each target service, use a shortest path algorithm to find the shortest dependency path from the target service to the remaining target services; based on the shortest dependency paths of each service, obtain the dependent services of the Top N target services; for each target service, use a depth-first search to traverse the graph, record the access order of each target service during the traversal process and the lowest index during backtracking; and identify strongly connected components based on the access order of each target service and the lowest index during backtracking.

14. The method according to any one of claims 8 to 10, characterized in that, The data acquisition module is also used to trigger the service discovery mechanism, call the preset data acquisition interface, traverse the nodes and service objects under the specified namespace, obtain the metadata of the nodes and the service objects, query the instance information of different cloud services, and integrate the metadata of the nodes and the service objects, as well as the instance information of the different cloud services, to obtain service discovery information of multiple cloud environments.

15. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images