Ai-based method and system for automatic optimization of distributed computing tasks for big data

Abstract

The present invention relates to the technical field of AI, and provides an AI-based method and system for automatic optimization of distributed computing tasks for big data. The method comprises: constructing a multi-modal spatiotemporal feature perception network to extract spatiotemporal features of tasks and resources, and training a hierarchical hybrid decision network to perform global task scheduling and local resource allocation. A hierarchical deep reinforcement learning architecture is used for a global policy network, which combines Monte Carlo tree search and prioritized experience replay to generate decisions. A local execution network optimizes local deployment on the basis of a graph neural network and multi-agent collaborative learning. In addition, a distributed anomaly detection network is deployed to monitor performance in real time, adaptive tuning is achieved by means of reinforcement transfer learning, and finally the perception network is updated by means of knowledge distillation, thereby achieving model evolution. The present invention can effectively improve the execution efficiency and resource utilization rate of distributed computing tasks for big data, and reduce system operation costs.

Description

AI-based automatic optimization method and system for big data distributed computing tasks Technical Field

[0001] This invention relates to AI technology, and more particularly to an automatic optimization method and system for big data distributed computing tasks based on AI. Background Technology

[0002] With the advent of the big data era, distributed computing has become a key technology for processing massive amounts of data. To improve the efficiency and resource utilization of distributed computing systems, task scheduling and resource allocation are crucial. Traditional task scheduling methods typically rely on human experience or rule-based strategies, making it difficult to adapt to complex and ever-changing computing environments and dynamic task loads. Furthermore, traditional resource allocation schemes often lack flexibility, failing to dynamically adjust according to task requirements and resource availability, leading to resource waste or task delays.

[0003] Difficulty in effectively handling complex task dependencies and data locality: In big data distributed computing scenarios, tasks often have complex dependencies, and data is usually distributed across different computing nodes. Traditional task scheduling methods struggle to effectively handle these dependencies and data locality, leading to low task execution efficiency.

[0004] Lack of adaptability to dynamic environments: Computing environments and task loads are often dynamic, and traditional task scheduling and resource allocation schemes are difficult to adapt to these changes, leading to a decline in system performance.

[0005] Insufficient anomaly handling capabilities: In distributed computing systems, hardware failures, network congestion, and other anomalies occur frequently. Traditional task scheduling and resource allocation schemes lack effective anomaly handling mechanisms, making it difficult to guarantee system stability and reliability. Summary of the Invention

[0006] This invention provides an AI-based method and system for automatically optimizing distributed big data computing tasks, which can solve the problems in the prior art.

[0007] A first aspect of the present invention,

[0008] Provides AI-based automatic optimization methods for big data distributed computing tasks, including:

[0009] A multimodal spatiotemporal feature perception network is constructed. This network employs a hierarchical bidirectional long short-term memory (LSTM) unit to build a temporal feature extraction layer, transforming task arrival time, task completion time, data dependencies, and computational load intensity from this layer into a multidimensional temporal feature sequence. A graph attention network is used to construct a spatial feature extraction layer, transforming the geographical distribution of computing nodes, network topology, computing power, storage capacity, and bandwidth resources from this layer into dynamic graph features. A dynamic weighted self-attention mechanism is used to adaptively fuse the multidimensional temporal feature sequence and the dynamic graph features, generating a heterogeneous feature tensor with spatiotemporal correlation. A contrastive learning method is then used to enhance the expressive power of this heterogeneous feature tensor.

[0010] A hierarchical hybrid decision network is trained based on the heterogeneous feature tensors. The hierarchical hybrid decision network includes a global policy network and a local execution network. The global policy network adopts a hierarchical deep reinforcement learning architecture, extracts hierarchical features of task queue state, resource utilization state, and system load state through a transformation encoder, and generates global decisions for task scheduling and resource allocation by combining Monte Carlo tree search and priority experience replay mechanism. The local execution network adopts a graph neural network structure, optimizes task deployment location and resource allocation ratio within the local computing node cluster based on the global decisions, and achieves dynamic coordination of local decisions through a multi-agent collaborative learning mechanism.

[0011] Adaptive scheduling is performed using the hierarchical hybrid decision network. During task execution, a distributed anomaly detection network is deployed. This network constructs a normal mode baseline for multidimensional performance indicators based on a variational autoencoder. It monitors performance fluctuations and load balancing status in real time using anomaly detectors and predicts resource demand trends using a graph temporal neural network. When anomalies are detected, an adaptive optimization mechanism is triggered. This mechanism uses reinforcement transfer learning to generate task migration strategies and resource reallocation schemes. System optimization is achieved through progressive task migration and elastic resource adjustment. The optimization results are updated through knowledge distillation to the multimodal spatiotemporal feature perception network, achieving knowledge accumulation and model evolution.

[0012] A hierarchical hybrid decision network is trained based on the heterogeneous feature tensors. This network comprises a global policy network and local execution networks. The global policy network employs a hierarchical deep reinforcement learning architecture, extracting hierarchical features of task queue states, resource utilization states, and system load states through a transform encoder. Combined with Monte Carlo tree search and priority experience replay mechanisms, it generates global decisions for task scheduling and resource allocation, including:

[0013] A hierarchical hybrid decision network is constructed, which includes a global policy network and a local execution network, wherein the global policy network adopts a hierarchical deep reinforcement learning architecture; the heterogeneous feature tensor is input into a transform encoder, and hierarchical features are extracted through the multi-head self-attention mechanism of the transform encoder. The hierarchical features are used to describe the multi-level correlation between task queue state, resource utilization state and system load state.

[0014] The hierarchical features are input into the global policy network to generate global decisions for task scheduling and resource allocation based on the hierarchical features; a Monte Carlo tree search is constructed, and the global decisions are used as the initial policy to guide the search process. Candidate decision sequences are obtained through tree search expansion; state evaluation values ​​are recorded at tree search nodes, and the optimal decision branch is selected based on the state evaluation values.

[0015] Design a priority experience replay mechanism to record decision trajectories and system feedback information; calculate the importance weights of the decision trajectories based on temporal difference errors, and construct a priority sampling probability distribution; sample high-value training data from the experience pool according to the priority sampling probability distribution, and use the high-value training data to optimize the global policy network;

[0016] The decision output of the global policy network is sent to the local execution network, which then generates specific scheduling instructions and resource allocation schemes.

[0017] The local execution network adopts a graph neural network structure. Based on the global decision, it optimizes the task deployment location and resource allocation ratio within the local computing node cluster. Dynamic coordination of local decisions is achieved through a multi-agent collaborative learning mechanism, including:

[0018] Receive the resource allocation scheme of the global decision network, the resource allocation scheme including the target area for task deployment, the baseline ratio of resource allocation and the constraints for load balancing;

[0019] Under the guidance of the resource allocation scheme, a graph neural network structure of a local computing node cluster is constructed. The nodes in the graph neural network structure include computing resource capabilities, current load status, and task execution records. The connection edges between nodes include network bandwidth, transmission latency, and connection stability.

[0020] Based on the graph neural network structure, the graph neural network is used to extract the state features of local computing node clusters, and the node information is aggregated through graph convolutional layers and attention layers to generate feature vectors that reflect the computing power and load distribution of nodes.

[0021] Based on the feature vector, a matching score between the task to be deployed and the computing node is calculated. The matching score takes into account the remaining resources of the node and the network connection quality, and generates a priority ranking for task deployment.

[0022] Based on the priority ranking, the task deployment location is optimized, and the resource allocation ratio of each task on the target computing node is determined by iterative adjustment. The resource allocation ratio includes CPU ratio, memory ratio, and bandwidth ratio.

[0023] The task deployment location and the resource allocation ratio are input into the multi-agent collaborative learning system. Each local execution network in the multi-agent collaborative learning system acts as an independent agent and makes decisions based on local observation information and the state of neighboring nodes. The multi-agent collaborative learning system enables dynamic coordination of local decisions.

[0024] Adaptive scheduling is performed using the hierarchical hybrid decision network. During task execution, a distributed anomaly detection network is deployed. This network constructs a normal mode baseline for multi-dimensional performance indicators based on a variational autoencoder. Anomaly detectors monitor performance fluctuations and load balancing status in real time. Combined with a graph temporal neural network, resource demand trends are predicted, including:

[0025] Adaptive scheduling is performed using the hierarchical hybrid decision network, which formulates an initial task deployment plan and resource allocation strategy based on the real-time status information of computing nodes, serving as a benchmark reference for subsequent anomaly detection.

[0026] Based on the initial task deployment scheme, a distributed anomaly detection network is deployed during task execution. The distributed anomaly detection network collects and standardizes the performance index data of each node in real time, including processor utilization, memory usage, network throughput, and task response time, to generate a performance index vector for anomaly detection.

[0027] The performance index vector is input into the variational autoencoder, and the variational autoencoder constructs a normal mode baseline of the multidimensional performance index. The encoder maps the performance index vector to the latent variable space to generate probability distribution parameters, and the decoder reconstructs the sampled latent variables back to the original feature space to establish a benchmark model for normal system operation.

[0028] Based on the baseline model, the anomaly detector continuously monitors performance fluctuations and load balancing status. By calculating the deviation between real-time performance indicators and the normal mode baseline, and combining load distribution status assessment, it generates a state description vector containing anomaly degree and load information. The state description vector is used as input, and combined with a graph temporal neural network, it predicts resource demand trends.

[0029] When an anomaly is detected, an adaptive optimization mechanism is triggered. This mechanism uses reinforcement transfer learning to generate task migration strategies and resource reallocation schemes. System optimization is achieved through progressive task migration and elastic resource adjustment. The optimization results are updated through knowledge distillation to update the multimodal spatiotemporal feature perception network, realizing knowledge accumulation and model evolution, including:

[0030] A multimodal spatiotemporal feature sensing network is used to monitor the status of computing nodes in real time and generate system anomaly detection results. When the multimodal spatiotemporal feature sensing network detects an anomaly, the anomaly detection results are input into an adaptive optimization mechanism. The adaptive optimization mechanism constructs a state space and an action space. The state space contains the anomaly detection results and real-time performance indicators, and the action space contains optional operations such as task migration and resource allocation.

[0031] The adaptive optimization mechanism is based on reinforcement transfer learning to construct an optimization network. The optimization network includes a main policy network and a target policy network. The state space information is input into the main policy network to generate an action probability distribution. The target policy network evaluates the value of the action and transfers the policy knowledge in historical optimization experience to the current scenario, outputting a task migration strategy and a resource reallocation scheme.

[0032] According to the task migration strategy, the task migration is performed in a gradual manner, the resource status of the target node is evaluated and migration checkpoints are established, incremental data transmission is performed according to the planned migration sequence, the running status of the migrated task is verified and the system topology information is updated.

[0033] Based on the resource reallocation scheme, perform elastic resource adjustment, dynamically adjust the number of processor cores, memory capacity limit and network bandwidth allocation of the migrated task, and generate optimized resource configuration parameters;

[0034] The optimization results of task migration and resource adjustment are fed back through knowledge distillation. A teacher network is built to store optimization experience, and the student network is used to learn the latest strategies. Knowledge transfer is guided by calculating the feature similarity loss function, integrating historical experience and new knowledge.

[0035] The fused knowledge is then updated to the multimodal spatiotemporal feature perception network to optimize the feature extraction and recognition capabilities of the anomaly detection model, improve the accuracy of subsequent anomaly detection, and form a closed-loop evolutionary mechanism of detection-tuning-optimization.

[0036] The optimization results of task migration and resource adjustment are fed back through knowledge distillation. A teacher network is constructed to store optimization experience, and a student network is used to learn the latest strategies. Knowledge transfer is guided by calculating a feature similarity loss function, integrating historical experience and new knowledge, including:

[0037] Collect task migration data and resource adjustment data from the computing system. Extract performance differences, resource consumption changes, and task completion time before and after migration from the task migration data. Extract resource allocation schemes, performance improvement degree, and system load status from the resource adjustment data. Generate optimization result vectors by characterizing the extracted data.

[0038] A teacher network is constructed to store optimization experience. The feature mapping layer of the teacher network receives the optimization result vector, extracts feature associations using a multilayer perceptron, highlights key information through an attention mechanism, and retains the original features using residual connections. The experience encoding layer of the teacher network uses recurrent units and memory modules to store the processed optimization experience.

[0039] The teacher network organizes the stored optimization experience into a scenario-strategy-effect mapping structure, establishes a decision tree index for optimization paths, uses incremental learning to update the optimization experience, eliminates inefficient experience based on a value evaluation mechanism, maintains the correlation between experiences, and forms an optimization experience knowledge base.

[0040] A student network is constructed to learn strategies for new scenarios. The student network uses lightweight convolutional layers to extract real-time input features and selects key features through pooling layers and attention mechanisms. The strategy generation module of the student network generates candidate schemes based on the extracted features and evaluates the network's prediction performance.

[0041] The similarity between the optimized experience features output by the teacher network and the policy features generated by the student network is calculated to obtain the feature similarity; a knowledge distillation loss function is constructed based on the feature similarity, which includes soft label cross-entropy loss and policy consistency constraint, and the degree of knowledge transfer is adjusted by a temperature parameter;

[0042] Knowledge transfer is carried out through a course learning approach. Training samples are selected based on the output of the knowledge distillation loss function, an increasing learning sequence is constructed, a knowledge graph is established to guide the transfer path, and the optimization experience is selectively inherited after evaluating the knowledge relevance.

[0043] The new optimization strategies learned by the student network are fed back to the optimization experience knowledge base, and the stored optimization experience is updated through the experience encoding layer of the teacher network, thereby realizing the dynamic integration of historical experience and new scenario knowledge.

[0044] A second aspect of the present invention,

[0045] Provides an AI-based automatic optimization system for distributed big data computing tasks, including:

[0046] The first unit is used to construct a multimodal spatiotemporal feature perception network. This network employs a hierarchical bidirectional long short-term memory (LSTM) unit to construct a temporal feature extraction layer, transforming task arrival time, task completion time, data dependencies, and computational load intensity from this layer into a multidimensional temporal feature sequence. A graph attention network is used to construct a spatial feature extraction layer, transforming the geographical distribution of computing nodes, network topology, computing power, storage capacity, and bandwidth resources from this layer into dynamic graph features. A dynamic weighted self-attention mechanism is used to adaptively fuse the multidimensional temporal feature sequence and the dynamic graph features to generate a heterogeneous feature tensor with spatiotemporal correlation. A contrastive learning method is then used to enhance the expressive power of this heterogeneous feature tensor.

[0047] The second unit is used to train a hierarchical hybrid decision network based on the heterogeneous feature tensors. The hierarchical hybrid decision network includes a global policy network and a local execution network. The global policy network adopts a hierarchical deep reinforcement learning architecture, extracts hierarchical features of task queue state, resource utilization state, and system load state through a transformation encoder, and generates global decisions for task scheduling and resource allocation by combining Monte Carlo tree search and priority experience replay mechanism. The local execution network adopts a graph neural network structure, optimizes task deployment location and resource allocation ratio within the local computing node cluster based on the global decisions, and achieves dynamic coordination of local decisions through a multi-agent collaborative learning mechanism.

[0048] The third unit is used to perform adaptive scheduling using the hierarchical hybrid decision network. During task execution, a distributed anomaly detection network is deployed. The distributed anomaly detection network constructs a normal mode baseline of multi-dimensional performance indicators based on a variational autoencoder. It monitors performance fluctuations and load balancing status in real time through anomaly detectors and predicts resource demand trends using a graph temporal neural network. When an anomaly is detected, an adaptive optimization mechanism is triggered. The adaptive optimization mechanism uses a reinforcement transfer learning method to generate task migration strategies and resource reallocation schemes. System optimization is achieved through progressive task migration and elastic resource adjustment. The optimization results are updated through knowledge distillation to update the multimodal spatiotemporal feature perception network, thereby achieving knowledge accumulation and model evolution.

[0049] A third aspect of the embodiments of the present invention,

[0050] An electronic device is provided, comprising:

[0051] processor;

[0052] Memory used to store processor-executable instructions;

[0053] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0054] Fourth aspect of the embodiments of the present invention,

[0055] A computer-readable storage medium is provided, having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0056] The beneficial effects of this application are as follows:

[0057] 1. Improved Resource Utilization and Task Execution Efficiency: This invention utilizes a multimodal spatiotemporal feature perception network and a hierarchical hybrid decision network to more accurately perceive task characteristics, resource status, and system load, thereby achieving better task scheduling and resource allocation. The global policy network, combined with Monte Carlo tree search and priority experience replay mechanisms, can generate more global and long-term decisions; the local execution network, through a multi-agent collaborative learning mechanism, dynamically coordinates local decisions, further improving resource utilization and task execution efficiency.

[0058] 2. Enhanced System Stability and Robustness: The distributed anomaly detection network deployed in this invention can monitor system performance in real time and predict resource demand trends using graph temporal neural networks, thereby identifying potential problems in advance. When anomalies are detected, the adaptive tuning mechanism can generate effective task migration strategies and resource reallocation schemes through enhanced transfer learning methods, achieving gradual task migration and elastic resource adjustment, ensuring stable system operation, and improving its robustness in the face of emergencies.

[0059] 3. Achieving Continuous Learning and Optimization: This invention utilizes a knowledge distillation method to feed the optimization results back to a multimodal spatiotemporal feature perception network, enabling knowledge accumulation and model evolution. This continuous learning mechanism allows the system to constantly adapt to new tasks and environmental changes, and continuously improve its performance over time. Attached Figure Description

[0060] Figure 1 is a flowchart illustrating the AI-based automatic optimization method for distributed big data computing tasks according to an embodiment of the present invention.

[0061] Figure 2 is a schematic diagram of the first AI-based automatic optimization technology process for big data distributed computing tasks according to an embodiment of the present invention;

[0062] Figure 3 is a schematic diagram of the second AI-based automatic optimization technology process for big data distributed computing tasks according to an embodiment of the present invention;

[0063] Figure 4 is a schematic diagram of the structure of the AI-based big data distributed computing task automatic optimization system according to an embodiment of the present invention. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0065] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0066] Figure 1 is a flowchart illustrating the AI-based automatic optimization method for distributed big data computing tasks according to an embodiment of the present invention. As shown in Figure 1, the method includes:

[0067] S11. Construct a multimodal spatiotemporal feature perception network. The multimodal spatiotemporal feature perception network uses a hierarchical bidirectional long short-term memory unit to construct a temporal feature extraction layer, which transforms the task arrival time, task completion time limit, data dependency relationship, and computational load intensity in the temporal feature extraction layer into a multidimensional temporal feature sequence. A graph attention network is used to construct a spatial feature extraction layer, which transforms the geographical distribution of computing nodes, network topology, computing power, storage capacity, and bandwidth resources in the spatial feature extraction layer into dynamic graph features. The multidimensional temporal feature sequence and the dynamic graph features are adaptively fused through a dynamic weight self-attention mechanism to generate a heterogeneous feature tensor with spatiotemporal correlation, and the expressive power of the heterogeneous feature tensor is enhanced through a contrastive learning method.

[0068] S12. A hierarchical hybrid decision network is trained based on the heterogeneous feature tensor. The hierarchical hybrid decision network includes a global policy network and a local execution network. The global policy network adopts a hierarchical deep reinforcement learning architecture, extracts hierarchical features of task queue state, resource utilization state, and system load state by transforming the encoder, and generates global decisions for task scheduling and resource allocation by combining Monte Carlo tree search and priority experience replay mechanism. The local execution network adopts a graph neural network structure, optimizes task deployment location and resource allocation ratio within the local computing node cluster based on the global decisions, and achieves dynamic coordination of local decisions through a multi-agent collaborative learning mechanism.

[0069] S13. Adaptive scheduling is performed using the hierarchical hybrid decision network. During task execution, a distributed anomaly detection network is deployed. The distributed anomaly detection network constructs a normal mode baseline for multidimensional performance indicators based on a variational autoencoder. It monitors performance fluctuations and load balancing status in real time through anomaly detectors and predicts resource demand trends using a graph temporal neural network. When an anomaly is detected, an adaptive optimization mechanism is triggered. The adaptive optimization mechanism uses a reinforcement transfer learning method to generate task migration strategies and resource reallocation schemes. System optimization is achieved through progressive task migration and elastic resource adjustment. The optimization results are updated through knowledge distillation to the multimodal spatiotemporal feature perception network, thereby achieving knowledge accumulation and model evolution.

[0070] In one optional implementation, a hierarchical hybrid decision network is trained based on the heterogeneous feature tensor. The hierarchical hybrid decision network includes a global policy network and a local execution network. The global policy network employs a hierarchical deep reinforcement learning architecture, extracting hierarchical features of task queue states, resource utilization states, and system load states through a transform encoder. It then combines Monte Carlo tree search and priority experience replay mechanisms to generate global decisions for task scheduling and resource allocation, including:

[0071] A hierarchical hybrid decision network is constructed, which includes a global policy network and a local execution network, wherein the global policy network adopts a hierarchical deep reinforcement learning architecture; the heterogeneous feature tensor is input into a transform encoder, and hierarchical features are extracted through the multi-head self-attention mechanism of the transform encoder. The hierarchical features are used to describe the multi-level correlation between task queue state, resource utilization state and system load state.

[0072] The hierarchical features are input into the global policy network to generate global decisions for task scheduling and resource allocation based on the hierarchical features; a Monte Carlo tree search is constructed, and the global decisions are used as the initial policy to guide the search process. Candidate decision sequences are obtained through tree search expansion; state evaluation values ​​are recorded at tree search nodes, and the optimal decision branch is selected based on the state evaluation values.

[0073] Design a priority experience replay mechanism to record decision trajectories and system feedback information; calculate the importance weights of the decision trajectories based on temporal difference errors, and construct a priority sampling probability distribution; sample high-value training data from the experience pool according to the priority sampling probability distribution, and use the high-value training data to optimize the global policy network;

[0074] The decision output of the global policy network is sent to the local execution network, which then generates specific scheduling instructions and resource allocation schemes.

[0075] A method based on training a hierarchical hybrid decision network using heterogeneous feature tensors is proposed to optimize task scheduling and resource allocation. This method includes a global policy network and a local execution network.

[0076] First, a hierarchical hybrid decision-making network is constructed. This network consists of a global policy network and a local execution network. The global policy network, employing a hierarchical deep reinforcement learning architecture, generates global decisions for task scheduling and resource allocation. The local execution network is responsible for translating these global decisions into specific scheduling instructions and resource allocation schemes.

[0077] Next, the heterogeneous feature tensor of the input is processed. This heterogeneous feature tensor contains information such as task queue status, resource utilization status, and system load status. This information is typically represented in different formats and dimensions; for example, the task queue can be represented by task ID, priority, and arrival time; resource utilization status can be represented by CPU utilization and memory usage; and system load status can be represented by average response time and request throughput. A transform encoder is used to process these heterogeneous features. The transform encoder uses a multi-head self-attention mechanism to extract hierarchical features. This mechanism allows multiple attention heads to focus on different parts of the input features in parallel, thus capturing multi-level relationships between features. For example, one attention head can focus on the priority of tasks in the task queue, while another can focus on resource availability, thereby linking task priority and resource availability. The extracted hierarchical features can more comprehensively describe the relationship between task queue status, resource utilization status, and system load status.

[0078] Next, the hierarchical features are fed into the global policy network. Based on these features, the global policy network generates global decisions for task scheduling and resource allocation. For example, the global policy network can decide to allocate high-priority tasks to idle resources or postpone the execution of low-priority tasks.

[0079] To further optimize global decision-making, a Monte Carlo tree search is constructed. The decisions generated by the global policy network are used as the initial policy to guide the search process. A sequence of candidate decisions is obtained through tree search expansion. A state evaluation value is recorded at each node of the tree search. The state evaluation value is used to assess the quality of the current state; for example, a pre-trained evaluation model or heuristic rules can be used to calculate the state evaluation value. The optimal decision branch is selected based on the state evaluation value. For example, the node with the highest state evaluation value is selected as the next expansion node.

[0080] To train the global policy network, a priority experience replay mechanism is designed. Decision trajectories and system feedback information are recorded. Decision trajectories include a series of decisions and state transitions, while system feedback information includes rewards or penalties. Importance weights of decision trajectories are calculated based on temporal difference errors. Temporal difference errors measure the impact of the current decision on future rewards. A priority sampling probability distribution is constructed. This distribution is built based on the importance weights of the decision trajectories; decision trajectories with higher importance weights have a higher probability of being sampled. For example, a decision trajectory that leads to system collapse will have a high importance weight and will therefore be sampled first. High-value training data is sampled from the experience pool according to the priority sampling probability distribution. This high-value training data is used to optimize the global policy network.

[0081] Finally, the decision output of the global policy network is sent to the local execution network. The local execution network generates specific scheduling instructions and resource allocation schemes. For example, if the global policy network decides to allocate task A to resource B, the local execution network generates specific instructions to transfer the data of task A to resource B and start the execution of task A on resource B.

[0082] For example, suppose there is a system with three tasks and two resources. The task queue state includes the priority and computation time of each task, the resource utilization state includes the availability and processing capacity of each resource, and the system load state includes the current average response time. A transformation encoder extracts hierarchical features, such as the correlation between task priority and resource availability. A global policy network generates global decisions based on these features, such as allocating high-priority, short-computation-time tasks to idle, high-capacity resources. Monte Carlo tree search further optimizes the global decisions, such as selecting the optimal scheme by simulating different scheduling schemes. A priority experience replay mechanism records the decision trajectory and system feedback information, such as recording the completion time and resource utilization of each task. Based on this information, the global policy network is trained to better adapt to different system load conditions. A local execution network translates the global decisions into specific scheduling instructions and resource allocation schemes, such as assigning tasks to specified resources and setting the task execution parameters.

[0083] The solution proposed in this application can:

[0084] Improve resource utilization: Through the collaborative work of the global policy network and the local execution network, resources can be allocated more effectively, avoiding waste and improving resource utilization. Reduce system latency: Through priority experience replay mechanism and Monte Carlo tree search, task scheduling strategy can be optimized, reducing task waiting time and lowering system latency. Enhance system stability: Through hierarchical feature extraction and hierarchical deep reinforcement learning architecture, changes in system state can be better captured and corresponding adjustments made, enhancing system stability.

[0085] In one optional implementation, the local execution network adopts a graph neural network structure, optimizes task deployment locations and resource allocation ratios within the local computing node cluster based on the global decision, and achieves dynamic coordination of local decisions through a multi-agent collaborative learning mechanism, including:

[0086] Receive the resource allocation scheme of the global decision network, the resource allocation scheme including the target area for task deployment, the baseline ratio of resource allocation and the constraints for load balancing;

[0087] Under the guidance of the resource allocation scheme, a graph neural network structure of a local computing node cluster is constructed. The nodes in the graph neural network structure include computing resource capabilities, current load status, and task execution records. The connection edges between nodes include network bandwidth, transmission latency, and connection stability.

[0088] Based on the graph neural network structure, the graph neural network is used to extract the state features of local computing node clusters, and the node information is aggregated through graph convolutional layers and attention layers to generate feature vectors that reflect the computing power and load distribution of nodes.

[0089] Based on the feature vector, a matching score between the task to be deployed and the computing node is calculated. The matching score takes into account the remaining resources of the node and the network connection quality, and generates a priority ranking for task deployment.

[0090] Based on the priority ranking, the task deployment location is optimized, and the resource allocation ratio of each task on the target computing node is determined by iterative adjustment. The resource allocation ratio includes CPU ratio, memory ratio, and bandwidth ratio.

[0091] The task deployment location and the resource allocation ratio are input into the multi-agent collaborative learning system. Each local execution network in the multi-agent collaborative learning system acts as an independent agent and makes decisions based on local observation information and the state of neighboring nodes. The multi-agent collaborative learning system enables dynamic coordination of local decisions.

[0092] Local execution networks optimize task deployment and resource allocation based on global decision-making, achieving efficient cluster resource management. This method utilizes graph neural networks and multi-agent collaborative learning to dynamically adjust task deployment and resource allocation within a local computing node cluster to adapt to constantly changing load and resource conditions.

[0093] First, the local execution network receives a resource allocation plan from the global decision-making network. This plan includes the target region for task deployment, the baseline ratio of resource allocation, and load balancing constraints. For example, the global decision-making network specifies that task A is deployed in region 1, with a baseline resource allocation ratio of 50% CPU, 30% memory, and 20% bandwidth, and requires that the load balance of region 1 be maintained above 90%.

[0094] Next, the local execution network constructs a graph neural network structure for the local computing node cluster based on the received resource allocation scheme. Nodes in this structure represent computing nodes and contain information such as computing resource capabilities, current load status, and task execution records. The edges connecting nodes represent the relationships between them and contain information such as network bandwidth, transmission latency, and connection stability. For example, node 1 has 16 CPU cores and 64GB of memory, with a current CPU load of 60% and a memory load of 40%. The connection bandwidth between node 1 and node 2 is 10Gbps, the transmission latency is 2ms, and the connection stability is 99%.

[0095] Then, the local execution network, based on the constructed graph neural network structure, uses the graph neural network to extract the state features of the local computing node cluster. Node information is aggregated through graph convolutional layers and attention layers to generate feature vectors reflecting the node's computing power and load distribution. For example, the feature vector of node 1 includes information such as its CPU resources, memory resources, current load status, and connection quality with its neighboring nodes.

[0096] Subsequently, the local execution network calculates a matching score between the tasks to be deployed and the computing nodes based on the generated feature vectors. This score comprehensively considers the remaining resources of the nodes and the quality of network connections to generate a priority ranking for task deployment. For example, if task A has a matching score of 0.8 with node 1 and a matching score of 0.6 with node 2, then task A will be deployed on node 1 first.

[0097] Next, the local execution network optimizes task deployment locations based on the generated priority ranking. Through iterative adjustments, it determines the resource allocation ratio for each task on the target computing node, including CPU, memory, and bandwidth. For example, task A is ultimately deployed on node 1, with an allocated resource ratio of 60% CPU, 20% memory, and 20% bandwidth.

[0098] Finally, the local execution network inputs the determined task deployment locations and resource allocation ratios into the multi-agent cooperative learning system. In this system, each local execution network acts as an independent agent, making decisions based on local observations and the states of neighboring nodes. The multi-agent cooperative learning system achieves dynamic coordination of local decisions. For example, node 1 and node 2 can dynamically adjust task deployment and resource allocation by exchanging their respective load states and task execution statuses to achieve globally optimal resource utilization.

[0099] The solution proposed in this application can:

[0100] Improve resource utilization: By dynamically adjusting task deployment locations and resource allocation ratios, the resources of computing nodes can be fully utilized, avoiding resource waste and improving overall resource utilization. Reduce task completion time: By deploying tasks to the most suitable computing nodes and dynamically adjusting resource allocation based on node load, task waiting and execution times can be reduced, thereby lowering task completion time. Enhance system robustness: Through a multi-agent collaborative learning mechanism, local execution networks can cooperate to jointly cope with unexpected situations, such as node failures or network congestion, thereby enhancing system robustness and stability.

[0101] In one optional implementation, the hierarchical hybrid decision network is used to perform adaptive scheduling. During task execution, a distributed anomaly detection network is deployed. This network constructs a normal mode baseline for multi-dimensional performance indicators based on a variational autoencoder. Anomaly detectors monitor performance fluctuations and load balancing status in real time. Combined with a graph temporal neural network, resource demand trends are predicted, including:

[0102] Adaptive scheduling is performed using the hierarchical hybrid decision network, which formulates an initial task deployment plan and resource allocation strategy based on the real-time status information of computing nodes, serving as a benchmark reference for subsequent anomaly detection.

[0103] Based on the initial task deployment scheme, a distributed anomaly detection network is deployed during task execution. The distributed anomaly detection network collects and standardizes the performance index data of each node in real time, including processor utilization, memory usage, network throughput, and task response time, to generate a performance index vector for anomaly detection.

[0104] The performance index vector is input into the variational autoencoder, and the variational autoencoder constructs a normal mode baseline of the multidimensional performance index. The encoder maps the performance index vector to the latent variable space to generate probability distribution parameters, and the decoder reconstructs the sampled latent variables back to the original feature space to establish a benchmark model for normal system operation.

[0105] Based on the baseline model, the anomaly detector continuously monitors performance fluctuations and load balancing status. By calculating the deviation between real-time performance indicators and the normal mode baseline, and combining load distribution status assessment, it generates a state description vector containing anomaly degree and load information. The state description vector is used as input, and combined with a graph temporal neural network, it predicts resource demand trends.

[0106] An adaptive resource scheduling method based on a hierarchical hybrid decision network and distributed anomaly detection is proposed, aiming to optimize resource utilization and task execution efficiency in a distributed computing environment. This method integrates hierarchical decision-making, anomaly detection, and prediction mechanisms, enabling dynamic adjustment of resource allocation strategies according to system operating status, thereby improving system stability and robustness.

[0107] First, a hierarchical hybrid decision-making network is constructed. This network combines expert knowledge and machine learning algorithms, enabling it to formulate initial task deployment plans and resource allocation strategies based on real-time status information of computing nodes, such as CPU utilization, memory usage, network bandwidth, and task queue length. For example, in a cluster with 10 computing nodes, the network formulates an initial plan based on the CPU and memory capacity of the nodes, allocating computationally intensive tasks to nodes with abundant CPU resources and memory-intensive tasks to nodes with abundant memory resources. This initial plan will serve as a benchmark for subsequent anomaly detection.

[0108] Next, after the task begins execution, a distributed anomaly detection network is deployed. This network is distributed across various computing nodes, collecting real-time performance metrics data from each node. The collected data includes CPU utilization, memory usage, network throughput, and task response time. To facilitate subsequent processing, the collected data is standardized, for example, using the Z-score standardization method to convert data of different dimensions into standardized data with a mean of 0 and a standard deviation of 1. The standardized data is then combined into a performance metric vector for anomaly detection. For example, if a node has a standardized CPU utilization of 0.8, a memory usage of 0.5, a network throughput of 0.9, and a task response time of 0.2, then the performance metric vector for that node would be [0.8, 0.5, 0.9, 0.2].

[0109] Then, a normal-mode baseline for multidimensional performance metrics is constructed using a variational autoencoder (VAE). The collected performance metric vectors are input into the VAE. The encoder maps the high-dimensional performance metric vectors to a low-dimensional latent variable space and generates the probability distribution parameters of these latent variables, such as mean and variance. The decoder then reconstructs the original performance metric vectors from the latent variables sampled from this probability distribution. By training the VAE to accurately reconstruct the performance metric vectors under normal operating conditions, a baseline model of normal system operation is established. For example, a trained VAE can reconstruct the input [0.8, 0.5, 0.9, 0.2] into [0.78, 0.52, 0.88, 0.21], which is very close to the original vector, indicating that the system is in a normal state.

[0110] Based on the established baseline model, the anomaly detector continuously monitors performance fluctuations and load balancing status. The anomaly detector calculates the deviation of real-time performance metrics from the normal mode baseline, such as reconstruction error. Simultaneously, it combines load distribution status assessment, such as calculating the load variance of nodes, to generate a state description vector containing anomaly degree and load information. For example, if a node has a large reconstruction error and its load is much higher than other nodes, the generated state description vector might be [0.5, 0.8], where 0.5 represents the anomaly degree and 0.8 represents the load imbalance degree.

[0111] Finally, the state description vector is used as input, combined with a graph temporal neural network to predict resource demand trends. The graph temporal neural network can capture the topological relationships and temporal dependencies between computing nodes and predict future resource demands based on the current system state. For example, based on the current state description vector [0.5, 0.8] and historical data, the graph temporal neural network predicts that the CPU utilization of this node will continue to increase in the near future, requiring increased CPU resource allocation. The hierarchical hybrid decision network dynamically adjusts task deployment schemes and resource allocation strategies based on the prediction results, such as migrating some tasks to nodes with lower loads or increasing the CPU resource quota of that node.

[0112] The solution proposed in this application can:

[0113] Improve resource utilization: By dynamically adjusting resource allocation strategies, resource waste is avoided and resource utilization efficiency is improved. Enhance system stability: Through real-time anomaly detection and prediction mechanisms, potential problems are promptly identified and addressed, ensuring stable system operation. Optimize task execution efficiency: Through adaptive scheduling, tasks are assigned to the most suitable computing nodes for execution, shortening task completion time and improving execution efficiency.

[0114] In one optional implementation, an adaptive optimization mechanism is triggered when an anomaly is detected. This mechanism employs reinforcement transfer learning to generate task migration strategies and resource reallocation schemes. System optimization is achieved through progressive task migration and elastic resource adjustment. The optimization results are then used to update the multimodal spatiotemporal feature perception network via knowledge distillation, enabling knowledge accumulation and model evolution.

[0115] A multimodal spatiotemporal feature sensing network is used to monitor the status of computing nodes in real time and generate system anomaly detection results. When the multimodal spatiotemporal feature sensing network detects an anomaly, the anomaly detection results are input into an adaptive optimization mechanism. The adaptive optimization mechanism constructs a state space and an action space. The state space contains the anomaly detection results and real-time performance indicators, and the action space contains optional operations such as task migration and resource allocation.

[0116] The adaptive optimization mechanism is based on reinforcement transfer learning to construct an optimization network. The optimization network includes a main policy network and a target policy network. The state space information is input into the main policy network to generate an action probability distribution. The target policy network evaluates the value of the action and transfers the policy knowledge in historical optimization experience to the current scenario, outputting a task migration strategy and a resource reallocation scheme.

[0117] According to the task migration strategy, the task migration is performed in a gradual manner, the resource status of the target node is evaluated and migration checkpoints are established, incremental data transmission is performed according to the planned migration sequence, the running status of the migrated task is verified and the system topology information is updated.

[0118] Based on the resource reallocation scheme, perform elastic resource adjustment, dynamically adjust the number of processor cores, memory capacity limit and network bandwidth allocation of the migrated task, and generate optimized resource configuration parameters;

[0119] The optimization results of task migration and resource adjustment are fed back through knowledge distillation. A teacher network is built to store optimization experience, and the student network is used to learn the latest strategies. Knowledge transfer is guided by calculating the feature similarity loss function, integrating historical experience and new knowledge.

[0120] The fused knowledge is then updated to the multimodal spatiotemporal feature perception network to optimize the feature extraction and recognition capabilities of the anomaly detection model, improve the accuracy of subsequent anomaly detection, and form a closed-loop evolutionary mechanism of detection-tuning-optimization.

[0121] An adaptive system optimization method based on a multimodal spatiotemporal feature perception network and reinforcement transfer learning is proposed to monitor and dynamically adjust computing nodes in real time to cope with system anomalies and improve overall performance. The core of this method is to construct a closed-loop evolutionary mechanism of detection-tuning-optimization.

[0122] First, a multimodal spatiotemporal feature perception network is constructed. This network integrates multiple data sources, such as CPU utilization, memory usage, network traffic, and task execution time, and converts this data into multimodal feature vectors. These feature vectors contain temporal information, such as feature change trends over a period of time, and spatial information, such as feature correlations between different computing nodes. For example, a sudden increase in CPU utilization on a node, accompanied by a surge in network traffic, may indicate an anomaly on that node. The perception network uses pre-trained deep learning models, such as Long Short-Term Memory (LSTM) networks or Graph Neural Networks (GNNs), to analyze these multimodal spatiotemporal features and output anomaly detection results, such as "Node 3 overload" or "Network congestion."

[0123] When the multimodal spatiotemporal feature perception network detects an anomaly, it triggers an adaptive tuning mechanism. This mechanism first constructs a state space and an action space. The state space contains the anomaly detection results and real-time performance metrics, such as "node 3 overload" and the CPU utilization and memory usage of node 3. The action space contains optional operations for task migration and resource allocation, such as migrating task A from node 3 to node 2, or increasing the memory capacity of node 2. Assuming the system has 3 nodes and 2 tasks, the state space can be represented as a vector containing the anomaly type, the CPU and memory usage of the 3 nodes, for example, ["node 3 overload", 0.9, 0.8, 0.2, 0.7, 0.3, 0.5]. The action space can be represented as a set containing all possible migration combinations and resource adjustment schemes, for example, {"migrate task 1 from node 3 to node 2 and increase the memory of node 2", "migrate task 2 from node 1 to node 3 and reduce the CPU cores of node 1"}.

[0124] The core of the adaptive tuning mechanism is a tuning network based on reinforcement transfer learning. This network consists of a main policy network and a target policy network. State space information is input into the main policy network to generate an action probability distribution. For example, if node 3 is overloaded, the probability of migrating the task from node 3 to other nodes will be higher. The target policy network evaluates the value of different actions, such as the degree of improvement in overall system performance after migrating the task. The key to reinforcement transfer learning is to transfer policy knowledge from historical optimization experience to the current scenario. For example, if migrating the task to node 2 was more effective in similar overload conditions in the past, the probability of migrating the task to node 2 will also be higher in the current scenario. Finally, the tuning network outputs the task migration strategy and resource reallocation scheme. Assuming the main policy network outputs a probability of 0.8 for migrating task 1 from node 3 to node 2 and a probability of 0.2 for migrating task 2 from node 1 to node 3, then the final choice is to migrate task 1 from node 3 to node 2.

[0125] Based on the generated strategy, the system performs a gradual task migration. First, it assesses the resource status of the target node (e.g., node 2) and establishes migration checkpoints. Then, it performs incremental data transmission according to the planned migration sequence to avoid network congestion caused by migrating a large amount of data at once. For example, the data for task 1 is divided into multiple small chunks and gradually transmitted to node 2. Finally, it verifies the running status of the migrated task and updates the system topology information to ensure that the task runs normally on the new node.

[0126] Simultaneously, the system performs elastic resource adjustments based on the resource reallocation scheme. For example, it dynamically adjusts the number of processor cores, memory capacity limits, and network bandwidth allocation of the migrated task, generating optimized resource configuration parameters. Assuming that node 2 initially has 4 CPU cores and 8GB of memory, after migrating task 1, the system adjusts its CPU core count to 6 and memory to 10GB.

[0127] The optimization results of task transfer and resource adjustment are fed back to the multimodal spatiotemporal feature perception network through knowledge distillation, thereby achieving knowledge accumulation and model evolution. Specifically, a teacher network is constructed to store optimization experience, while a student network learns the latest strategies. Knowledge transfer is guided by calculating a feature similarity loss function, integrating historical experience and new knowledge. Finally, the fused knowledge is updated to the multimodal spatiotemporal feature perception network to optimize the feature extraction and recognition capabilities of the anomaly detection model, improving the accuracy of subsequent anomaly detection and forming a closed-loop evolutionary mechanism of detection-tuning-optimization. For example, if knowledge distillation learns that the strategy of transferring the task to node 2 is more effective when node 2 has sufficient resources, then in similar situations, the multimodal spatiotemporal feature perception network will be more inclined to identify node 2 as a suitable transfer target.

[0128] The solution proposed in this application can:

[0129] Enhancing System Stability: This method can monitor system status in real time and respond promptly to anomalies, preventing system crashes or performance degradation, thereby improving system stability and reliability. Optimizing Resource Utilization: By dynamically adjusting task migration and resource allocation, this method can effectively balance the load on computing nodes, avoiding resource waste and improving resource utilization. Achieving Adaptive Optimization: This method employs reinforcement transfer learning and knowledge distillation techniques, enabling continuous learning and accumulation of optimization experience to achieve adaptive optimization of the system and continuously improve performance over time.

[0130] In one optional implementation, the optimization results of task migration and resource adjustment are fed back through knowledge distillation. A teacher network is constructed to store optimization experience, and a student network is used to learn the latest strategies. Knowledge transfer is guided by calculating a feature similarity loss function, integrating historical experience and new knowledge, including:

[0131] Collect task migration data and resource adjustment data from the computing system. Extract performance differences, resource consumption changes, and task completion time before and after migration from the task migration data. Extract resource allocation schemes, performance improvement degree, and system load status from the resource adjustment data. Generate optimization result vectors by characterizing the extracted data.

[0132] A teacher network is constructed to store optimization experience. The feature mapping layer of the teacher network receives the optimization result vector, extracts feature associations using a multilayer perceptron, highlights key information through an attention mechanism, and retains the original features using residual connections. The experience encoding layer of the teacher network uses recurrent units and memory modules to store the processed optimization experience.

[0133] The teacher network organizes the stored optimization experience into a scenario-strategy-effect mapping structure, establishes a decision tree index for optimization paths, uses incremental learning to update the optimization experience, eliminates inefficient experience based on a value evaluation mechanism, maintains the correlation between experiences, and forms an optimization experience knowledge base.

[0134] A student network is constructed to learn strategies for new scenarios. The student network uses lightweight convolutional layers to extract real-time input features and selects key features through pooling layers and attention mechanisms. The strategy generation module of the student network generates candidate schemes based on the extracted features and evaluates the network's prediction performance.

[0135] The similarity between the optimized experience features output by the teacher network and the policy features generated by the student network is calculated to obtain the feature similarity; a knowledge distillation loss function is constructed based on the feature similarity, which includes soft label cross-entropy loss and policy consistency constraint, and the degree of knowledge transfer is adjusted by a temperature parameter;

[0136] Knowledge transfer is carried out through a course learning approach. Training samples are selected based on the output of the knowledge distillation loss function, an increasing learning sequence is constructed, a knowledge graph is established to guide the transfer path, and the optimization experience is selectively inherited after evaluating the knowledge relevance.

[0137] The new optimization strategies learned by the student network are fed back to the optimization experience knowledge base, and the stored optimization experience is updated through the experience encoding layer of the teacher network, thereby realizing the dynamic integration of historical experience and new scenario knowledge.

[0138] A task migration and resource adjustment optimization method based on knowledge distillation is implemented in the following steps:

[0139] First, collect task migration data and resource adjustment data from the computing system. For example, record the CPU utilization, memory usage, network bandwidth usage, and task completion time before and after task A is migrated to server B; record the percentage increase in database query speed and changes in overall system load after adjusting the number of CPU cores on server C. Preprocess this raw data, such as normalizing and standardizing, and then extract key features. For example, extract performance differences before and after migration (e.g., throughput increased by 10%), changes in resource consumption (e.g., memory usage decreased by 20%), and changes in task completion time (e.g., a reduction of 5 seconds) from the task migration data. Extract resource allocation schemes (e.g., adjusting the number of CPU cores from 4 to 8), the degree of performance improvement (e.g., database query speed increased by 30%), and changes in system load status (e.g., the average system load increased from 0.8 to 1.2) from the resource adjustment data. Convert the extracted features into numerical vectors to form an optimization result vector, such as [0.1, -0.2, -5, 2, 0.3, 0.4].

[0140] Next, a teacher network is constructed to store optimization experience. The feature mapping layer of the teacher network receives the optimization result vector as input. This layer uses a multilayer perceptron to extract the correlations between features, such as the relationship between performance improvement and resource consumption reduction. Simultaneously, an attention mechanism is used to highlight key information; for example, for some tasks, changes in completion time are more important than changes in resource consumption. Furthermore, residual connections are used to preserve original feature information, preventing information loss during multilayer perceptron processing. The experience encoding layer of the teacher network uses recurrent units (e.g., LSTM) and a memory module to store the processed optimization experience. The recurrent units capture temporal relationships in the optimization result vector, while the memory module stores long-term dependencies. For example, if a resource adjustment strategy has performed well over the past month, this information is stored in the memory module. The teacher network organizes the stored optimization experience into a "scenario-strategy-effect" mapping structure. For example, "Increasing the number of CPU cores can improve performance in high-load scenarios." A decision tree index is built to index the optimization path for easy retrieval and matching. Simultaneously, the teacher network updates the optimization experience using incremental learning; for example, when a new optimization result vector is input, the teacher network updates it based on its similarity to existing experience. Furthermore, the teacher network also eliminates inefficient experiences based on a value evaluation mechanism. For example, if a strategy fails to perform well after multiple attempts, it will be eliminated. Finally, the teacher network maintains the relationships between experiences; for example, "increasing memory" and "increasing the number of CPU cores" are often used together to form an optimization experience knowledge base.

[0141] Then, a student network is constructed to learn policies for new scenarios. The student network uses lightweight convolutional layers to extract real-time input features, such as the current system's CPU utilization, memory usage, and network bandwidth consumption. Key features are selected through pooling layers and attention mechanisms. The student network's policy generation module generates candidate solutions based on the extracted features, such as increasing the number of CPU cores, increasing memory, or transferring tasks. An evaluation network is then used to predict the performance of each candidate solution, such as the degree of performance improvement and changes in resource consumption.

[0142] Next, the similarity between the optimization experience features output by the teacher network and the policy features generated by the student network is calculated. For example, the cosine similarity between the two feature vectors is calculated. A knowledge distillation loss function is constructed based on the feature similarity. This loss function includes soft-label cross-entropy loss and a policy consistency constraint. The soft-label cross-entropy loss measures the difference between the policy generated by the student network and the optimization experience provided by the teacher network. The policy consistency constraint encourages the policy generated by the student network to be consistent with the experience provided by the teacher network. The degree of knowledge transfer is adjusted by a temperature parameter. For example, the higher the temperature parameter, the smoother the degree of knowledge transfer.

[0143] Knowledge transfer is achieved through a course-based learning approach. Training samples are selected based on the output of the knowledge distillation loss function, constructing learning sequences with increasing difficulty. For example, simple scenarios are learned first, followed by complex ones. A knowledge graph is established to guide the transfer path; for example, the strategies of "increasing memory" and "increasing the number of CPU cores" are linked. After evaluating knowledge relevance, optimization experience is selectively inherited; for example, only experience relevant to the current scenario is inherited.

[0144] Finally, the new optimization strategies learned by the student network are fed back into the optimization experience knowledge base. The stored optimization experience is updated through the teacher network's experience encoding layer, achieving a dynamic fusion of historical experience and knowledge of new scenarios. For example, if the student network learns a new and effective resource adjustment strategy, it is added to the teacher network's experience knowledge base.

[0145] The solution proposed in this application can:

[0146] Improve resource utilization efficiency: By learning from and transferring historical optimization experience, suitable resource adjustment strategies can be found more quickly, avoiding unnecessary resource waste and thus improving resource utilization efficiency. For example, when facing new task loads, the system can quickly determine the optimal resource allocation scheme based on historical experience, avoiding resource surplus or shortage. Accelerate optimization convergence speed: With the guidance of the teacher network, the student network can learn effective optimization strategies more quickly, thereby accelerating optimization convergence speed. For example, when adjusting resources in new scenarios, the student network can draw on the experience of the teacher network, avoiding repeated exploration of ineffective strategies and finding the optimal solution more quickly. Enhance system adaptability: By continuously learning new scenarios and strategies, the system can continuously accumulate optimization experience, thereby enhancing system adaptability. For example, as business loads change, the system can dynamically adjust resource allocation strategies according to new situations, maintaining efficient system operation.

[0147] Figure 2-3 is a schematic diagram of the first and second AI-based automatic optimization technology processes for distributed big data computing tasks according to embodiments of the present invention. As shown in Figure 2-3, the method further includes:

[0148] Establish task optimization standards: On the test cluster or idle business cluster, use industry-standard benchmarks such as 10TB TeraSort as test examples to test scenarios with different file sizes, file counts, data skew, etc., and fine-tune the computing task parameters to achieve the theoretical optimal performance of the computing task. Use the various performance indicators at this time as the standard values.

[0149] Collect and analyze task data: Collect the operation logs of all tasks scheduled and running on the big data cluster over a period of time; calculate and analyze the data of the computing tasks during the operation to determine the resource request status, peak usage, waste rate, performance indicators, etc. of each task; and further analyze the data according to some dimensions.

[0150] Task Problem Classification and Labeling: AI is used to perform cluster analysis on the preliminary analysis results from step two, calculating the categories of problems that cause approximately 80% of all resource waste and a feature library of high-performance tasks, thus improving the performance model. The results are then compared to the standard values ​​from step one to classify and label each task with problems, such as high memory allocation but low peak memory usage, too many subtasks, or severe data skew in certain subtasks.

[0151] Training task optimization model: Based on the standard values ​​and standard task parameters from step one, the parameters for different problem types are specifically tuned and trained in step two until the optimal performance for that type of problem is achieved. The corresponding parameter configuration at this point is the expected optimal parameter configuration for that task type.

[0152] Automatic parameter adjustment before task scheduling: In the preparation stage before task scheduling, the original configuration parameters of the task are overwritten based on the tags on the task and the optimal parameters of the problem type corresponding to the training model, and then submitted to the scheduling system and big data cluster for operation.

[0153] The solution proposed in this application can:

[0154] The method provided by the embodiments of the present invention can reliably improve task performance and automatically optimize big data distributed computing tasks. It ensures that tasks are always in the best performance state without human intervention, saving enterprises more than 35% of computing resources under the current cluster scale, and achieving huge benefits in cost reduction and efficiency improvement of big data clusters.

[0155] Figure 4 is a schematic diagram of the structure of the AI-based big data distributed computing task automatic optimization system according to an embodiment of the present invention. As shown in Figure 4, the system includes:

[0156] The first unit is used to construct a multimodal spatiotemporal feature perception network. This network employs a hierarchical bidirectional long short-term memory (LSTM) unit to construct a temporal feature extraction layer, transforming task arrival time, task completion time, data dependencies, and computational load intensity from this layer into a multidimensional temporal feature sequence. A graph attention network is used to construct a spatial feature extraction layer, transforming the geographical distribution of computing nodes, network topology, computing power, storage capacity, and bandwidth resources from this layer into dynamic graph features. A dynamic weighted self-attention mechanism is used to adaptively fuse the multidimensional temporal feature sequence and the dynamic graph features to generate a heterogeneous feature tensor with spatiotemporal correlation. A contrastive learning method is then used to enhance the expressive power of this heterogeneous feature tensor.

[0157] The second unit is used to train a hierarchical hybrid decision network based on the heterogeneous feature tensors. The hierarchical hybrid decision network includes a global policy network and a local execution network. The global policy network adopts a hierarchical deep reinforcement learning architecture, extracts hierarchical features of task queue state, resource utilization state, and system load state through a transformation encoder, and generates global decisions for task scheduling and resource allocation by combining Monte Carlo tree search and priority experience replay mechanism. The local execution network adopts a graph neural network structure, optimizes task deployment location and resource allocation ratio within the local computing node cluster based on the global decisions, and achieves dynamic coordination of local decisions through a multi-agent collaborative learning mechanism.

[0158] The third unit is used to perform adaptive scheduling using the hierarchical hybrid decision network. During task execution, a distributed anomaly detection network is deployed. The distributed anomaly detection network constructs a normal mode baseline of multi-dimensional performance indicators based on a variational autoencoder. It monitors performance fluctuations and load balancing status in real time through anomaly detectors and predicts resource demand trends using a graph temporal neural network. When an anomaly is detected, an adaptive optimization mechanism is triggered. The adaptive optimization mechanism uses a reinforcement transfer learning method to generate task migration strategies and resource reallocation schemes. System optimization is achieved through progressive task migration and elastic resource adjustment. The optimization results are updated through knowledge distillation to update the multimodal spatiotemporal feature perception network, thereby achieving knowledge accumulation and model evolution.

[0159] A third aspect of the present invention,

[0160] An electronic device is provided, comprising:

[0161] processor;

[0162] Memory used to store processor-executable instructions;

[0163] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0164] Fourth aspect of the present invention,

[0165] A computer-readable storage medium is provided, having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0166] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0167] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An AI-based automatic optimization method for distributed big data computing tasks, characterized in that, include: A multimodal spatiotemporal feature perception network is constructed. The multimodal spatiotemporal feature perception network uses a hierarchical bidirectional long short-term memory unit to construct a temporal feature extraction layer, and transforms the task arrival time, task completion time limit, data dependency relationship, and computational load intensity in the temporal feature extraction layer into a multidimensional temporal feature sequence. A spatial feature extraction layer is constructed using a graph attention network, which transforms the geographical distribution, network topology, computing power, storage capacity, and bandwidth resources of the computing nodes in the spatial feature extraction layer into dynamic graph features. The multidimensional temporal feature sequence and the dynamic graph feature are adaptively fused by a dynamic weight self-attention mechanism to generate a heterogeneous feature tensor with spatiotemporal correlation, and the expressive power of the heterogeneous feature tensor is enhanced by a contrastive learning method. A hierarchical hybrid decision network is trained based on the heterogeneous feature tensors. The hierarchical hybrid decision network includes a global policy network and a local execution network. The global policy network adopts a hierarchical deep reinforcement learning architecture, extracts hierarchical features of task queue state, resource utilization state, and system load state through a transformation encoder, and generates global decisions for task scheduling and resource allocation by combining Monte Carlo tree search and priority experience replay mechanism. The local execution network adopts a graph neural network structure, optimizes task deployment location and resource allocation ratio within the local computing node cluster based on the global decisions, and achieves dynamic coordination of local decisions through a multi-agent collaborative learning mechanism. Adaptive scheduling is performed using the hierarchical hybrid decision network. During task execution, a distributed anomaly detection network is deployed. This network constructs a normal mode baseline for multidimensional performance indicators based on a variational autoencoder. It monitors performance fluctuations and load balancing status in real time using anomaly detectors and predicts resource demand trends using a graph temporal neural network. When anomalies are detected, an adaptive optimization mechanism is triggered. This mechanism uses reinforcement transfer learning to generate task migration strategies and resource reallocation schemes. System optimization is achieved through progressive task migration and elastic resource adjustment. The optimization results are updated through knowledge distillation to the multimodal spatiotemporal feature perception network, achieving knowledge accumulation and model evolution.

2. The method according to claim 1, characterized in that, A hierarchical hybrid decision network is trained based on the heterogeneous feature tensors. This network comprises a global policy network and local execution networks. The global policy network employs a hierarchical deep reinforcement learning architecture, extracting hierarchical features of task queue states, resource utilization states, and system load states through a transform encoder. Combined with Monte Carlo tree search and priority experience replay mechanisms, it generates global decisions for task scheduling and resource allocation, including: A hierarchical hybrid decision network is constructed, which includes a global policy network and a local execution network, wherein the global policy network adopts a hierarchical deep reinforcement learning architecture; the heterogeneous feature tensor is input into a transform encoder, and hierarchical features are extracted through the multi-head self-attention mechanism of the transform encoder. The hierarchical features are used to describe the multi-level correlation between task queue state, resource utilization state and system load state. The hierarchical features are input into the global policy network to generate global decisions for task scheduling and resource allocation based on the hierarchical features; a Monte Carlo tree search is constructed, and the global decisions are used as the initial policy to guide the search process. Candidate decision sequences are obtained through tree search expansion; state evaluation values ​​are recorded at tree search nodes, and the optimal decision branch is selected based on the state evaluation values. Design a priority experience replay mechanism to record decision trajectories and system feedback information; calculate the importance weights of the decision trajectories based on temporal difference errors, and construct a priority sampling probability distribution; sample high-value training data from the experience pool according to the priority sampling probability distribution, and use the high-value training data to optimize the global policy network; The decision output of the global policy network is sent to the local execution network, which then generates specific scheduling instructions and resource allocation schemes.

3. The method according to claim 1, characterized in that, The local execution network adopts a graph neural network structure. Based on the global decision, it optimizes the task deployment location and resource allocation ratio within the local computing node cluster. Dynamic coordination of local decisions is achieved through a multi-agent collaborative learning mechanism, including: Receive the resource allocation scheme of the global decision network, the resource allocation scheme including the target area for task deployment, the baseline ratio of resource allocation and the constraints for load balancing; Under the guidance of the resource allocation scheme, a graph neural network structure of a local computing node cluster is constructed. The nodes in the graph neural network structure include computing resource capabilities, current load status, and task execution records. The connection edges between nodes include network bandwidth, transmission latency, and connection stability. Based on the graph neural network structure, the graph neural network is used to extract the state features of local computing node clusters, and the node information is aggregated through graph convolutional layers and attention layers to generate feature vectors that reflect the computing power and load distribution of nodes. Based on the feature vector, a matching score between the task to be deployed and the computing node is calculated. The matching score takes into account the remaining resources of the node and the network connection quality, and generates a priority ranking for task deployment. Based on the priority ranking, the task deployment location is optimized, and the resource allocation ratio of each task on the target computing node is determined by iterative adjustment. The resource allocation ratio includes CPU ratio, memory ratio, and bandwidth ratio. The task deployment location and the resource allocation ratio are input into the multi-agent collaborative learning system. Each local execution network in the multi-agent collaborative learning system acts as an independent agent and makes decisions based on local observation information and the state of neighboring nodes. The multi-agent collaborative learning system enables dynamic coordination of local decisions.

4. The method according to claim 1, characterized in that, Adaptive scheduling is performed using the hierarchical hybrid decision network. During task execution, a distributed anomaly detection network is deployed. This network constructs a normal mode baseline for multi-dimensional performance indicators based on a variational autoencoder. Anomaly detectors monitor performance fluctuations and load balancing status in real time. Combined with a graph temporal neural network, resource demand trends are predicted, including: Adaptive scheduling is performed using the hierarchical hybrid decision network, which formulates an initial task deployment plan and resource allocation strategy based on the real-time status information of computing nodes, serving as a benchmark reference for subsequent anomaly detection. Based on the initial task deployment scheme, a distributed anomaly detection network is deployed during task execution. The distributed anomaly detection network collects and standardizes the performance index data of each node in real time, including processor utilization, memory usage, network throughput, and task response time, to generate a performance index vector for anomaly detection. The performance index vector is input into the variational autoencoder, and the variational autoencoder constructs a normal mode baseline of the multidimensional performance index. The encoder maps the performance index vector to the latent variable space to generate probability distribution parameters, and the decoder reconstructs the sampled latent variables back to the original feature space to establish a benchmark model for normal system operation. Based on the baseline model, the anomaly detector continuously monitors performance fluctuations and load balancing status. By calculating the deviation between real-time performance indicators and the normal mode baseline, and combining load distribution status assessment, it generates a state description vector containing anomaly degree and load information. The state description vector is used as input, and combined with a graph temporal neural network, it predicts resource demand trends.

5. The method according to claim 1, characterized in that, When an anomaly is detected, an adaptive optimization mechanism is triggered. This mechanism uses reinforcement transfer learning to generate task migration strategies and resource reallocation schemes. System optimization is achieved through progressive task migration and elastic resource adjustment. The optimization results are updated through knowledge distillation to update the multimodal spatiotemporal feature perception network, realizing knowledge accumulation and model evolution, including: A multimodal spatiotemporal feature sensing network is used to monitor the status of computing nodes in real time and generate system anomaly detection results. When the multimodal spatiotemporal feature sensing network detects an anomaly, the anomaly detection results are input into an adaptive optimization mechanism. The adaptive optimization mechanism constructs a state space and an action space. The state space contains the anomaly detection results and real-time performance indicators, and the action space contains optional operations such as task migration and resource allocation. The adaptive optimization mechanism is based on reinforcement transfer learning to construct an optimization network. The optimization network includes a main policy network and a target policy network. The state space information is input into the main policy network to generate an action probability distribution. The target policy network evaluates the value of the action and transfers the policy knowledge in historical optimization experience to the current scenario, outputting a task migration strategy and a resource reallocation scheme. According to the task migration strategy, the task migration is performed in a gradual manner, the resource status of the target node is evaluated and migration checkpoints are established, incremental data transmission is performed according to the planned migration sequence, the running status of the migrated task is verified and the system topology information is updated. Based on the resource reallocation scheme, perform elastic resource adjustment, dynamically adjust the number of processor cores, memory capacity limit and network bandwidth allocation of the migrated task, and generate optimized resource configuration parameters; The optimization results of task migration and resource adjustment are fed back through knowledge distillation. A teacher network is built to store optimization experience, and the student network is used to learn the latest strategies. Knowledge transfer is guided by calculating the feature similarity loss function, integrating historical experience and new knowledge. The fused knowledge is then updated to the multimodal spatiotemporal feature perception network to optimize the feature extraction and recognition capabilities of the anomaly detection model, improve the accuracy of subsequent anomaly detection, and form a closed-loop evolutionary mechanism of detection-tuning-optimization.

6. The method according to claim 5, characterized in that, The optimization results of task migration and resource adjustment are fed back through knowledge distillation. A teacher network is constructed to store optimization experience, and a student network is used to learn the latest strategies. Knowledge transfer is guided by calculating a feature similarity loss function, integrating historical experience and new knowledge, including: Collect task migration data and resource adjustment data from the computing system. Extract performance differences, resource consumption changes, and task completion time before and after migration from the task migration data. Extract resource allocation schemes, performance improvement degree, and system load status from the resource adjustment data. Generate optimization result vectors by characterizing the extracted data. A teacher network is constructed to store optimization experience. The feature mapping layer of the teacher network receives the optimization result vector, extracts feature associations using a multilayer perceptron, highlights key information through an attention mechanism, and retains the original features using residual connections. The experience encoding layer of the teacher network uses recurrent units and memory modules to store the processed optimization experience. The teacher network organizes the stored optimization experience into a scenario-strategy-effect mapping structure, establishes a decision tree index for optimization paths, uses incremental learning to update the optimization experience, eliminates inefficient experience based on a value evaluation mechanism, maintains the correlation between experiences, and forms an optimization experience knowledge base. A student network is constructed to learn strategies for new scenarios. The student network uses lightweight convolutional layers to extract real-time input features and selects key features through pooling layers and attention mechanisms. The strategy generation module of the student network generates candidate schemes based on the extracted features and evaluates the network's prediction performance. The similarity between the optimized experience features output by the teacher network and the policy features generated by the student network is calculated to obtain the feature similarity; a knowledge distillation loss function is constructed based on the feature similarity, which includes soft label cross-entropy loss and policy consistency constraint, and the degree of knowledge transfer is adjusted by a temperature parameter; Knowledge transfer is carried out through a course learning approach. Training samples are selected based on the output of the knowledge distillation loss function, an increasing learning sequence is constructed, a knowledge graph is established to guide the transfer path, and the optimization experience is selectively inherited after evaluating the knowledge relevance. The new optimization strategies learned by the student network are fed back to the optimization experience knowledge base, and the stored optimization experience is updated through the experience encoding layer of the teacher network, thereby realizing the dynamic integration of historical experience and new scenario knowledge.

7. An AI-based automatic optimization system for distributed big data computing tasks, used to implement the method of any one of claims 1-6, characterized in that, include: The first unit is used to construct a multimodal spatiotemporal feature perception network. The multimodal spatiotemporal feature perception network uses a hierarchical bidirectional long short-term memory unit to construct a temporal feature extraction layer, which transforms the task arrival time, task completion time limit, data dependency relationship, and computational load intensity in the temporal feature extraction layer into a multidimensional temporal feature sequence. A spatial feature extraction layer is constructed using a graph attention network, which transforms the geographical distribution, network topology, computing power, storage capacity, and bandwidth resources of the computing nodes in the spatial feature extraction layer into dynamic graph features. The multidimensional temporal feature sequence and the dynamic graph feature are adaptively fused by a dynamic weight self-attention mechanism to generate a heterogeneous feature tensor with spatiotemporal correlation, and the expressive power of the heterogeneous feature tensor is enhanced by a contrastive learning method. The second unit is used to train a hierarchical hybrid decision network based on the heterogeneous feature tensors. The hierarchical hybrid decision network includes a global policy network and a local execution network. The global policy network adopts a hierarchical deep reinforcement learning architecture, extracts hierarchical features of task queue state, resource utilization state, and system load state through a transformation encoder, and generates global decisions for task scheduling and resource allocation by combining Monte Carlo tree search and priority experience replay mechanism. The local execution network adopts a graph neural network structure, optimizes task deployment location and resource allocation ratio within the local computing node cluster based on the global decisions, and achieves dynamic coordination of local decisions through a multi-agent collaborative learning mechanism. The third unit is used to perform adaptive scheduling using the hierarchical hybrid decision network. During task execution, a distributed anomaly detection network is deployed. The distributed anomaly detection network constructs a normal mode baseline of multi-dimensional performance indicators based on a variational autoencoder. It monitors performance fluctuations and load balancing status in real time through anomaly detectors and predicts resource demand trends using a graph temporal neural network. When an anomaly is detected, an adaptive optimization mechanism is triggered. The adaptive optimization mechanism uses a reinforcement transfer learning method to generate task migration strategies and resource reallocation schemes. System optimization is achieved through progressive task migration and elastic resource adjustment. The optimization results are updated through knowledge distillation to update the multimodal spatiotemporal feature perception network, thereby achieving knowledge accumulation and model evolution.

8. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 6.

Images