76 results about "Workflow scheduling" patented technology

The invention relates to a calculation-storage stream joint scheduling optimization method and system based on DDQN and a heuristic strategy, and belongs to the technical field of distributed calculation and intelligent scheduling. The calculation-storage stream joint scheduling optimization method comprises the following steps: (1) defining a calculation stream and a storage stream; (2) workflow scheduling and execution: when the workflow is triggered, a scheduler evaluates the executable states of the tasks one by one according to the dependency relationship between the tasks and a scheduling strategy, and submits the executable tasks to a task executor; the task executor is responsible for executing the tasks on the working nodes and ensuring that the tasks are completed according to a preset sequence and priority; (3) collecting a DAG structure, queue information and an operation number; (4) task operation time and resource use prediction based on historical operation data; (5) scheduling optimization based on DDQN and a heuristic strategy; and (6) acquiring and storing scheduling information. The task execution efficiency and the resource utilization rate of the system are remarkably improved.

The invention discloses a hybrid heterogeneous cloud workflow scheduling method based on reinforcement learning, and belongs to the technical field of cloud computing. The method comprises the following steps: aiming at a cloud workflow scheduling problem, by taking minimization of completion time as a target and taking cost and resources as constraints, a three-dimensional collaborative constraint model is constructed, and the cost constraints comprise server-free function budget and virtual machine budget; integrating the hyper-heuristic framework into a reinforcement learning algorithm; an improved reinforcement learning algorithm is adopted to solve the cloud workflow scheduling problem, optimal execution resources are selected, and an optimal scheduling scheme is obtained; and performing real-time scheduling according to the optimal scheduling scheme, and introducing a deviation feedback mechanism to monitor an execution error in real time. According to the method, the workflow scheduling problem is decomposed into a closed-loop optimization process of state perception and action decision, dynamic environment perception, multi-target tradeoff and online strategy optimization are deeply fused, the global search function is achieved, local optimization can be achieved, the algorithm complexity is low, and robustness is high.

The invention discloses an AI agent construction system based on a vector knowledge base and large model workflow, and belongs to the technical field of process management. The problem that the whole process of workflow scheduling and quality control cannot be managed smoothly due to the fact that an existing AI system cannot assemble heterogeneous and unstructured domain knowledge and business rules dispersed in internal documents, databases, APIs and expert experience through a standardized and low-code platform in a rapid and high-quality mode is solved. Comprising a project initialization module, a knowledge base configuration module, a model service module, a workflow arrangement module, a test verification module and a deployment monitoring module. According to the method, data, knowledge, reasoning and quality control are communicated through a unified vector knowledge base and a low-code drag workflow, a dynamic hypergraph is driven by using computable information potential energy / cognitive entropy, automatic topology operation cutting circulation and shortcut establishment are performed before deadlock outbreak, and predictable hard constraint management of knowledge fusion, process arrangement and cost process is realized.

The invention relates to a side cloud collaborative adaptive workflow scheduling method and system, and belongs to the technical field of distributed computing and artificial intelligence. The method comprises the following steps of: firstly, in a macroscopic candidate screening stage, reducing problem granularity through task clustering, and obtaining balance between utilization and exploration based on a weighted distance probabilistic preferential strategy; then, in a collaborative scheduling decision-making stage, a global network state diagram is constructed through a graph neural network, deep spatial features of nodes and neighborhoods of the nodes are extracted, context-aware state representation is formed, and a reinforcement learning agent makes an optimal collaborative decision in multiple options such as local execution, edge migration or cloud unloading according to the state representation; and finally, in a local adaptive optimization stage, performing fine-grained optimization after the task is issued, dynamically adjusting a scheduling frequency and a multi-target weight through an online learning mechanism, realizing balance between a task deadline and a resource utilization rate, and ensuring efficient and robust execution of a node level.

The invention discloses a fault-tolerant workflow scheduling method and system supporting dependence and failure retry of multiple upstream nodes. According to the method, the scheduling configuration, the dependency relationship and the real-time state of each node are maintained through task metadata storage; after the scheduler triggers task execution, the state monitor updates an execution result; when the task fails, the retry coordinator switches the task into a retry state according to a pre-configuration strategy and calculates retry time; before a scheduler triggers a downstream task, the states of all upstream nodes are checked by relying on an analyzer, and if an unsuccessful node exists, the downstream task is marked as waiting and placed in a waiting queue; and when the upstream node succeeds, traversing the waiting queue by depending on the resolver, carrying out secondary verification on the downstream task depending on the node, and only when all the upstream nodes succeed, recovering the upstream nodes to be in a to-be-scheduled state. According to the method, closed-loop linkage of retry and waiting is realized, dependency consistency under a heterogeneous scheduling strategy is supported, and the reliability, resource utilization rate and observability of the workflow are remarkably improved.

The invention relates to the technical field of ground disaster prevention and control report generation, in particular to an intelligent ground disaster prevention and control report generation method and system based on indoor and outdoor work integration. Retrieving geological information of the associated area, and determining the type of a hidden danger point; matching a target template; generating a target template directory structure; calling a workflow agent module according to a directory structure sequence to generate a first draft text; inserting chart data into the first draft text to obtain a complete report; and performing compliance examination and optimization on the complete report, and outputting a final report. The generation system comprises a data acquisition module, an intelligent retrieval engine, a template management unit, a workflow scheduling center and an examination optimization module. The data acquisition module, the intelligent retrieval engine, the template management unit, the workflow scheduling center and the review optimization module are connected in sequence and realize data transmission; synchronous real-time updating of indoor and outdoor data is met, data dispersion and information omission are avoided, and the generation efficiency and accuracy of a ground disaster prevention and control report are improved.

The invention discloses a zero-code heterogeneous data integration method. The method comprises the following steps: S1, configuring a multi-source data connector; s2, processing differences of different data formats and structures by adopting an adapter mode; s3, performing data cleaning; s4, performing data conversion; s5, loading the converted data into a target database or a data warehouse, and adopting a partition loading and incremental updating strategy; s6, providing a graphical user interface, and constructing a data integration workflow by dragging the component and connecting the data flow; s7, providing a visual workflow management tool, and supporting workflow version control, collaborative editing and parallel processing; s8, realizing an automatic workflow scheduling system, and supporting timed tasks and event driving; s9, performing data consistency check and integrity verification; and S10, a modular and extensible system architecture design is adopted. A zero code integration method is adopted, efficient integration of heterogeneous data sources is achieved, and the method has the advantages of being high in usability, high in processing efficiency and capable of guaranteeing data quality.

The invention provides a spark task running state monitoring method and device and a storage medium. The method comprises the steps of obtaining workflow data running in a spark cluster; a data monitor in a kubernetes cluster is adopted to monitor the workflow data running in the spark cluster, a running state corresponding to the workflow data is determined according to the workflow data, and the running state is one of running overtime, running failure and queuing waiting; and when the running state of the workflow data is running timeout, terminating the spark task corresponding to the workflow data, and releasing machine resources occupied by the spark task. According to the method, the corresponding processing flow is executed according to the monitored resource change based on the watch mechanism of the k8s container platform, so that the monitoring efficiency of the spark task scheduled by the workflow scheduling system is improved, and the resource occupation is reduced.

The invention discloses a CAE (Computer Aided Engineering) workflow control subsystem and method based on cloud computing, and belongs to the technical field of crossing of computer simulation and cloud computing, and the subsystem comprises a workflow configuration analysis module which is used for converting a CAE workflow configuration file possibly containing cyclic dependence into a directed acyclic graph state machine through strong connectivity component analysis; the workflow state machine operation module is used for dynamically deriving task groups which can be executed in parallel in the next stage in the state machine according to the current state; and the workflow scheduling module is used for submitting the tasks to a cloud solving platform in parallel and realizing fault recovery through state persistence. Through deep fusion of a graph theory method and a cloud computing architecture, automatic scheduling, efficient parallel execution and reliable fault tolerance of the CAE workflow are realized, and the computing resource utilization rate and the system reliability are remarkably improved.

A workflow scheduling fault-tolerant system based on a Watcher mechanism of a ZooKeeper comprises the ZooKeeper, the ZooKeeper is connected with at least one workflow executor and at least one workflow scheduler, and the ZooKeeper is further used for receiving a live event. The invention further provides a workflow scheduling fault-tolerant method based on the Watcher mechanism of the ZooKeeper and a workflow scheduling fault-tolerant system based on the Watcher mechanism of the ZooKeeper, and the workflow scheduling fault-tolerant method based on the Watcher mechanism of the ZooKeeper is characterized by comprising the following steps that the workflow scheduling fault-tolerant method comprises the steps that the workflow scheduling fault-tolerant method comprises the steps that the workflow scheduling The method comprises the following steps: S1, constructing a mapping relation between a workflow model and a ZooKeeper node; s2, task execution and state monitoring; s3, fault detection and processing; s4, fault recovery and task re-triggering; and S5, dynamic task adjustment and load balancing.

A dynamic competition perception workflow scheduling method for a heterogeneous computing system comprises the following steps: constructing a scheduling problem model based on a scheduling object set and bus resources of an obtained system; constructing a dynamic competition perception simulator according to the scheduling problem model, wherein the method comprises the following steps of: decomposing a time-varying behavior of cross-domain bus data transmission into a time-invariant meta-process; each meta-process is mapped into a timeout event carrying a specific timestamp; the completion event indicates that all the timeout events to which the same scheduling object belongs are completed, so that the end of the task or cross-domain bus data transmission is marked; according to an event-driven mechanism, a completion event is arranged on the premise that the preorder constraint of a scheduling object is met, and total completion time estimation is generated while integration of multiple scheduling strategies is supported; according to the total completion time estimation, generating an approximate optimal scheduling solution by using a hybrid earliest completion time genetic algorithm; and optimizing the approximate optimal scheduling solution by adopting a quota weighted ant colony algorithm to generate a final scheduling solution.

The present disclosure relates to a workflow scheduling method and device, and a computer readable storage medium. The workflow scheduling method comprises: defining a workflow according to user input, wherein the workflow is a process arranged for a plurality of tasks, the plurality of tasks are tasks included in a model training whole process, and the workflow comprises a plurality of tasks in data labeling and model evaluation in the model training whole process; and scheduling execution of the workflow, wherein for different tasks, pre-configured program data in shared storage is run according to a configured type to execute the tasks. In the labeling and effect evaluation process in the model training process, the present disclosure can automatically link the preceding and subsequent tasks, thereby reducing the communication cost of algorithm personnel and labeling personnel.

The invention discloses a CHUNJUN data synchronization workflow scheduling method and system based on a directed acyclic graph, and the method comprises the following steps: (1) packaging a CHUNJUN data synchronization task into standardized components, and exposing an input interface, an output interface and a configuration interface of each component; (2) receiving user operation through a visual interface to establish a component connection relationship, and executing connection normalization check; (3) constructing a directed acyclic graph (DAG) based on the connection relationship, and generating a task execution sequence; (4) triggering a CHUNJUN data synchronization task according to the execution sequence; according to the scheme, the construction time of a single task is compressed to be within 2 hours from 5-6 hours of a traditional command line mode, and the configuration error rate of a novice user is reduced by 60% through visual operation; and meanwhile, a DAG scheduling mechanism guarantees the reliability of multi-task sequential execution, the data synchronization failure rate is reduced by 60%, and the data intermediate station construction efficiency is remarkably improved.

The application discloses a space-air-ground integrated vehicle networking slice-based cooperative task offloading method, and the design steps comprise the following steps: firstly, a service-oriented RAN slice framework is designed; then, a maximum long-term task completion number problem is decoupled into a slice window length division problem, a resource allocation problem and a cooperative workflow scheduling problem; a MEC controller collects workflow scheduling decisions in a current slice window to determine a resource allocation strategy for a next slice window; at the beginning of the current slice window, resources of each base station are allocated according to the workflow scheduling decisions in a previous slice window; at the beginning of each scheduling time slot in the slice window, the controller transfers the collected tasks to different base stations for processing; and at the end of each slice window, the controller collects the workflow scheduling decisions in the window for use in the next resource allocation. Simulation results show that, compared with a benchmark method, the proposed scheme exhibits superiority in adaptability, task completion rate and control overhead.

The application provides a workflow scheduling method considering security risks and cost optimization in a cloud environment, which reduces the overall security risks and execution costs of the workflow under the given deadline constraint of a user. Firstly, the cloud computing resources are defined, the workflow structure is abstracted, and a security risk calculation method is proposed. Then, a reinforcement learning model oriented to security risks and cost optimization is constructed. Next, an action selection algorithm in the learning process is determined. Finally, iterative learning is carried out based on the Q learning algorithm idea, and finally the task scheduling scheme is determined, and the overall security risks and total costs of the workflow are obtained.

The present invention provides a workflow scheduling method, system, and device. The method includes: constructing a workflow directed acyclic graph (DAG); using a center-point-based DAG clustering method based on the workflow directed acyclic graph (DAG) to obtain a set of clusters; based on the set of clusters and a pre-allocation model, obtaining a CPU allocation plan for each cluster; and based on the workflow directed acyclic graph (DAG) and the CPU allocation plan for each cluster, obtaining a final workflow scheduling plan based on a virtual earliest end time scheduling model. The present invention uses a center-point-based DAG clustering method to obtain a set of clusters, and uses the set of clusters to guide scheduling, thereby obtaining a more optimal workflow scheduling plan, thereby fundamentally solving the communication congestion problem in existing scheduling for shared bus heterogeneous computing systems.

This invention discloses a method for predicting the load of stochastic workflows with missing data. The method includes: a workflow preprocessing stage, where the workflow is transformed into an embedded vector form and clustered based on historical data similarity; a workflow missing value completion and load prediction stage, where a kernel density missing data filling method based on common features is used to fill in missing data, and a workflow load analysis and prediction method is proposed based on the characteristics of the computing network-cloud-edge-device structure and historical information on workflow computation, storage, and communication; a cluster modeling and feature engineering stage; and a cluster load prediction stage, where this invention combines kernel methods and Kalman filtering techniques to predict missing values ​​and workflow load, optimizing the robustness and generalization of the prediction. This method has broad application value and promising prospects in the fields of load prediction and workflow scheduling.

The application discloses a kind of Yarn cluster workflow scheduling methods based on deep reinforcement learning, its characteristics are the method includes: workflow is modeled as directed acyclic graph, it is encoded as workflow state vector using graph neural convolution network, then workflow state vector and Yarn cluster queue resource state vector are jointly input into policy neural network, using proximal policy optimization algorithm to train it etc. steps.Compared with prior art, the application has the advantages of dynamically adjusting scheduling decisions according to the current environment state, improving cluster resource utilization while reducing the overall completion time of the workflow and the delay time of high-priority tasks, the model is simple and easy to train, can significantly improve user experience, and provides technical support for related fields such as Hadoop big data analysis.

The application discloses a cloud workflow scheduling method based on an improved snake optimizer, applies the snake optimization algorithm to a discrete optimization problem, realizes scheduling of multiple workflow tasks in a cloud environment, provides a new solution path for a workflow scheduling problem in the cloud environment, and simultaneously modifies parameters of different stages of the SO algorithm on the basis of analyzing the SO algorithm and characteristics of the workflow scheduling problem, so as to take into account exploration and development, speed up convergence of the algorithm, and improve calculation efficiency.

The invention relates to a video service workflow scheduling processing method and device, equipment and a storage medium. The method comprises the following steps: receiving a task workflow formed by connecting a plurality of functional units; based on the semantic information and the data processing type information of each function unit, function type division is carried out, and type labels are given; distributing the function units to the matched heterogeneous computing resource groups according to the labels; for each function unit, matching a plurality of candidate micro-services with function similarity meeting conditions according to semantic information and function type labels of the function unit to form a candidate micro-service set; based on the real-time resource state information, selecting an executed computing resource instance for the candidate micro-service, and generating a scheduling scheme of a task workflow; a dynamic adjustment operation is triggered in response to a workflow, microservice, or computing resource state change event. By adopting the method, the adaptation degree of each workflow fragment and the resource can be improved, so that the completion efficiency of the whole task workflow is improved.

The invention discloses an application system and method of a self-adaptive RAG frame in titanium alloy casting process optimization based on a large model, and belongs to the technical field of intelligent manufacturing and metal material processing. By constructing a query analysis module, a multi-source knowledge retrieval module, a physical constraint embedding generation module, a distributed verification module and an adaptive workflow scheduling module, a structured knowledge graph and an unstructured case library are integrated, and multi-scale semantic retrieval is realized by using ChromaDB + FAISS; field fine tuning is carried out based on a large language model, and a differentiable physical constraint loss function is embedded; finite element analysis is executed in parallel based on a Ray framework, and accelerated verification of thermal-mechanical coupling simulation is achieved; and the extensible workflow is constructed by using LangGraph. According to the method, through a closed-loop optimization mechanism of retrieval-generation-verification, the process optimization period is shortened by 37.6%, the parameter accuracy is improved to 92.4%, the residual stress deviation is controlled within 8.3 MPa, the argon consumption is reduced by 17.3%, and the technical problems of multi-physical field coupling, high nonlinearity, long optimization period and the like in the titanium alloy casting process are effectively solved.

The technology described herein is directed towards workload prioritization (ranking) in a cloud (e.g., private / hybrid) environment based on multi-criteria decision analysis using Mahalanobis distance-based variance-covariance matrix that takes into account the pair-wise correlation between criteria attributes from a dataset. The technology described herein calculates relative closeness values for alternatives (e.g., virtual machines), and can perform a final ranking of a group of virtual machines based on their associated relative closeness values. The workload prioritization process can be incorporated into a decision engine. The decision engine can be used in various use cases, including backup prioritization, shutdown prioritization during power failures, workflow scheduling prioritization, and the like.

This invention belongs to the field of cloud computing, specifically relating to a cost-based workflow scheduling method in an IaaS environment. The method includes: obtaining the execution time of each task in the workflow on different virtual machines, and the data communication time between different virtual machines; constructing a dynamic budget allocation model based on user-defined total cost constraints; merging the workflow into single parent-child task pairs to obtain a new directed acyclic graph (DAG); calculating the priority score of each task in the new DAG and sorting them to obtain a sorted list; calculating the cost constraint of each task sequentially according to the sorted list, and obtaining the currently allowed budget for scheduling the task based on the dynamic budget allocation model; calculating the earliest completion time of the task before and after copying the critical parent task on different virtual machines, and selecting the virtual machine with the smallest earliest completion time to place the task; this invention effectively reduces the workflow completion time.

The invention discloses a computing power network-oriented function, namely a service resource sensing overall planning method, and belongs to the fields of cloud computing, server-free computing and computing power networks. The invention provides an efficient resource awareness and unified scheduling scheme which is divided into three stages: a task execution time prediction stage, a workflow scheduling stage and a data-driven FaaS platform construction stage. In the task execution time prediction stage, the transmission and execution time of the task is predicted by utilizing the dynamic change characteristics of the task in the system. In the workflow scheduling stage, the predicted transmission and execution time of the task is utilized, attributes such as the structure and the data size of the workflow and the function category are comprehensively considered, and computing power nodes are reasonably distributed so as to reduce the execution time delay of the workflow. In the data-driven FaaS platform construction stage, a FaaS platform normal form capable of making decisions by depending on data streams is constructed, so that system time delay caused by a workflow processing flow in the platform is reduced. According to the method, the computing power resource utilization rate and the FaaS service response efficiency are remarkably improved, the cold start delay is reduced, unified perception and efficient overall planning of heterogeneous and cross-domain resources are achieved, reliable support is provided for high-performance and cross-domain server-free application, and the method has wide application value and development prospects.

The invention provides a distributed workflow task scheduling method and system, and relates to the technical field of computers, and the method comprises the steps: defining a scheduling task node and a dependency relationship in a dragging manner through a visual canvas, and storing the scheduling task node and the dependency relationship; the Worker nodes with the ZooKeeper center hop state being normal are screened, node weights are calculated based on dynamic load balancing, and tasks are distributed to the Worker node with the highest weight to be executed; when the Worker node has a fault, monitoring a ZooKeeper node deletion event, and identifying an IP (Internet Protocol) and a port of the fault Worker; marking an unterminated task state on the fault node as a state needing fault tolerance and redistributing; when the Master node fails, the surviving Master takes over the workflow instance of the failed Master based on a slot allocation mechanism; and updating the task reporting address in the Worker to the current Master, and re-distributing the uncompleted tasks. The usability, efficiency and stability of workflow scheduling are integrally improved, and diversified business requirements are met.

The invention belongs to the technical field of data processing. According to the large model workflow scheduling method and system, a Raft cluster is combined, and a leader node performs task division and generation and synchronization of a workflow record log according to a problem request and user permission information carried in the problem request; a workflow full life cycle state (a stage state, intermediate data and follower node binding relationship) is used as a log copying target, each state change needs to be submitted through consensus of most nodes, state machine views of all nodes are strictly consistent, the state divergence risk is thoroughly eliminated, execution nodes are screened based on user permission information, and log copying efficiency is improved. The data security is guaranteed, resource conflicts are avoided, and the processing efficiency and precision of the complex workflow are improved.

The application discloses a geological survey project whole-process management system based on a workflow engine, relates to the field of automatic control of geological and mineral exploration, and comprises a central management control platform, a central workflow scheduling center, a cross-agency collaboration gateway and a distributed business database which are internally arranged, a field operation sensing subsystem, a portable geological survey terminal which is integrated with a double-frequency multi-constellation GNSS receiver board card, an intelligent checking calculation base which comprises a geological semantic checking engine and a multi-source spatial data fusion module, a resource and monitoring feedback subsystem which comprises a laboratory resource scheduler, and asynchronous communication between the central management control platform and the intelligent checking calculation base through a message queue bus.The application realizes real-time strong constraints of a process on quality through an evaluation system, solves the problem of inconsistent field positioning accuracy by using a BursaWolf model, and realizes optimal allocation of resources in the whole life cycle of a geological survey project through an incentive type scheduling index.

The present application relates to a kind of edge cloud collaborative adaptive workflow scheduling method and system, belong to distributed computing and artificial intelligence technical field.First, in macroscopic candidate screening stage, through task clustering reduces problem granularity, and based on the probabilistic optimization strategy of weighted distance, balance is achieved between use and exploration;Then, in collaborative scheduling decision stage, by graph neural network, global network state diagram is constructed, deep space features of node and its neighborhood are extracted, form context-aware state representation, and reinforcement learning agent makes optimal collaborative decision in local execution, edge migration or cloud offloading etc. Multi-option accordingly;Finally, in local adaptive optimization stage, fine-grained optimization is executed after task issuing, and by online learning mechanism, scheduling frequency and multi-objective weight are dynamically adjusted, balance is realized between task deadline and resource utilization, to ensure the efficient and robust execution of node level.

This application discloses a workflow processing system and method. It relates to the fields of cloud computing, automated task scheduling, and container orchestration. The system includes: a server, used to interact with users through a graphical interface, create workflow template files, and send the workflow template files to a scheduler; a scheduler, used to receive workflow instances, extract a directed acyclic graph (DAG) from the workflow instances, and control clients to execute workflow instances according to the DAG; and a client, used to execute tasks of N task nodes through N containers, updating the status of each task node to the server. This application solves the problem of low resource utilization in workflow scheduling in cloud-native environments in related technologies.

The present invention discloses a pipeline-oriented network processing workflow scheduling method, device, and medium, wherein the method includes: generating a DAG and generating a global resource usage document; serializing the network processing workflow; training a network work scheduling model; and forming a network workflow scheduling strategy. The pipeline-oriented network processing workflow scheduling method proposed by the present invention can handle network processing workflows of variable length and network processing resources of variable scale, and utilizes a reinforcement learning model to simultaneously implement intelligent network processing resource planning and intelligent work scheduling for network processing workflows, thereby improving the overall utilization of network processing resources and reducing the waste of network processing resources. The present invention can be widely applied to fields such as reinforcement learning, parallel computing, and work scheduling.