104 results about "Workflow scheduling" patented technology

The invention discloses a cross-domain workflow scheduling system for dynamic resource arrangement, and the system comprises a resource abstraction subsystem which is used for collecting all resource platform information and network topology information, and constructing a resource directory and all adaptation layers; the task granularity division subsystem is used for receiving a target task, determining task characteristics of the target task, the cost of each task and an initial dependency graph, determining a target granularity according to the cost of each task, and optimizing the initial dependency graph to obtain a target dependency graph; the workflow arrangement subsystem is used for determining an initial execution plan according to the target dependency graph and the resource directory, determining a corresponding adaptation layer after each target execution platform receives target data, and submitting a corresponding subtask to each target execution platform; and the data preheating management subsystem is used for determining the transmission priority of the cross-platform sub-tasks and sending the target data to each target execution platform in combination with the network topology information, so that the effect of uniformly managing different platform resources is realized, and the stability of the cross-domain tasks is improved.

The invention relates to a calculation-storage stream joint scheduling optimization method and system based on DDQN and a heuristic strategy, and belongs to the technical field of distributed calculation and intelligent scheduling. The calculation-storage stream joint scheduling optimization method comprises the following steps: (1) defining a calculation stream and a storage stream; (2) workflow scheduling and execution: when the workflow is triggered, a scheduler evaluates the executable states of the tasks one by one according to the dependency relationship between the tasks and a scheduling strategy, and submits the executable tasks to a task executor; the task executor is responsible for executing the tasks on the working nodes and ensuring that the tasks are completed according to a preset sequence and priority; (3) collecting a DAG structure, queue information and an operation number; (4) task operation time and resource use prediction based on historical operation data; (5) scheduling optimization based on DDQN and a heuristic strategy; and (6) acquiring and storing scheduling information. The task execution efficiency and the resource utilization rate of the system are remarkably improved.

The invention discloses a hybrid heterogeneous cloud workflow scheduling method based on reinforcement learning, and belongs to the technical field of cloud computing. The method comprises the following steps: aiming at a cloud workflow scheduling problem, by taking minimization of completion time as a target and taking cost and resources as constraints, a three-dimensional collaborative constraint model is constructed, and the cost constraints comprise server-free function budget and virtual machine budget; integrating the hyper-heuristic framework into a reinforcement learning algorithm; an improved reinforcement learning algorithm is adopted to solve the cloud workflow scheduling problem, optimal execution resources are selected, and an optimal scheduling scheme is obtained; and performing real-time scheduling according to the optimal scheduling scheme, and introducing a deviation feedback mechanism to monitor an execution error in real time. According to the method, the workflow scheduling problem is decomposed into a closed-loop optimization process of state perception and action decision, dynamic environment perception, multi-target tradeoff and online strategy optimization are deeply fused, the global search function is achieved, local optimization can be achieved, the algorithm complexity is low, and robustness is high.

The invention discloses an AI agent construction system based on a vector knowledge base and large model workflow, and belongs to the technical field of process management. The problem that the whole process of workflow scheduling and quality control cannot be managed smoothly due to the fact that an existing AI system cannot assemble heterogeneous and unstructured domain knowledge and business rules dispersed in internal documents, databases, APIs and expert experience through a standardized and low-code platform in a rapid and high-quality mode is solved. Comprising a project initialization module, a knowledge base configuration module, a model service module, a workflow arrangement module, a test verification module and a deployment monitoring module. According to the method, data, knowledge, reasoning and quality control are communicated through a unified vector knowledge base and a low-code drag workflow, a dynamic hypergraph is driven by using computable information potential energy / cognitive entropy, automatic topology operation cutting circulation and shortcut establishment are performed before deadlock outbreak, and predictable hard constraint management of knowledge fusion, process arrangement and cost process is realized.

The invention relates to a side cloud collaborative adaptive workflow scheduling method and system, and belongs to the technical field of distributed computing and artificial intelligence. The method comprises the following steps of: firstly, in a macroscopic candidate screening stage, reducing problem granularity through task clustering, and obtaining balance between utilization and exploration based on a weighted distance probabilistic preferential strategy; then, in a collaborative scheduling decision-making stage, a global network state diagram is constructed through a graph neural network, deep spatial features of nodes and neighborhoods of the nodes are extracted, context-aware state representation is formed, and a reinforcement learning agent makes an optimal collaborative decision in multiple options such as local execution, edge migration or cloud unloading according to the state representation; and finally, in a local adaptive optimization stage, performing fine-grained optimization after the task is issued, dynamically adjusting a scheduling frequency and a multi-target weight through an online learning mechanism, realizing balance between a task deadline and a resource utilization rate, and ensuring efficient and robust execution of a node level.

The invention discloses a fault-tolerant workflow scheduling method and system supporting dependence and failure retry of multiple upstream nodes. According to the method, the scheduling configuration, the dependency relationship and the real-time state of each node are maintained through task metadata storage; after the scheduler triggers task execution, the state monitor updates an execution result; when the task fails, the retry coordinator switches the task into a retry state according to a pre-configuration strategy and calculates retry time; before a scheduler triggers a downstream task, the states of all upstream nodes are checked by relying on an analyzer, and if an unsuccessful node exists, the downstream task is marked as waiting and placed in a waiting queue; and when the upstream node succeeds, traversing the waiting queue by depending on the resolver, carrying out secondary verification on the downstream task depending on the node, and only when all the upstream nodes succeed, recovering the upstream nodes to be in a to-be-scheduled state. According to the method, closed-loop linkage of retry and waiting is realized, dependency consistency under a heterogeneous scheduling strategy is supported, and the reliability, resource utilization rate and observability of the workflow are remarkably improved.

The invention relates to the technical field of ground disaster prevention and control report generation, in particular to an intelligent ground disaster prevention and control report generation method and system based on indoor and outdoor work integration. Retrieving geological information of the associated area, and determining the type of a hidden danger point; matching a target template; generating a target template directory structure; calling a workflow agent module according to a directory structure sequence to generate a first draft text; inserting chart data into the first draft text to obtain a complete report; and performing compliance examination and optimization on the complete report, and outputting a final report. The generation system comprises a data acquisition module, an intelligent retrieval engine, a template management unit, a workflow scheduling center and an examination optimization module. The data acquisition module, the intelligent retrieval engine, the template management unit, the workflow scheduling center and the review optimization module are connected in sequence and realize data transmission; synchronous real-time updating of indoor and outdoor data is met, data dispersion and information omission are avoided, and the generation efficiency and accuracy of a ground disaster prevention and control report are improved.

The invention discloses a zero-code heterogeneous data integration method. The method comprises the following steps: S1, configuring a multi-source data connector; s2, processing differences of different data formats and structures by adopting an adapter mode; s3, performing data cleaning; s4, performing data conversion; s5, loading the converted data into a target database or a data warehouse, and adopting a partition loading and incremental updating strategy; s6, providing a graphical user interface, and constructing a data integration workflow by dragging the component and connecting the data flow; s7, providing a visual workflow management tool, and supporting workflow version control, collaborative editing and parallel processing; s8, realizing an automatic workflow scheduling system, and supporting timed tasks and event driving; s9, performing data consistency check and integrity verification; and S10, a modular and extensible system architecture design is adopted. A zero code integration method is adopted, efficient integration of heterogeneous data sources is achieved, and the method has the advantages of being high in usability, high in processing efficiency and capable of guaranteeing data quality.

The embodiment of the invention provides a workflow scheduling method and device, equipment, a storage medium and a program product, and is applied to the technical field of cloud computing. The method comprises the steps of obtaining target attribute data corresponding to a target workflow; according to the target attribute data, constructing a multi-target optimization model corresponding to the target workflow; the multi-target optimization model comprises a target hybrid optimization algorithm; determining a target solution set corresponding to the target workflow according to the target attribute data and a multi-target optimization model; and decoding the target solution set to obtain a target workflow scheduling mode corresponding to the target workflow. Therefore, the electronic equipment calculates and optimizes the workflow scheduling based on the multi-objective optimization model, so that the diversity of the workflow scheduling process and the scheduling result can be improved, and meanwhile, the optimization effect of the workflow scheduling can also be improved.

The invention provides a spark task running state monitoring method and device and a storage medium. The method comprises the steps of obtaining workflow data running in a spark cluster; a data monitor in a kubernetes cluster is adopted to monitor the workflow data running in the spark cluster, a running state corresponding to the workflow data is determined according to the workflow data, and the running state is one of running overtime, running failure and queuing waiting; and when the running state of the workflow data is running timeout, terminating the spark task corresponding to the workflow data, and releasing machine resources occupied by the spark task. According to the method, the corresponding processing flow is executed according to the monitored resource change based on the watch mechanism of the k8s container platform, so that the monitoring efficiency of the spark task scheduled by the workflow scheduling system is improved, and the resource occupation is reduced.

The invention discloses a CAE (Computer Aided Engineering) workflow control subsystem and method based on cloud computing, and belongs to the technical field of crossing of computer simulation and cloud computing, and the subsystem comprises a workflow configuration analysis module which is used for converting a CAE workflow configuration file possibly containing cyclic dependence into a directed acyclic graph state machine through strong connectivity component analysis; the workflow state machine operation module is used for dynamically deriving task groups which can be executed in parallel in the next stage in the state machine according to the current state; and the workflow scheduling module is used for submitting the tasks to a cloud solving platform in parallel and realizing fault recovery through state persistence. Through deep fusion of a graph theory method and a cloud computing architecture, automatic scheduling, efficient parallel execution and reliable fault tolerance of the CAE workflow are realized, and the computing resource utilization rate and the system reliability are remarkably improved.

A workflow scheduling fault-tolerant system based on a Watcher mechanism of a ZooKeeper comprises the ZooKeeper, the ZooKeeper is connected with at least one workflow executor and at least one workflow scheduler, and the ZooKeeper is further used for receiving a live event. The invention further provides a workflow scheduling fault-tolerant method based on the Watcher mechanism of the ZooKeeper and a workflow scheduling fault-tolerant system based on the Watcher mechanism of the ZooKeeper, and the workflow scheduling fault-tolerant method based on the Watcher mechanism of the ZooKeeper is characterized by comprising the following steps that the workflow scheduling fault-tolerant method comprises the steps that the workflow scheduling fault-tolerant method comprises the steps that the workflow scheduling The method comprises the following steps: S1, constructing a mapping relation between a workflow model and a ZooKeeper node; s2, task execution and state monitoring; s3, fault detection and processing; s4, fault recovery and task re-triggering; and S5, dynamic task adjustment and load balancing.

A dynamic competition perception workflow scheduling method for a heterogeneous computing system comprises the following steps: constructing a scheduling problem model based on a scheduling object set and bus resources of an obtained system; constructing a dynamic competition perception simulator according to the scheduling problem model, wherein the method comprises the following steps of: decomposing a time-varying behavior of cross-domain bus data transmission into a time-invariant meta-process; each meta-process is mapped into a timeout event carrying a specific timestamp; the completion event indicates that all the timeout events to which the same scheduling object belongs are completed, so that the end of the task or cross-domain bus data transmission is marked; according to an event-driven mechanism, a completion event is arranged on the premise that the preorder constraint of a scheduling object is met, and total completion time estimation is generated while integration of multiple scheduling strategies is supported; according to the total completion time estimation, generating an approximate optimal scheduling solution by using a hybrid earliest completion time genetic algorithm; and optimizing the approximate optimal scheduling solution by adopting a quota weighted ant colony algorithm to generate a final scheduling solution.

The invention discloses a task scheduling method and system based on an artificial bee colony and a near-end strategy optimization algorithm. The method comprises the steps of obtaining workflow task data in an industrial production process; performing global search processing on the workflow task data through an artificial bee colony algorithm to generate an initial task scheduling scheme; performing local optimization processing on the initial task scheduling scheme through a near-end strategy optimization algorithm, and outputting a target task scheduling scheme; and performing task scheduling processing on the workflow task data according to the target task scheduling scheme to obtain a task scheduling result. The problem of insufficient workflow scheduling flexibility in a dynamic environment can be solved, the optimal task scheduling scheme can be quickly searched, the optimization of the resource utilization rate is guaranteed, the task scheduling efficiency in the industrial production process is improved, and the method can be widely applied to the technical field of industrial intelligent production.

The invention relates to the field of cloud computing and deep reinforcement learning, and discloses a large-scale scientific computing workflow scheduling method based on deep reinforcement learning. According to the method, the graph attention network is introduced to extract workflow structure features, and a deep reinforcement learning sampling strategy based on a genetic algorithm is designed to generate an initial sample trajectory for a deep reinforcement learning agent, so that the training process is accelerated. Experiments show that a faster training speed can be obtained in a simulated cloud environment, the workflow is executed on the cloud computing platform according to the generated large-scale scientific computing workflow scheduling strategy, and better completion time, energy consumption and reliability can be reflected.

The present disclosure relates to a workflow scheduling method and device, and a computer readable storage medium. The workflow scheduling method comprises: defining a workflow according to user input, wherein the workflow is a process arranged for a plurality of tasks, the plurality of tasks are tasks included in a model training whole process, and the workflow comprises a plurality of tasks in data labeling and model evaluation in the model training whole process; and scheduling execution of the workflow, wherein for different tasks, pre-configured program data in shared storage is run according to a configured type to execute the tasks. In the labeling and effect evaluation process in the model training process, the present disclosure can automatically link the preceding and subsequent tasks, thereby reducing the communication cost of algorithm personnel and labeling personnel.

The invention discloses a CHUNJUN data synchronization workflow scheduling method and system based on a directed acyclic graph, and the method comprises the following steps: (1) packaging a CHUNJUN data synchronization task into standardized components, and exposing an input interface, an output interface and a configuration interface of each component; (2) receiving user operation through a visual interface to establish a component connection relationship, and executing connection normalization check; (3) constructing a directed acyclic graph (DAG) based on the connection relationship, and generating a task execution sequence; (4) triggering a CHUNJUN data synchronization task according to the execution sequence; according to the scheme, the construction time of a single task is compressed to be within 2 hours from 5-6 hours of a traditional command line mode, and the configuration error rate of a novice user is reduced by 60% through visual operation; and meanwhile, a DAG scheduling mechanism guarantees the reliability of multi-task sequential execution, the data synchronization failure rate is reduced by 60%, and the data intermediate station construction efficiency is remarkably improved.

The invention provides a workflow scheduling method and device, an execution engine, equipment and a program product, and the method comprises the steps: determining a current layer node based on a topological relation between nodes in a target workflow and an execution state and / or an execution result of each node, the current layer node comprises a starting node of the target workflow or a rear node of a previous layer node in each branch path of the target workflow, and when the current layer node is not empty, executing each current layer node based on the target attribute information of each current layer node. According to the workflow scheduling method, the execution efficiency of the target workflow can be effectively improved on the basis of ensuring correct execution of the target workflow.

The invention provides a neural image data preprocessing system and a neural image data preprocessing method, and the neural image data preprocessing system comprises a file input module, a workflow module, a workflow scheduling module and a file output module. The file input module is used for carrying out format recognition and conversion on original nerve image data to obtain nerve image data in a standard format; the workflow module is used for generating a corresponding to-be-processed task according to the data information of the neural image data in the standard format; the workflow scheduling module is used for distributing computational nodes for tasks to be processed and completing preprocessing of the neural image data in the standard format; and the file output module is used for performing standardized naming and storage on the preprocessing result corresponding to the original nerve image data. The invention provides a neural image data preprocessing system and a neural image data preprocessing method. The processing efficiency of neural image data is improved.

The application discloses a space-air-ground integrated vehicle networking slice-based cooperative task offloading method, and the design steps comprise the following steps: firstly, a service-oriented RAN slice framework is designed; then, a maximum long-term task completion number problem is decoupled into a slice window length division problem, a resource allocation problem and a cooperative workflow scheduling problem; a MEC controller collects workflow scheduling decisions in a current slice window to determine a resource allocation strategy for a next slice window; at the beginning of the current slice window, resources of each base station are allocated according to the workflow scheduling decisions in a previous slice window; at the beginning of each scheduling time slot in the slice window, the controller transfers the collected tasks to different base stations for processing; and at the end of each slice window, the controller collects the workflow scheduling decisions in the window for use in the next resource allocation. Simulation results show that, compared with a benchmark method, the proposed scheme exhibits superiority in adaptability, task completion rate and control overhead.

The application provides a workflow scheduling method considering security risks and cost optimization in a cloud environment, which reduces the overall security risks and execution costs of the workflow under the given deadline constraint of a user. Firstly, the cloud computing resources are defined, the workflow structure is abstracted, and a security risk calculation method is proposed. Then, a reinforcement learning model oriented to security risks and cost optimization is constructed. Next, an action selection algorithm in the learning process is determined. Finally, iterative learning is carried out based on the Q learning algorithm idea, and finally the task scheduling scheme is determined, and the overall security risks and total costs of the workflow are obtained.

The present invention provides a workflow scheduling method, system, and device. The method includes: constructing a workflow directed acyclic graph (DAG); using a center-point-based DAG clustering method based on the workflow directed acyclic graph (DAG) to obtain a set of clusters; based on the set of clusters and a pre-allocation model, obtaining a CPU allocation plan for each cluster; and based on the workflow directed acyclic graph (DAG) and the CPU allocation plan for each cluster, obtaining a final workflow scheduling plan based on a virtual earliest end time scheduling model. The present invention uses a center-point-based DAG clustering method to obtain a set of clusters, and uses the set of clusters to guide scheduling, thereby obtaining a more optimal workflow scheduling plan, thereby fundamentally solving the communication congestion problem in existing scheduling for shared bus heterogeneous computing systems.

The invention discloses a workflow scheduling method and device, electronic equipment and a storage medium, which are used for solving the technical problem that analysis and scheduling of a workflow graph cannot be isolated due to the fact that a large number of cards are logically coupled in the same service. The method comprises the steps that a management background receives a workflow graph triggered by a first user at a management page end; the management background sends the workflow graph to a scheduling analysis center; the scheduling analysis center analyzes the workflow graph to generate a scheduling topological graph, and returns the scheduling topological graph to the management background; the management background obtains the current state of each card in the scheduling topological graph, and sends the current state to the card application; the card application receives a workflow API initiation request sent by a second user at a user side, and sends a target scheduling topological graph to a scheduling analysis center; and the scheduling analysis center sequentially schedules the target cards in the target scheduling topological graph to obtain a card execution result, and returns the card execution result to the second user.

This invention discloses a method for predicting the load of stochastic workflows with missing data. The method includes: a workflow preprocessing stage, where the workflow is transformed into an embedded vector form and clustered based on historical data similarity; a workflow missing value completion and load prediction stage, where a kernel density missing data filling method based on common features is used to fill in missing data, and a workflow load analysis and prediction method is proposed based on the characteristics of the computing network-cloud-edge-device structure and historical information on workflow computation, storage, and communication; a cluster modeling and feature engineering stage; and a cluster load prediction stage, where this invention combines kernel methods and Kalman filtering techniques to predict missing values ​​and workflow load, optimizing the robustness and generalization of the prediction. This method has broad application value and promising prospects in the fields of load prediction and workflow scheduling.

The application discloses a kind of Yarn cluster workflow scheduling methods based on deep reinforcement learning, its characteristics are the method includes: workflow is modeled as directed acyclic graph, it is encoded as workflow state vector using graph neural convolution network, then workflow state vector and Yarn cluster queue resource state vector are jointly input into policy neural network, using proximal policy optimization algorithm to train it etc. steps.Compared with prior art, the application has the advantages of dynamically adjusting scheduling decisions according to the current environment state, improving cluster resource utilization while reducing the overall completion time of the workflow and the delay time of high-priority tasks, the model is simple and easy to train, can significantly improve user experience, and provides technical support for related fields such as Hadoop big data analysis.

The present invention discloses a workflow scheduling optimization method based on reliability constraints, comprising the following steps: Step S1: Modeling the application programs executed by mobile devices as a workflow; Step S2: Processing the workflow and generating a workflow scheduling scheme that meets the constraints; Step S3: Decoding the scheduling scheme generated in Step S2 and scheduling tasks to corresponding server nodes for execution; wherein, in Step S1, a workflow model is constructed for the workflow structure to determine the execution order of subtasks; and a reliability model for workflow execution is constructed according to the server failure rate. The present invention improves the success rate of workflow task offloading by satisfying reliability constraints, and at the same time defines the system cost in the scheduling problem of workflow applications as the weighted sum of time and energy, and can find the optimal scheme for workflow scheduling while balancing task processing delay and energy consumption.

The invention provides an intelligent workflow scheduling system for a data intensive application. The total time overhead of task execution is calculated through a time estimation module arranged on a computing power node in a computing power network, the total energy consumption is calculated through an energy prediction module, and the utility of all functions of intelligent workflow scheduling is calculated through a UE utility analysis module. The priorities of the tasks are determined through a priority strategy module so as to adjust the execution sequence of the tasks, and an intelligent workflow used for scheduling data intensive applications is formed through an application scheduling module; compared with the prior art, the method has the advantages that the task time, energy consumption and priority can be accurately predicted and optimized, the resource utilization rate is increased, delay and energy consumption are reduced, and time consumption, energy consumption and cost in data intensive application are optimized by applying intelligent workflow scheduling in a computing network brain layer, so that the overall efficiency of a computing power network is improved.

The application discloses a cloud workflow scheduling method based on an improved snake optimizer, applies the snake optimization algorithm to a discrete optimization problem, realizes scheduling of multiple workflow tasks in a cloud environment, provides a new solution path for a workflow scheduling problem in the cloud environment, and simultaneously modifies parameters of different stages of the SO algorithm on the basis of analyzing the SO algorithm and characteristics of the workflow scheduling problem, so as to take into account exploration and development, speed up convergence of the algorithm, and improve calculation efficiency.

The invention relates to a video service workflow scheduling processing method and device, equipment and a storage medium. The method comprises the following steps: receiving a task workflow formed by connecting a plurality of functional units; based on the semantic information and the data processing type information of each function unit, function type division is carried out, and type labels are given; distributing the function units to the matched heterogeneous computing resource groups according to the labels; for each function unit, matching a plurality of candidate micro-services with function similarity meeting conditions according to semantic information and function type labels of the function unit to form a candidate micro-service set; based on the real-time resource state information, selecting an executed computing resource instance for the candidate micro-service, and generating a scheduling scheme of a task workflow; a dynamic adjustment operation is triggered in response to a workflow, microservice, or computing resource state change event. By adopting the method, the adaptation degree of each workflow fragment and the resource can be improved, so that the completion efficiency of the whole task workflow is improved.

The invention discloses an application system and method of a self-adaptive RAG frame in titanium alloy casting process optimization based on a large model, and belongs to the technical field of intelligent manufacturing and metal material processing. By constructing a query analysis module, a multi-source knowledge retrieval module, a physical constraint embedding generation module, a distributed verification module and an adaptive workflow scheduling module, a structured knowledge graph and an unstructured case library are integrated, and multi-scale semantic retrieval is realized by using ChromaDB + FAISS; field fine tuning is carried out based on a large language model, and a differentiable physical constraint loss function is embedded; finite element analysis is executed in parallel based on a Ray framework, and accelerated verification of thermal-mechanical coupling simulation is achieved; and the extensible workflow is constructed by using LangGraph. According to the method, through a closed-loop optimization mechanism of retrieval-generation-verification, the process optimization period is shortened by 37.6%, the parameter accuracy is improved to 92.4%, the residual stress deviation is controlled within 8.3 MPa, the argon consumption is reduced by 17.3%, and the technical problems of multi-physical field coupling, high nonlinearity, long optimization period and the like in the titanium alloy casting process are effectively solved.