35 results about "Job scheduler" patented technology

An autonomous job scheduling system for distributed ledger technology is disclosed, leveraging photonic quantum computing and generative artificial intelligence (AI). The system collects transactional and operational data from various nodes within a distributed ledger network, focusing on transaction types, sizes, and priorities. This data is filtered for urgency and resource intensity, analyzed against real-time network conditions and business rules to ascertain job execution parameters. A photonic quantum computing system processes these parameters to optimize job scheduling strategies, including quantum annealing and simulations. The AI engine generates dynamic smart contracts deployed to distributed ledgers by a job orchestration engine. The system monitors job execution, gathering performance data and providing feedback for real-time adjustments. The job orchestration engine updates job scheduling in response to these adjustments, enhancing the efficiency and responsiveness of the blockchain network.

This application relates to the technical field of natural language processing, and provides a distributed pipeline-parallel LLM fine-tuning method for heterogeneous GPUs. A plurality of LoRA models are fine-tuned simultaneously based on a multi-job fine-tuning system; each LoRA model is partitioned into a plurality of parts distributed on a corresponding number of GPUs, and the GPUs are sorted. A job configuration module generates a plurality of jobs according to a user request, and divides each job into a plurality of training batches; a dynamic job scheduler generates a scheduling scheme based on a training batch sequence of each job and a dynamic scheduling strategy; and the scheduling scheme is sent to a multi-job training module on each corresponding GPU according to a positive sequence of the GPUs, to train all the LoRA models.

Methods, systems, and devices for data management are described. A data management system (DMS) may provide backup and recovery services for customer computing systems or databases, which may involve scheduling jobs to perform the backup and recovery services. Each job may define a set of semaphores which may be acquired prior to execution of the job. The semaphores may be representative of an availability of computing resources associated with the DMS. Jobs may be grouped into job groups based on the semaphores associated with each job. Each job group may be scheduled independently by separate dispatchers or job schedulers. Within a job group, the associated dispatcher(s) may schedule jobs if the semaphore(s) associated with the job group are available. Within a job group, the associated dispatchers may refrain from dispatching additional jobs if the semaphore(s) for the job group are full until the semaphore(s) are available.

The present invention proposes a potential risk personnel early warning system based on XXL scheduled scheduling, which includes: a custom early warning module, a rule verification module, a scheduling module and a matching module, wherein the custom early warning module is used to configure an early warning template containing data source rules, scoring rules and proportions; the rule verification module is used to determine whether the template configured in the custom early warning module complies with the configuration rules. If it does not comply, the custom early warning module is re-entered to modify the early warning template; if it complies, the early warning template is submitted and an early warning task is initiated; the scheduling module receives the early warning task, schedules the execution of the early warning task through the XXL Job scheduler, presents the execution result on the front-end page, and directly optimizes the early warning template on the front-end page to obtain the optimized early warning model; the matching module is used to match the input data one by one with the early warning model, calculate the early warning score and obtain the early warning result.

The invention discloses a distributed machine learning training system based on network flow rate control. The distributed machine learning training system comprises a flow information collector module and a job scheduler module, a traffic information collector module collects the number of bytes in a job stream communication mode on a switch, records the timestamp of arrival of a data packet, and calculates the remaining iteration completion time of the job stream; and the job scheduler module determines the priority of the job according to the residual iteration completion time of the job flow, so that the switch preferentially sends the job flow with the shortest residual iteration completion time. According to the distributed machine learning training system and method based on network flow rate control, the utilization rate of a network link can be remarkably improved; by optimizing the communication stage of the job flow, training tasks, such as DNN training tasks, can share network resources more efficiently, thereby reducing network congestion and idle time.

The embodiment of the invention discloses a quick starting method, system and equipment of an operating system and a storage medium, according to the embodiment of the invention, a distribution scheduling communication mechanism of jobs is realized by deploying a job scheduler and a job executor in BootLoader, and the jobs are executed in an asynchronous task distribution mode, so that the BootLoader is optimized. According to the embodiment of the invention, the operating system can be quickly started in an environment needing quick response and high efficiency, the BootLoader does not need to be replaced, the adjustment of an overall software scheme of the chip is not involved, the dependence on a chip manufacturer is small, and the feasibility of chip development is greatly improved; and meanwhile, depth function tailoring customization is not involved, and universality is achieved.

A system and method may manage a computer by executing, by a job scheduler on the computer, a framework script written in a scripting language which discovers installed sub-task scripts and executes discovered sub-task scripts. Sub-task scripts may be stored in directories corresponding to accounts in which a sub-task script is to be run. Sub-task scripts may perform maintenance or monitoring tasks. A system and method may assign resource access or permissions for a set of computers, by, for each computer of a set of computers, determining the organizational unit and sub-organizational unit of the computer; and for each computer of the set of computers assigning the computer to a security group, based on the organizational unit and sub-organizational unit of the computer. For each computer, the security group the computer is assigned to may be associated with permissions enforced for each computer in the group.

Computer-implemented methods, systems and computer program products for job scheduling in a computing system are provided. The computing resources of the computing system is allocated to one or more users of the computing system using a predefined policy. The method comprises receiving, by a job scheduler associated with the computing system one or more resource utilization scores of the one or more users, each of the one or more resource utilization scores indicating resource utilization efficiency over time at a job level for the one or more users. The method comprises receiving, by the job scheduler, one or more jobs submitted by the one or more users. The method further comprises scheduling, by the job scheduler, the received one or more jobs based on the predefined policy and further on the received one or more resource utilization scores of the one or more users.

This invention discloses an effective reinforcement learning and graph neural network fusion method for solving flexible job scheduling, comprising the following steps: formalizing the Flexible Job Scheduler (FJSP) into a Multi-Level Product (MDP) by setting states, actions, state transitions, and rewards, while describing the scheduling process as dynamically allocating ready operations to compatible and idle machines; during the scheduling process, the scheduling state is first transformed into a heterogeneous graph structure; a heterogeneous graph attention network is used for two-stage embedding to extract feature embeddings of jobs and machines from the heterogeneous graph; a decision network uses the feature embeddings to generate an action probability distribution, from which scheduling actions are sampled; and a deep reinforcement learning model is trained using a proximal policy optimization algorithm. Compared with traditional PDR and SOTA DRL methods, the algorithm achieves significantly high-quality solutions on two synthetic datasets with different distributions, improving performance and generalization ability. It also exhibits good generalization ability because the model is trained on a small number of instances on a large number of or out-of-distribution instances.

The invention discloses a job scheduling and resource allocation system in a high-performance computing cluster, and relates to the technical field of high-performance computing. The system comprises a job scheduler, a resource manager, a newly-added job DAG (directed acyclic graph) analyzer, a cluster data distribution monitor, a cache preheating priority calculator, a collaborative decision engine, a distributed cache preheating actuator and a performance optimization module, and the job DAG analyzer constructs a job dependence directed acyclic graph and analyzes a critical path; the cache preheating priority calculator adopts a dynamic priority drift algorithm to determine a data preheating priority, the collaborative decision engine coordinates scheduling and preheating resource allocation, the performance optimization module dynamically adjusts system parameters, the system realizes deep collaboration of job scheduling and cache preheating, the job access delay is reduced, and the system performance is improved. The method improves the cluster computing efficiency and the resource utilization rate, and is suitable for processing a high-performance computing cluster of complex operation workflow.

The present invention provides a pipeline job scheduling method and system. The pipeline job scheduling method of the present invention includes obtaining n jobs to be scheduled, combining the obtained n jobs to obtain all job combinations, determining the job combinations that meet the preset requirements among all the obtained job combinations, sorting the job combinations that meet the preset requirements according to the preset priority principle, and outputting the sorting result. The pipeline job scheduling method of the present invention can obtain the priority sorting of different job combinations according to the input job information, so as to facilitate the determination of the optimal job combination. It can achieve fast job scheduling, improve the job scheduling efficiency, and reduce the labor cost of job schedulers, thus having good practicability.

A method for monitoring a job scheduler may be performed on a computing device including one or more processors and a memory storing one or more programs executed by the one or more processors. The method may include checking whether it is necessary to determine whether to expand a resource of a computing environment to be scheduled when a job is performed by scheduling by a job scheduler; calculating a value of a score function for a scheduling policy currently executing and a maximum value of a predetermined score function when it is necessary to determine whether to expand a resource of the computing environment to be scheduled; and determining to expand a resource of the computing environment to be scheduled based on the value of the score function and the maximum value of the predetermined score function.

Aspects of the disclosure include a dynamic cloud workload reallocation based on an active ransomware attack. An example method includes receiving a first message that a computing instance is potentially infected by ransomware. The method further includes receiving a security state-based metric related to the computing instance based at least in part on the first message. The method further includes comparing the security state-based metric to a threshold metric. The method further incudes determining a likelihood of a ransomware attack based at least in part on the comparison. The method further includes transmitting second message to a job scheduler to reschedule workloads directed toward the computing instance based at least in part on the determination.

Disclosed herein are a system, method, and computer program product embodiments for elastic job pulling. For example, a first indication of a number of jobs at the job scheduler stage that are yet to be executed is obtained. This indication may be obtained from a job scheduler. A second indication of a total number of available job executors communicatively coupled to the job scheduler may be obtained from the job scheduler. A pull request for a job may be issued to the job scheduler based on at least one of the first indication, the second indication, or a computing resource status of a job executor. The job executor may obtain and execute the job based on the pull request. A status indication of the job may be transmitted, where the status indication indicates whether the job executed successfully.

A job processing method based on a cloud service includes: receiving a job request and obtaining an identification string of the job according to the job request by a job scheduler, sending the identification string to a memory cache and generating a worker by the job scheduler, obtains the identification string from the memory cache and obtaining data from a database according to the identification string by the worker, and executing the job based on the data and sending a result of the job to the database by the worker.

The invention relates to a method for realizing a thread pool for executing a decomposable task, which comprises the following steps of: S1, defining attributes: defining the attributes required for creating the thread pool, including Task, Job, MAXPOOLSIZE, MAXTaskSIZE, BLOCKTIMEOUT, TaskQueue, JobQueue, RunningTaskList and LastTaskIndex, and defining the attributes required for creating the thread pool, including Task, Job, MAXPOOLSIZE, MAXTaskSIZE, BLOCKTIMEOUT, TaskQueue, JobQueue, RunningTaskList and LastTaskIndex; s2, initialization: S2.1, creating the thread pool according to each attribute, completing initialization of each attribute, and setting a task scheduler, a Task scheduler and a Job scheduler; s2.2, a plurality of threads are created in the thread pool, the number of the threads is MAXPOOLSIZE, and all the threads are put into a thread array and started; s3, updating the array: when a new Task is submitted every time, adding the new Task into the TaskQueue; s4, executing the thread: obtaining a sub-task Job through a task scheduler, a Task scheduler and a Job scheduler, and executing the sub-task Job; and S5, loop execution: after the execution of the sub-task Job is finished, the loop should be continued and repeated, so that the S3 is returned.

Embodiments of the present invention disclose a method, system, device, and storage medium for quickly starting an operating system. In the embodiments of the present invention, a job scheduler and a job executor are deployed in the BootLoader to implement a job distribution and scheduling communication mechanism, and the BootLoader is optimized by using the method of asynchronously distributing tasks to execute jobs. In an environment that requires quick response and high efficiency, the embodiments of the present invention can quickly start the operating system without replacing the BootLoader, without involving adjustments to the overall chip software solution, with less dependence on chip manufacturers, greatly improving the feasibility of chip development; at the same time, it does not involve deep function trimming and customization and has generality.

The invention discloses a log processing method, device, platform and system, relates to the technical field of computers, and can improve the accuracy and efficiency of a job scheduler for matching job resources for a job request. The method comprises the steps that a compressed package of a log data package and a compressed package of a snapshot package of the log data package are obtained, the log data package comprises at least one piece of log information, one piece of log information comprises parameters and parameter values, and the parameters included in different pieces of log information are the same; then, the compressed packet of the log data packet and the compressed packet of the snapshot packet of the log data packet are sent to remote storage equipment; wherein the package name of the compressed package of the log data package comprises a first key value, the package name of the compressed package of the snapshot package comprises a second key value, and the first key value and the second key value are key values corresponding to the dimension reduction key of the log data package.

Aspects of the disclosure include a dynamic cloud workload reallocation based on an active ransomware attack. An example method includes receiving a first message that a computing instance is potentially infected by ransomware. The method further includes receiving a security state-based metric related to the computing instance based at least in part on the first message. The method further includes comparing the security state-based metric to a threshold metric. The method further incudes determining a likelihood of a ransomware attack based at least in part on the comparison. The method further includes transmitting second message to a job scheduler to reschedule workloads directed toward the computing instance based at least in part on the determination.

The invention relates to a distributed training system and method based on mercuric chloride NPU, and the method comprises the steps: a batch job scheduler is configured to receive a batch job configuration list, and creates and manages a head node Pod and a plurality of work nodes Pod in batches on nodes with idle mercuric chloride NPU in a cluster managed by a container arrangement platform, the head node Pod and the working node Pod are automatically networked based on a unified distributed computing framework to form a distributed computing cluster special for tasks; the main control service module is further configured to monitor the networking state of the distributed computing cluster, and submit a stand-alone training starting command to the head node Pod after confirming that networking is completed; in addition, the head node Pod is further configured to take the received starting command as a distributed task, and the distributed task is scheduled to each working node Pod for parallel execution through a unified distributed computing framework, so that the use threshold is greatly reduced, and the development efficiency and the system reliability are improved.

The invention discloses a data output timeliness measurement method and device for a big data platform and a storage medium, and the method comprises the steps: obtaining the actual completion timeliness of each data processing operation in the big data platform and the corresponding reference timeliness, and calculating the individual timeliness index of each data processing operation; acquiring preset attribute information and a data dependency relationship of each data processing job; determining a dynamic factor according to preset attribute information and the data dependency relationship, and determining an aggregation weight of the data processing job in each evaluation dimension based on the dynamic factor; calculating a dimension timeliness index through the individual timeliness index and the aggregation weight, and generating a comprehensive timeliness index of the big data platform based on the dimension timeliness index; and inputting the comprehensive timeliness index into a job scheduler of the big data platform to monitor the scheduling state of the big data platform. According to the method and the device, the technical effects of prospective scheduling and resource allocation and global aging landslide caused by local delay are realized by quantifying the aging risk conduction dynamic factors of the data dependency relationship between the operations.

A multi-actuator storage device includes a first actuator supporting a first read element coupled to a shared read channel, a second actuator supporting a second read element coupled to the shared read channel, and a job scheduler that employs a lowest-access time selection methodology when selecting each next job to schedule on the first actuator. When the lowest-access-time methodology results in the selection of a read job for execution by the first actuator, the job scheduler identifies a pending job for execution by the second actuator that can be executed concurrent to the first read job without concurrently accessing the read channel and schedules the pending job on the second actuator concurrent to the first read job.

The apparatus includes a section extraction module configured to receive an automation scenario including a plurality of sections, and to extract one or more sections of a plurality of job types, from among the plurality of sections, the one or more sections being executable in parallel, a section optimization module configured to generate a parallel execution section list by sorting the one or more sections based on priorities predetermined for the plurality of job types and a job scheduler configured to generate an idle bot list including an idle bot having an idle state between an execution start time and an execution end time of the automation scenario, and to match the idle bot included in the idle bot list to each of the one or more sections included in the parallel execution section list with reference to the at least one expected execution time period and the priorities.

A workload automation (WLA) system and method operates a job scheduler to automate jobs, that each specify one or more tasks to be performed by corresponding software programs, within and across computer systems. The job scheduler enables secure transfer of a job that includes sensitive information from a first instance of the job scheduler to a second instance by decrypting each encrypted item of sensitive information for transfer to the second instance and encrypting each such item with a transfer key which is used by the second instance to decrypt the transferred sensitive items and to then re-encrypt such sensitive items with a different key for storage and usage by the second instance.

Aspects of the present disclosure include dynamic cloud workload reallocation based on active ransomware attacks. An exemplary method includes receiving a first message that a computing instance may be infected with ransomware. The method further includes receiving an indicator based on a security state associated with the computing instance, at least partially based on the first message. The method further includes comparing the security state-based indicator to a threshold indicator. The method further includes determining a likelihood of a ransomware attack, at least partially based on this comparison. The method further includes sending a second message to a job scheduler and rescheduling the workload directed to the computing instance, at least partially based on this determination.

Methods and systems are described herein for facilitating distributed job execution without a central job scheduler. The system may cause a container to, prior to executing job execution code for a job associated with a job data record, update a record instance of the job data record to indicate an updated status for the job and attempt to update the job data record at a database based on the record instance of the job data record. If the container successfully updates the job data record, the container may execute the job execution code for the job. If the container fails to update the job data record, the container may refrain from executing the job execution code for the job. The system may then update a first job data record associated with a first job at the database based on execution of the first job by a first container.

The present invention discloses a method for quickly locating an abnormal process based on program performance counting, comprising the following steps: S1, a user submits an application job; S2, a job scheduler initializes a preset hardware performance counter on a processor corresponding to each task while starting the application job; S3, if the program is executed normally until the end, the performance counting based on the performance counter does not interfere with the program; S4, if the program execution is wrong, the error location processing flow is started; S45, an abnormal task is reported for further accurate error cause location. The present invention effectively improves the debugging and development efficiency of developers and improves the utilization rate of system resources.

A multi-actuator storage device includes a first actuator supporting a first read element coupled to a shared read channel, a second actuator supporting a second read element coupled to the shared read channel, and a job scheduler that employs a lowest-access time selection methodology when selecting each next job to schedule on the first actuator. When the lowest-access-time methodology results in the selection of a read job for execution by the first actuator, the job scheduler identifies a pending job for execution by the second actuator that can be executed concurrent to the first read job without concurrently accessing the read channel and schedules the pending job on the second actuator concurrent to the first read job.

A system includes storage of job data describing a computing job in a job data location, creation of a queue entry associated with the computing job, the queue entry comprising a pointer to the job data location and a subset of the job data, storage of the queue entry in a job scheduler queue of a lock-free skiplist at a position based on a priority of the job, and reading of data from the queue entry and from a plurality of other queue entries of the job scheduler queue.

The described technology relates to scheduling jobs of a plurality of types in an enterprise web application. A processing system configures a job database having a plurality of job entries, and concurrently executes a plurality of job schedulers independently of each other. Each job scheduler is configured to schedule for execution jobs in the jobs database that are of a type different from types of jobs others of the plurality of job schedulers are configured to schedule. The processing system also causes performance of jobs scheduled for execution by any of the plurality of schedulers. Method and computer readable medium embodiments are also provided.