66 results about "Check point" patented technology

The invention belongs to the field of large model distributed training, and particularly relates to a large model distributed training fault processing method based on a dynamic check point strategy. Comprising (1) a check point distributed access strategy of cluster topology and environment dynamic perception; (2) a large model iteration time prediction method of variation trend perception; and (3) a check point frequency decision method of model training iteration time and trend perception. According to the method, the problems of speed and loss in the traditional check point recovery process are solved, the resource utilization rate is optimized by dynamically adjusting the check point updating frequency, and the redundancy overhead in the training process is reduced. Through the improvements, the large-scale deep learning model can deal with faults more flexibly and efficiently when facing a complex training environment, and the continuity and stability of training are ensured, so that a new thought and direction are provided for the development of a large model training technology.

The invention relates to a check point reading and writing method and device in distributed training, a storage medium and a program product. The method comprises the steps that in response to check point file generation and the number of check point files in a memory of a training node cluster corresponding to a target distributed training task does not reach N, an idle target memory is determined in the memory of the training node cluster, and the check point files are written into the target memory, the memory of the training node cluster comprises the memory of each training node in the training node cluster; and in response to a fault of any training node in the training node cluster, reading the target check point file from the memory of the training node cluster according to the metadata of the target check point file corresponding to the training node, and recovering training based on the target check point file. According to the method and the device, the check point file can be quickly read from the memory of the training node for recovery when the training node fails.

The embodiment of the invention provides a distributed training system and a distributed training method. The distributed training system and the distributed training method are used for improving the training efficiency of model distributed training. Training nodes in the distributed training system are used for storing check points in video memories of GPUs of the training nodes and sending the check points to memories of CPUs of the training nodes; the training node is also used for sending the check points in the memory of the CPU to the storage system through asynchronous operation; the training node is also used for sending abnormal information to the management node under the condition that the node state information of the training node is abnormal information; the management node is also used for determining whether the training node is a fault node based on the abnormal information, and isolating the fault node under the condition that the training node is the fault node; and the management node is also used for loading a target check point corresponding to the fault node from the storage system to other training nodes under the condition that the training node is the fault node, so that other training nodes complete a model training task of the fault node based on the target check point.

A check point remote backup method based on large model parallel strategy perception comprises the following steps: S1, constructing a Checkpoint backup scheme: S11, constructing different cross-node parameter backup mapping relationships according to different parallel modes; s12, constructing a model parameter fragment distribution scheme according to the parameter load capacity of each card and the redundant copy number; s13, grouping the training nodes according to the number of backup copies set by a user, and generating a plurality of full mutual backup groups with equal sizes; the training nodes in the full mutual backup group are mutually backed up; s2, managing memory allocation in a Checkpoint backup process on the basis of a memory multiplexing mechanism; scheduling of CPU resources in the Checkpoint data transmission process is managed according to the asynchronous transmission process; s3, realizing punctual writing of the Checkpoint through a round and time threshold combined triggering method; and S4, controlling the startup and scheduling of the Checkpoint backup. According to the method, when the node breaks down, the system can quickly recover the model state from other nodes according to a set backup strategy, and the continuity of a training task and the fault-tolerant capability of the system are ensured.

The invention relates to a GPU (Graphics Processing Unit) check point storage method in large model distributed training, which comprises the following steps of: 1, loading check point configuration information when training is started; 2, monitoring a back propagation completion signal of each layer in the training process, and immediately triggering check point fragment storage operation of a certain layer after parameter updating of the layer is completed; 3, the model state of the layer is asynchronously copied to a CPU memory from a GPU memory, and consistency verification is carried out; 4, asynchronously storing the check point fragments from the CPU memory into a persistent storage; 5, when training needs to be recovered, the check point file is loaded from the persistent storage, integrity verification is carried out, all layers of fragmented data are recombined into a complete model state, and training is recovered. And the synchronous blocking and I / O bottleneck of the check points in the training process are reduced, so that the training efficiency is improved, and the fault recovery time is shortened.

The invention discloses a multi-device collaborative integrated kitchen management system, and relates to the technical field of intelligent kitchen control, and the system comprises the steps: analyzing an order or menu request into a task graph, building dependence, and adding a capability demand, resource mutual exclusion, a quality window, a risk level and migration window, and a check point strategy label for a task node; the edge arrangement gateway collects the equipment state, the resource occupation and the network health degree, and reschedules and generates a main plan and a shadow plan under the constraint of a rolling window and a freezing window; a main plan is converted into an intention type transaction to be issued, idempotent processing is carried out according to a transaction identifier, feeding, discharging and valve opening are carried out, token reservation-consumption is carried out, and account checking and verification are carried out; generating a process check point according to a check point strategy, and executing thermal migration, compensation or security degradation and account checking synchronization according to a migration window and a shadow plan when the process check point is abnormal; repeated actions and state drifting caused by re-delivery are reduced, and the stability of continuous meal delivery is guaranteed.

One or more implementations of this specification provide model training and checkpoint file storage systems and methods. In an implementation, a method includes executing, by a model training module of a storage system, a training task of an artificial intelligence model, during execution of the training task, suspending, by the model training module, the training task if a first checkpoint file is generated and sending a request to a checkpoint file processing module of the storage system to cache the first checkpoint file, locally caching, by the checkpoint file processing module, the first checkpoint file based on the request, and concurrently performing, by the check file processing module, a notification operation and a storage operation.

The invention relates to a large model training check point storage method and device and a storage medium, and the method comprises the steps: when a main training thread executes an iterative training task, through an independently-running asynchronous thread, executing a check point storage operation on the main training thread before the main training thread executes the check point storage operation; a memory space matched with the check point data structure is applied and organized in a CPU memory in advance; when the check point needs to be stored, the main training thread can directly write the latest check point data into the pre-allocated target memory in an overwriting manner, so that time-consuming memory application and organization operation are stripped from a key path of the main thread. Through the method and the device, the influence of check point storage on the training task is reduced, and the utilization rate of the GPU in the training process is effectively improved.

The invention discloses an assembly line check point implementation and fault recovery method and system based on a distributed memory, and belongs to the technical field of cross-center assembly lines. The method comprises the following steps: dividing a cross-data-center assembly line into a plurality of sub-assembly lines, and executing the sub-assembly lines on corresponding data centers in a cluster respectively; when the execution of the sub-assembly line is finished, acquiring Arrow format data of the current check point; storing the Arrow format data of the current check point in a distributed memory system or a disk storage system of a cluster; and when a system fault occurs or the execution of the cross-data center assembly line task fails, fault recovery is performed based on the Arrow format data of each check point. According to the method, the interruption and data loss risks of assembly line tasks can be reduced, and the reliability and fault tolerance of a large-scale data processing assembly line are improved.

The invention provides a data processing method and device, electronic equipment and a storage medium, and relates to the technical field of deep learning, in particular to the technical field of distributed large model training optimization. According to the specific implementation scheme, an original check point file corresponding to a current training model is obtained, wherein the original check point file comprises weight parameters and optimizer state parameters; segmenting optimizer state parameters in the original check point file to obtain a first check point file; and adding the newly added weight parameters into the first check point file, segmenting and merging the optimizer state parameters corresponding to the newly added weight parameters into the first check point file according to the number of the parameters to obtain a second check point file, and performing hot start training through the parameters in the second check point file. The multi-modal data processing capability of the model is remarkably enhanced, meanwhile, time and resource consumption caused by training from the beginning is avoided, and the continuity of model training and the stability of model performance are improved.

The invention relates to the technical field of artificial intelligence, and particularly provides a check point storage method and device of an artificial intelligence model, a medium and a product. The method comprises the following steps: acquiring storage equipment information and PCIe topological information, and constructing a transmission topological graph based on the storage equipment information and the PCIe topological information; based on the transmission topological graph, determining a target parallel transmission path from the GPU to the storage device; dividing the global check point data into a plurality of check point fragments based on the bandwidth capability of the target parallel transmission path; transmitting the plurality of check point fragments from the GPU to corresponding storage equipment in parallel through a plurality of independent transmission streams; and executing a two-stage persistence operation, and transmitting and storing the check points in the storage device to a remote storage system in a fragmented manner. According to the method provided by the invention, a data transmission path can be systematically optimized, heterogeneous storage resources are cooperatively utilized, and high reliability and rapid recovery capability are considered, so that the training efficiency of a large-scale artificial intelligence model is improved.

The embodiment of the invention provides a training cluster check point storage method, a training cluster check point recovery method and electronic equipment. The training cluster check point storage method comprises the steps that distributed cluster node memories cooperatively store check points, and the memories are used for storing check point data of each round of training. According to the method provided by the embodiment of the invention, cluster memory collaborative management is adopted, the node memory stores the whole check point, and the storage frequency of the check point is improved; in addition, a cluster memory control manages multi-node check points and supports grouped redundant backup; and moreover, the check points are directly stored in the memory, so that the time delay of storing the check points is greatly reduced.

The invention relates to the technical field of artificial intelligence, and discloses a model check point file management system which comprises a data downloader, a metadata storage system, a local storage system and a remote storage system. The data downloader is used for receiving a check point file acquisition request and sending a metadata acquisition request for acquiring a storage address of a first check point file to the metadata storage system; the metadata storage system is used for querying first metadata corresponding to the first check point file according to the metadata acquisition request and sending a first storage address to the data downloader; the data downloader is used for acquiring the first check point file from the remote storage system according to the first storage address and storing the first check point file to the local storage system; the local storage system is used for returning the first check point file to the first training task. According to the method, massive check point files can be stored, the storage performance is optimized, and reliable operation of model training tasks can be guaranteed.

A page distortion correction method and system, an electronic device and a computer readable medium. The page distortion correction method comprises: determining a target text line to be corrected, and selecting a plurality of check points in the target text line, the plurality of check points satisfying that the connecting line of the plurality of check points is coincident with or parallel to a correction reference line when the page is not distorted; comparing the positions of the plurality of check points with the correction reference line, and calculating the offset amount of the check points to be corrected; and correcting each character of the target text line according to the calculated offset amount. The method is simple and efficient, has good restoration, strong adaptability, and the correction result is more stable.

The invention discloses a program execution management method and device, a vehicle and a medium. The method comprises the steps that a program instruction of a vehicle electronic control unit is received; wherein the program instruction is generated when a check point is triggered based on a program execution sequence running in the electronic control unit; analyzing the program instruction to obtain address information of the check point; matching the address information with pre-stored address information to obtain a first matching result; wherein the pre-stored address information is obtained from the cache table; if the first matching result is that the address information is not matched, the address information is locked in a door lock trigger, and fault reporting is carried out through the door lock trigger. According to the technical scheme, the program execution sequence is managed in the system-level chip according to the matching operation of the pre-stored address information in the cache table, an extra CPU period can be avoided or minimized, extra cost and performance overhead are reduced, and safety of vehicle functions is guaranteed.

The invention relates to a method for accelerating model training recovery, electronic equipment and a storage medium. The method comprises the steps of executing a first task for constructing a training component based on an instruction of recovery model training, and independently executing a second task through a concurrent execution unit corresponding to a main execution unit; wherein the second task is used for loading the check point file corresponding to the training component to the memory from the external storage system; wherein the training component comprises a model object and an optimizer object, the check point file comprises state data, stored in historical model training, of the training component, and the first task and the second task are at least partially overlapped in execution time; and in response to an event of completing the first task and the second task, loading the state data in the memory into the corresponding training component to complete state recovery of the training component. According to the method, the parallel support of each task in the model training recovery process can be realized, and the time consumption of model training recovery is effectively reduced.

The invention provides a safety function block active defense method and system for a safety instrument system. The method comprises the following steps: before executing a core algorithm, performing pointer security check and numerical field check on input data of a security function block; if the check is passed, entering a pre-operation stage, and monitoring an operation risk in real time through a preset check point by utilizing a reverse analysis pre-operation algorithm or a soft floating point pre-operation algorithm on the premise of not executing an actual instruction which possibly causes hardware exception; and executing a deterministic output strategy according to a pre-operation result, and outputting a unified hierarchical state code (ERR) through a special state pin. The method has the advantages of endogenous safety, fault behavior determination, high diagnosis precision, high configuration efficiency and the like, controller shutdown caused by data exception can be effectively avoided, and the robustness of an industrial control system is improved.

The invention relates to a simulation analysis method and device for program fragments. Comprises: constructing an original program file of a target program; running a target program fragment in the original program file through a simulator, capturing a corresponding running state, and generating check point data; merging the check point data into the original program file to obtain a to-be-tested program file; and according to the check point data in the to-be-tested program file, recovering the running state of the target program fragment in the to-be-tested program file in the test environment, and obtaining the performance data of the target program fragment during running of the target program. By optimizing storage and recovery processes of check points, the number of times of operation of a specified program on a simulator is reduced, tracking of a CPU register, codes and data is avoided, and the performance is improved; meanwhile, the construction method of the test instruction slicing system is improved, so that the generated program slices can run in target environments such as a simulator, a simulation environment and an FPGA without depending on a specific test system, and portability and test efficiency are improved.

The invention relates to processing of containerized environments. The present disclosure relates to a computer-implemented method of processing a containerized environment related to a service requested by a client, comprising the steps of: (a) monitoring the time taken by a container to initialize and / or the number of requests for a service received by the container from the client; (b) determining a container selected from among the containers based on the monitoring; (c) performing checkpoint setting on the selected containers and storing the checkpoint of each selected container in a container checkpoint cache; and (d) when the client requests the service, reading the corresponding check point in the container check point cache and recovering the corresponding selected container from the stored check point so as to serve the service.

The invention discloses a dynamic setting method and device of check points and a storage medium. In the method, in a training process, resource use data of a large model training cluster in a previous historical time period can be periodically collected, and first dynamic relation data used for representing a dynamic relation of a system resource occupation state along with time change in a future target time period is predicted according to the collected resource use data; and comparing the first dynamic relation data with predetermined global dynamic relation data expected in the whole process under the condition that a fault does not occur in the training process of the large model, so as to determine a prediction result whether the large model training cluster may have the fault in the target time period. And the check points are configured according to the prediction result, so that the effectiveness of the configured check points in the training process of the large-scale model can be improved, and the fixed overhead and fault recovery loss of the periodic check points are reduced.

The invention relates to a check point loading method and device in distributed training, a storage medium and a program product. The method comprises the following steps: segmenting a check point file in the shared storage device into multiple pieces of block data; for any training node in a training node cluster corresponding to the target distributed training task, loading block data corresponding to the training node from a shared storage device, and storing the block data corresponding to the training node into a local storage device of the training node; and exchanging block data among different training nodes of the training node cluster by adopting a point-to-point technology until each training node of the training node cluster respectively obtains each piece of block data corresponding to the check point file. According to the method and the device, the loading speed of the check point can be obviously increased, and the loading bottleneck problem caused by storage bandwidth limitation is effectively avoided.

The invention relates to a multi-mode check point design method based on an embedded virtualization environment, and belongs to the field of embedded system simulation. The invention relates to an embedded system testing method combining a check point function and fault injection. According to the method, check points are regularly stored in the operational process so as to be quickly recovered when a fault occurs. Meanwhile, by introducing a fault injection mechanism, an embedded developer can combine a check point function and a fault injection module to store the specific state of one or more modules before fault injection, various abnormal conditions can be simulated and tested to verify the fault tolerance and stability of the system, and the fault injection efficiency is improved. And the state of the whole system or a specific module can be recovered at any time according to needs. According to the method, the storage and processing efficiency of the check points is optimized, the resource overhead is reduced, the efficient operation of the embedded system in a resource-constrained environment is ensured, and the reliability and the fault recovery capability of the system are improved.

The invention provides a method and a device for performing model training through computing power of an intelligent computing center, and relates to the technical field of intelligent computing centers, intelligent computing centers and computing power infrastructure, the method comprises the following steps: S1, under the condition that training interruption of a model is detected, obtaining target check point information through a first function, the target Checkpoint information is the Checkpoint information with the minimum time difference between the Checkpoint information generation moment and the current moment, and the first function is a preset function used for acquiring the Checkpoint information; s2, obtaining training information of the model according to the target Checkpoint information, wherein the training information of the model is in one-to-one correspondence with the target Checkpoint information; and S3, training the model based on the training information of the model. The method can greatly improve the training efficiency of the model, thereby avoiding a large amount of waste of computing power resources.

The invention relates to the technical field of block chains, in particular to a block chain local account book data synchronization method and system. Comprising the following steps: S1, when block checking is carried out on a block chain, storing the positions of checked blocks every time checking is completed, marking the positions as two check points, and respectively writing hash values into the check points; s2, starting the backtracking check of the new block, and when one check point is backtracked, updating the other check point as the mark of the position of the new block; s3, alternately updating the two check points on the block chain; s4, when the block chain is not bifurcated, the two check points are alternately updated; s5, when the block chain is bifurcated, one check point is located in an effective bifurcated chain, and the two check points are updated alternately; and S6, when the block chain is bifurcated and one check point is located in an invalid bifurcated chain, generating a new check point by taking the check point of the main chain as a backtracking end point, and continuously and alternately updating. The front check point and the rear check point are set in the block chain, so that the repeated workload of checking the local account book can be reduced.

A payment management system including a payment management unit that initiates a transaction, an operation unit that manages the issuance of at least one certificate, a governance unit that operates based on at least one parameter defined in the at least one certificate; at least one validation unit that receives the at least one certificate and validates portions of the transaction; and a payment unit that issues payment based on the terms of the at least one certificate, where the operation unit crates a governance protocol for each transaction with the governance protocol including at least one epoch and at least one checkpoint.

The invention discloses a storage method of a model check point, which is applied to persistent storage of the model check point in the field of artificial intelligence (AI). According to the storage method of the model check points, the operation of updating model parameters in the model training process is deployed on a central processing unit (CPU) to be executed, and the updated model parameters can be reserved after the CPU executes the operation of updating the model parameters. Thus, during the period that the accelerator continues to execute model training based on the updated model parameters, the CPU can store the reserved model parameters in the target storage medium at the same time, persistent storage of the model parameters can be achieved without interrupting the training process, and normal training of the AI model is prevented from being affected.

The present invention relates to a method for setting collision information for simulating robot motion and an apparatus for performing the same, the method including the steps of: obtaining robot installation information about an operation space of a robot, dangerous region information, and body part information in which a collision can occur in each dangerous region; setting a check point for judging a collision danger with respect to the robot; and calculating coordinate information and shape information of the check point based on at least a part of the obtained information and the set check point.

The invention relates to the technical field of data processing, in particular to an electric energy meter AI code review method and system based on check point cues. The method is applied to a system architecture comprising a client and an AI server, and comprises the following steps: inputting an embedded engineering code and a cue word for examination through the client; the client carries out grammatical analysis on the embedded code to form an analysis block; the client combines the analysis blocks with the corresponding cue words and matches standardized information in combination with an external knowledge base to form an examination unit; the AI server receives the review unit, dynamically identifies the problems of array crossing, excessive memory applications and variable defense, and outputs a structured review result; and the client summarizes all review results, performs classified statistics according to defect types, and generates visual reports and logs. According to the invention, the detection accuracy of specific scene defects is obviously improved; and the time cost of diamagical examination and the risk of artificial omission are greatly reduced.

The invention discloses an environment security check method and device for task automatic verification, and a storage medium, and the method comprises the steps: screening all security check personnel according to the check point information of a check point, and determining a target security check personnel in all security check personnel; generating a security check task according to the check frequency and position information in the check point information and the shift information and personnel information of the target security inspector, and sending the security check task to a target security check terminal of the target security inspector; at any moment in the security check task execution process, automatically or responding to a control instruction to obtain a task execution image; and performing compliance examination on the task execution content in the task execution image, and giving an alarm if the task execution content does not meet the rule. According to the invention, high management and control of security check task execution are realized, and the efficiency of environmental security check is improved.

A memory system includes a memory device and a controller. The memory device includes a plurality of memory blocks. The controller recognizes a change in a pattern of a plurality of write data, delays a checkpoint operation associated with a write operation regarding the plurality of write data entries when operational data regarding the plurality of write data entries is within a range that is capable of being stored in a buffer, and performs the checkpoint operation when the operational data is beyond the range.