Model training task performance testing method, system, and computing device
By employing monitoring and feature extraction methods, the problem of low diagnostic efficiency for performance issues during large model training is addressed, enabling online performance diagnosis and improving the accuracy and efficiency of anomaly localization and diagnosis.
Patent Information
- Application Number
- PCT/CN2026/095218
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-09
- Filing Date
- 2026-05-07
- Publication Date
- 2026-11-12
AI Technical Summary
In the training of large models, performance problems occur frequently and are difficult to locate, resulting in low diagnostic efficiency. This is especially true in GPU-centric large model training systems, where existing methods are difficult to apply, and data acquisition affects task training performance.
By monitoring the performance of training tasks, acquiring performance analysis data, extracting features, determining the operational behavior characteristics of the objective function, and determining performance detection results based on these features, the system focuses on abnormal functions that affect the performance of training tasks, thereby achieving online diagnosis.
It improves the efficiency of diagnosing complex performance problems during model training, reduces the additional overhead of data acquisition, improves the accuracy of anomaly localization, and realizes online performance diagnosis.
Smart Images

Figure CN2026095218_12112026_PF_FP_ABST
Abstract
Description
Performance testing methods, systems, and computing devices for model training tasks Technical Field
[0001] This disclosure relates to the fields of large model technology and performance diagnostics, and more specifically, to a performance testing method, system, and computing device for model training tasks. Background Technology
[0002] During the training of large models, the interaction between hardware resources such as GPUs (Graphics Processing Units), CPUs (Central Processing Units), and network devices and software elements such as training frameworks, data loading, and communication libraries is quite complex, leading to frequent performance problems and making it difficult to pinpoint the cause of these problems, resulting in low efficiency in diagnosing performance issues.
[0003] To address this issue, a method relying on fine-grained tracking and data analysis is employed to diagnose performance problems. However, this approach is only suitable for small-scale CPU-based systems and is difficult to apply to large model training systems centered around GPUs. Furthermore, obtaining data for performance analysis requires probing system components, which can negatively impact task training performance.
[0004] There is currently no effective solution to the above problems. Summary of the Invention
[0005] This disclosure provides a method, system, and computing device for performance testing of model training tasks, in order to at least solve the technical problem of low diagnostic efficiency for complex performance issues during model training.
[0006] According to one aspect of the present disclosure, a performance detection method for a model training task is provided, comprising: monitoring the performance of a training task during the execution of a training task on a target model using at least one computing node; when an abnormality is detected in the performance of the training task, obtaining performance analysis data of the training task from at least one computing node, wherein the performance analysis data is used to characterize data that has a correlation with multiple functions contained in the target model; performing feature extraction on the performance analysis data to obtain the operational behavior features of at least one target function, wherein the at least one target function is a function among multiple functions that affects the performance of the training task; and determining the performance detection result of the training task based on the operational behavior features of the at least one target function, wherein the performance detection result includes: at least one target function that exhibits abnormal behavior.
[0007] According to another aspect of the present disclosure, a performance detection method for a model training task is also provided, comprising: responding to a model training request sent by a client, determining the target model and training task corresponding to the model training request; executing the training task on the target model using at least one computing node and monitoring the performance of the training task; if an abnormality is detected in the performance of the training task, obtaining performance analysis data of the training task from at least one computing node; performing feature extraction on the performance analysis data to obtain the operational behavior features of at least one objective function, wherein the at least one objective function is a function among multiple functions that affects the performance of the training task; determining the performance detection result of the training task based on the operational behavior features of the at least one objective function, wherein the performance detection result includes: the function in the at least one objective function that exhibits abnormal behavior; and outputting the performance detection result to the client.
[0008] According to another aspect of the present disclosure, a performance detection system for a model training task is also provided, comprising: at least one computing node configured to perform a training task on a target model; and a detection device connected to the at least one computing node, configured to monitor the performance of the training task, and, when an abnormality is detected in the performance of the training task, acquiring performance analysis data of the training task from the at least one computing node, extracting features from the performance analysis data to obtain the operational behavior features of at least one objective function, and determining the performance detection result of the training task based on the operational behavior features of the at least one objective function, wherein the performance analysis data is used to characterize data that has a correlation with multiple functions contained in the target model, the at least one objective function is a function among the multiple functions that affects the performance of the training task, and the performance detection result includes: at least one objective function that exhibits abnormal behavior.
[0009] According to another aspect of the present disclosure, a computing device is also provided, including: a memory storing an executable program; and a processor configured to run the program, wherein the program executes the methods in various embodiments of the present disclosure when it runs.
[0010] According to another aspect of the present disclosure, an electronic device is also provided, including: a memory storing an executable program; and a processor connected to the memory via a bus and configured to run the program, wherein the program executes the methods in various embodiments of the present disclosure when it runs.
[0011] According to another aspect of the embodiments of the present disclosure, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is executed, it controls the device where the computer-readable storage medium is located to perform the methods of the various embodiments of the present disclosure.
[0012] According to another aspect of the embodiments of this disclosure, a computer program product is also provided, including a computer program that, when executed by a processor, implements the methods of various embodiments of this disclosure.
[0013] According to another aspect of the embodiments of this disclosure, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods of various embodiments of this disclosure.
[0014] According to another aspect of the embodiments of this disclosure, a computer program is also provided, which, when executed by a processor, implements the methods of the various embodiments of this disclosure.
[0015] In this embodiment, during the training of a target model using at least one computing node, the performance of the training task is monitored. If an anomaly is detected in the training task's performance, performance analysis data is obtained from at least one computing node. Feature extraction is performed on the performance analysis data to obtain the operational behavior features of at least one objective function. Based on the operational behavior features of at least one objective function, the performance detection result of the training task is determined. It is noteworthy that by monitoring the training task online to capture performance anomalies, and only initiating performance analysis data collection when an anomaly is detected, the additional overhead and impact on training performance caused by continuous collection are avoided. Then, the performance analysis data is transformed into operational behavior features of the objective function through feature extraction, thereby reducing the amount of data and improving the efficiency of performance detection. Simultaneously, focusing on objective functions correlated with the performance of the training task further narrows the diagnostic scope. Determining the performance detection result of the training task online based on the operational behavior features of the objective function improves the accuracy of anomaly localization, thus achieving the technical effect of improving the diagnostic efficiency of complex performance problems during model training. This realizes online diagnosis of performance anomalies and solves the technical problem of low diagnostic efficiency for complex performance problems during model training.
[0016] It is worth noting that the above general description and the following detailed description are merely for illustrative and explanatory purposes and do not constitute a limitation thereof. Attached Figure Description
[0017] The accompanying drawings, which are included to provide a further understanding of this disclosure and form part of this disclosure, illustrate exemplary embodiments of the present disclosure and are used to explain the disclosure, but do not constitute an undue limitation of the disclosure. In the drawings:
[0018] Figure 1 is a schematic diagram of an application scenario of a performance detection method for a model training task according to an embodiment of the present disclosure;
[0019] Figure 2 is a flowchart of a performance detection method for a model training task according to an embodiment of the present disclosure;
[0020] Figure 3 is a schematic diagram of an optional function priority arrangement according to an embodiment of the present disclosure;
[0021] Figure 4 is a schematic diagram of an optional extraction function runtime behavior feature according to an embodiment of the present disclosure;
[0022] Figure 5 is a flowchart of another performance detection method for a model training task according to an embodiment of the present disclosure;
[0023] Figure 6 is a schematic diagram of a performance detection system for a model training task according to an embodiment of the present disclosure;
[0024] Figure 7 is a structural block diagram of a computing device according to an embodiment of the present disclosure;
[0025] Figure 8 is a structural block diagram of an electronic device according to an embodiment of the present disclosure. Detailed Implementation
[0026] To enable those skilled in the art to better understand the present disclosure, the technical solutions of the present disclosure will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present disclosure, and not all embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present disclosure.
[0027] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data are interchangeable where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0028] The technical solution disclosed herein is primarily implemented using large-scale model technology. Here, "large-scale model" refers to a deep learning model with a massive number of parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of parameters. Large-scale models are also known as foundation models. They are pre-trained using large-scale unlabeled corpora to produce pre-trained models with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and exhibit good generalization ability. Examples include Large Language Models (LLMs) and multi-modal pre-training models.
[0029] It's important to note that in practical applications, large models are fine-tuned using a small number of samples after pre-training, allowing them to be applied to different tasks. For example, large models are widely used in Natural Language Processing (NLP), computer vision, and speech processing, specifically in computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as in NLP tasks such as text-based sentiment classification, text summarization, and machine translation. Therefore, the main application scenarios for large models include, but are not limited to, digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.
[0030] First, some nouns or terms that appear in the description of the embodiments of this disclosure shall be interpreted as follows:
[0031] Large-scale model training tasks involve training neural network models with a massive number of parameters (typically billions or more). This usually requires significant computing resources and distributed training techniques.
[0032] PyTorch is a widely used deep learning framework that supports the training of large models, provides dynamic computation graphs and distributed training tools, and is widely used in the development and training of neural networks.
[0033] GPU: Graphics Processing Unit, is the main computing hardware for training large models. It has powerful parallel computing capabilities and performs a large number of matrix and tensor operations.
[0034] CPU: Central Processing Unit, used in large model training to perform data preprocessing, task scheduling, and some non-parallel computation.
[0035] PCIe (Peripheral Component Interconnect Express) is a high-speed communication bus used to connect the CPU and GPU, transmitting data between them. Its bandwidth and latency affect the GPU's data acquisition speed, thus impacting training performance.
[0036] Network interface card (NIC): A card used to connect computing nodes. In distributed training of large models, NIC performance affects the efficiency of data communication between nodes.
[0037] GPU computation kernel functions: Parallel computation functions that run on the GPU and are used to perform the core computations of the model, such as matrix multiplication and convolution operations.
[0038] Collective communication functions: In distributed training, communication functions are used to synchronize and exchange data between multiple GPUs or nodes, including but not limited to AllReduce (a distributed communication function) and Broadcast (another distributed communication function).
[0039] Measurement and analysis (Profiling): A method used to collect and analyze runtime performance data of programs, identify performance bottlenecks, and optimize the training efficiency of large models.
[0040] Training iteration: In model training, this refers to a complete process of forward propagation, loss calculation, backpropagation, and parameter update. The iteration time within a training iteration is a crucial metric for evaluating training performance.
[0041] According to embodiments of this disclosure, a performance testing method for a model training task is provided. It should be noted that the steps shown in the flowcharts in the accompanying drawings are executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowcharts, in some cases, the steps shown or described are executed in a different order than that shown here.
[0042] Considering the large number of model parameters in the large model and the limited computing resources of mobile terminals, the method provided in this disclosure is applied to the application scenario shown in Figure 1, but is not limited thereto. In the application scenario shown in Figure 1, the large model is deployed on server 10. Server 10 connects to one or more client devices 20 via a local area network (LAN), wide area network (WAN), internet connection, or other types of data network. These client devices 20 include, but are not limited to, smartphones, tablets, laptops, PDAs, personal computers, smart home devices, and in-vehicle devices. The client devices 20 interact with the user through a graphical user interface to invoke the large model, thereby implementing the method provided in this disclosure.
[0043] In this embodiment, the system comprising a client device and a server performs the following steps: the client device sends a model training request and outputs performance detection results. The server determines the target model and training task corresponding to the model training request; performs the training task on the target model using at least one computing node and monitors the performance of the training task; if an anomaly is detected in the performance of the training task, it obtains performance analysis data of the training task from at least one computing node; it extracts features from the performance analysis data to obtain the operational behavior features of at least one objective function; and based on the operational behavior features of at least one objective function, it determines the performance detection results of the training task.
[0044] It should be noted that with the rapid development of high-performance computing units, the methods provided in this disclosure are also applied to integrated model machines in other application scenarios. In one optional embodiment, the integrated model machine has multiple built-in models. The user selects one model to adjust as needed to obtain their own model. The high-performance computing unit built into the integrated model machine then directly calls the adjusted model to execute the methods provided in this disclosure. In another optional embodiment, the large integrated model machine has a pre-trained model built-in. The high-performance computing unit built into the integrated model machine then directly calls this model to execute the methods provided in this disclosure.
[0045] Furthermore, when users need to train their own models, they can upload their datasets via the client. These datasets are then sent to the server, which uses the dataset to adjust the pre-trained model, resulting in the user's customized model, which can then be deployed to the production environment. To facilitate user adjustments, the server provides complete adjustment tools, development frameworks, and processes, supporting various adjustment strategies. This allows the adjusted model to better adapt to different application domains and achieve a high degree of customization.
[0046] Under the aforementioned operating environment, this disclosure provides a performance detection method for a model training task, as shown in Figure 2. Figure 2 is a flowchart of the performance detection method for a model training task according to an embodiment of this disclosure. As shown in Figure 2, the method includes the following steps:
[0047] Step S202: During the process of performing a training task on the target model using at least one computing node, monitor the performance of the training task.
[0048] The computing nodes in the above steps are independent computer systems that participate in executing computational tasks. In scenarios involving large model training, computing nodes are servers equipped with GPUs, but are not limited to this. Computing nodes collaborate with each other to jointly execute model training tasks. Computing nodes are used to perform core computational tasks such as forward propagation, back propagation, and parameter updates of the target model, while also handling auxiliary tasks such as data preprocessing, communication, and synchronization.
[0049] The target model in the above steps is a neural network model used to perform training tasks. It is a complex neural network model with billions or even more parameters, or a neural network model with relatively few parameters and a relatively simple structure.
[0050] The training tasks in the above steps include, but are not limited to, forward propagation of the target model, loss calculation, backpropagation, and parameter update. The performance of the training tasks is characterized by metrics such as training iteration time and model convergence speed.
[0051] The performance anomalies mentioned above refer to situations where the performance of the training task on the target model does not meet expectations or fall within the normal range. For example, performance anomalies may manifest as a sudden increase in iteration time, abnormal fluctuations in hardware resource utilization, or increased data communication latency, but are not limited to these. Causes of performance anomalies include, but are not limited to, hardware failures, improper software configuration, code defects, and resource contention.
[0052] In one optional embodiment, the performance anomaly of the training task is determined by the iteration time of the iterative task. Specifically, a standard iteration time is preset, and then the iteration time of the iterative task is compared with the standard iteration time. If the difference between the iteration time of the iterative task and the standard iteration time exceeds a preset threshold, it is determined that the training task has a performance anomaly. It should be noted that the standard iteration time is obtained through analysis of experimental data, and is also set based on existing data and industry experience, but is not limited to these. When setting the preset threshold, the inherent volatility of model training and external interference factors are taken into account, and the preset threshold is also set according to the actual application situation, but the method of setting the preset threshold is not limited to these.
[0053] In another optional embodiment, performance anomalies in the training task are determined by hardware resource utilization. Specifically, a threshold for hardware resource utilization is preset, and then the hardware resource utilization of the iterative task is compared with the threshold. The comparison result determines whether the training task exhibits performance anomalies. For example, if the GPU resource utilization is lower than the GPU resource utilization threshold, or the CPU resource utilization is higher than the CPU resource utilization threshold without significant load, then performance anomalies in the training task are determined. It should be noted that the hardware resource utilization threshold is obtained through analysis of experimental data, and is also set based on existing information and industry experience, but is not limited to these methods.
[0054] Step S204: If an abnormality is detected in the performance of the training task, obtain the performance analysis data of the training task from at least one computing node, wherein the performance analysis data is used to characterize data that has a correlation with multiple functions contained in the target model.
[0055] The performance analysis data in the above steps is a detailed record of the training task's runtime performance, including but not limited to function call sequences, hardware resource usage (such as CPU and GPU utilization), memory access patterns, and network communication status. This performance analysis data is used to pinpoint the root cause of performance anomalies.
[0056] In one optional embodiment, upon detecting an anomaly in the performance of the training task, a daemon process on the compute node is initiated. This daemon process collects data such as function call data, hardware resource usage, and network communication metrics during the training task. The data collected by the daemons on multiple compute nodes is then aggregated to obtain performance analysis data.
[0057] In another optional embodiment, when anomalies are detected in the performance of the training task, a measurement and analysis tool is used to record data such as function call data, hardware resource usage, and network communication metrics during the execution of the training task. Then, a daemon process on the compute node collects the data recorded by the measurement and analysis tool, and subsequently, the data collected by the daemon process is aggregated to obtain performance analysis data.
[0058] In another alternative embodiment, if an abnormality is detected in the performance of the training task, the cloud platform notifies the computing nodes participating in the training, so that at least one computing node sends data related to multiple functions contained in the target model to the cloud platform via streaming, thereby obtaining performance analysis data.
[0059] Step S206: Extract features from the performance analysis data to obtain the operational behavior features of at least one objective function, wherein the at least one objective function is a function that affects the performance of the training task among multiple functions.
[0060] The objective function in the above steps is a function whose execution efficiency affects the overall performance of the training task. For example, it may be a function on the performance-critical path of the training task, and the execution status of the objective function during the training process is a key focus of performance diagnostics. For example, the objective function may include, but is not limited to, GPU computation kernel functions, set communication functions, and memory operation functions.
[0061] The operational behavior features in the above steps are used to describe the behavior of the objective function during the training task execution process, including but not limited to function execution time, average utilization of hardware resources, and standard deviation of resource occupancy, which are used to reflect the stability and efficiency of the objective function.
[0062] In one optional embodiment, a function correlated with the performance of the training task is determined from multiple functions based on performance analysis data, and this function is used as the target function. Then, performance analysis data corresponding to the target function is obtained, and feature extraction is performed on this data to obtain the operational behavior features of the target function.
[0063] In another alternative embodiment, a deep neural network-based autoencoder is used to extract features from the performance analysis data. During feature extraction, the performance analysis data is input into a trained autoencoder, which extracts features from the data and outputs at least one behavioral feature of the objective function.
[0064] Step S208: Based on the operational behavior characteristics of at least one objective function, determine the performance detection results of the training task, wherein the performance detection results include: at least one objective function that exhibits abnormal behavior.
[0065] The performance detection results in the above steps are used to characterize which objective functions exhibit abnormal behavior in the training task. Abnormal behavior indicates that the operating behavior characteristics of the objective function deviate from the normal range, or that the objective function has significant differences in characteristics compared to the same function on other computing nodes.
[0066] In one optional embodiment, based on the operational behavior characteristics of the objective function, it is determined whether the operational behavior characteristics of the objective function deviate from a preset normal range. If the operational behavior characteristics of the objective function deviate from the preset normal range, the objective function is marked as having abnormal behavior, thereby determining the performance detection result of the training task.
[0067] In another optional embodiment, based on the operational behavior characteristics of the objective function, it is determined whether the operational behavior characteristics of the objective function are different from those of the same function on other nodes. When the operational behavior characteristics of the objective function are different from those of the same function on other nodes, the objective function is marked as having abnormal behavior, thereby determining the performance detection result of the training task.
[0068] In another alternative embodiment, a clustering algorithm is used to cluster the behavioral characteristics of at least one objective function, identifying clusters whose behavior is significantly different from that of the majority of clusters. Objective functions in these significantly different clusters are identified as objective functions exhibiting anomalous behavior, and the performance detection results of the training task are determined based on these anomalous objective functions.
[0069] In this embodiment, during the training of a target model using at least one computing node, the performance of the training task is monitored. If an anomaly is detected in the training task's performance, performance analysis data is obtained from at least one computing node. Feature extraction is performed on the performance analysis data to obtain the operational behavior features of at least one objective function. Based on the operational behavior features of at least one objective function, the performance detection result of the training task is determined. It is noteworthy that by monitoring the training task online to capture performance anomalies, and only initiating performance analysis data collection when an anomaly is detected, the additional overhead and impact on training performance caused by continuous collection are avoided. Then, the performance analysis data is transformed into operational behavior features of the objective function through feature extraction, thereby reducing the amount of data and improving the efficiency of performance detection. Simultaneously, focusing on objective functions correlated with the performance of the training task further narrows the diagnostic scope. Determining the performance detection result of the training task online based on the operational behavior features of the objective function improves the accuracy of anomaly localization, thus achieving the technical effect of improving the diagnostic efficiency of complex performance problems during model training. This realizes online diagnosis of performance anomalies and solves the technical problem of low diagnostic efficiency for complex performance problems during model training.
[0070] In one embodiment of this disclosure, monitoring the performance of a training task includes: determining the iteration time of any one iteration during the execution of multiple rounds of iterative training using at least one computing node; determining that the performance of the training task is abnormal when the iteration time meets a first preset condition, wherein the first preset condition is used to determine whether the performance of the training task is abnormal; and determining that the performance of the training task is not abnormal when the iteration time does not meet the first preset condition.
[0071] The iterative training mentioned above refers to the process in which the target model performs forward and backward propagation on a batch of data in the dataset during the execution of the training task. This includes, but is not limited to, calculating the prediction results, evaluating the loss function, and updating the model parameters based on the loss function through backpropagation.
[0072] The iteration time in the above steps refers to the time required to complete one iteration of training, including but not limited to the execution time of multiple steps such as data loading, forward propagation, back propagation, and parameter updates. It is also the time of the iteration training terminal. The iteration time directly affects the total time of the target model to perform the training task and the required computing resources, serving as an indicator for evaluating model training efficiency.
[0073] The first preset condition in the above steps is a pre-set condition used to determine whether the performance of the training task is abnormal. It is preset according to the actual application situation, and the specific content of the first preset condition is not limited here. For example, the first preset condition is that the iteration time exceeds a certain percentage of the normal iteration time, such as exceeding 5% of the average iteration time; or the iteration time shows an upward trend for several consecutive iterations; or the current iteration time exceeds a preset threshold compared with the average or median of the recent iteration times; or the iteration time suddenly increases to a level that has never appeared before, or exceeds the statistical range of historical iteration times, but it is not limited to these.
[0074] In one optional embodiment, the start and end times of any round of iterative training are recorded, and the difference between the end time and the start time is calculated to obtain the iteration time of any round of iterative training. In another optional embodiment, the interruption time of any round of iterative training is recorded as the iteration time of any round of iterative training.
[0075] For example, the first preset condition is that when the iteration time of any round of training exceeds a preset percentage compared to the statistical average iteration time, the performance of the training task is determined to be abnormal. The preset percentage could be 5%, 10%, or 15%, but the values listed above are merely examples, and the actual selectable preset percentage is not limited to these.
[0076] If the time taken for a certain iteration is found to be higher than the statistical average time by a preset percentage, it is determined that the iteration time meets the first preset condition, thus indicating that the performance of the training task is abnormal. If the time taken for a certain iteration is found not to be higher than the statistical average time by a preset percentage, it is determined that the iteration time does not meet the first preset condition. It is considered that the time taken for the current iteration is within an acceptable range, does not deviate significantly from the historical or expected iteration time, and the performance remains stable or normal, thus indicating that the performance of the training task is not abnormal.
[0077] In one embodiment of this disclosure, determining the iteration time corresponding to any round of iterative training includes: monitoring the invocation events of the training task and recording the invocation time of the invocation events; if there are a first preset number of preset invocation events among the recorded invocation events, determining the iteration time of any round of iterative training based on the invocation time of the first preset number of preset invocation events; if there are no first preset number of preset invocation events among the recorded invocation events, determining the iteration time of any round of iterative training based on the waiting time of the last invocation event.
[0078] The events invoked in the above steps are used to characterize the start of a series of computational or processing activities. For example, in deep learning model training, calling dataloader.next() (a function in deep learning model training) marks the start of the data loading process, while calling optimizer.step() (a function in deep learning model training) characterizes the start of the parameter update phase.
[0079] The call time in the above steps refers to the specific moment when the call event occurs, that is, the precise timestamp of the function or module call.
[0080] The preset call events in the above steps are pre-set call times, which are pre-screened according to the actual application situation. The specific content of the preset call events is not limited here. For example, the preset call events begin with the call to the dataloader.next() function and end with the call to the optimizer.step() function, but the preset call events are not limited to this.
[0081] The first preset quantity in the above steps is a pre-set quantity used to determine when the program has entered a stable training iteration cycle. It is used for preliminary review based on the actual application situation, and no specific value is limited to the first preset quantity here. For example, the first preset quantity is 10, but it is not limited to this; it could also be 15, 20, or other values selected based on the actual application situation.
[0082] In one optional embodiment, an event listener is set up on the computing node, and timestamp recording code is inserted at the start and end of the invocation event. The invocation event of the training task is detected by the listener, and the start and end times of the invocation event are obtained by the timestamp recording code, thereby obtaining the invocation time of the invocation event.
[0083] When the listener detects a first preset number of preset call events, these first preset number of preset call events are determined as the target event sequence. Based on the start and end times of the preset call events obtained by the timestamp recording code, the call times of the preset call events are determined, thus identifying the target event sequence. Further, the call times of the first preset number of preset call events in the target event sequence are added together to obtain the duration of the target event sequence, and the duration of the target event sequence is determined as the iteration time for this round of iterative training.
[0084] For example, the first preset number of preset invocation events is at least 10 events that begin with a call to the dataloader.next() function and end with a call to the optimizer.step() function.
[0085] In another optional embodiment, since blocking or abnormalities may occur during the training process, causing the listener to be unable to detect the first preset number of preset call events for a long time, the iteration time of this round of iterative training is based on the time elapsed since the last event was received, i.e., the interruption time.
[0086] In one embodiment of this disclosure, the method further includes the steps of: deleting the call time of the recorded call events, re-executing the call events of the monitoring training task, and recording the call time of the call events when the number of recorded call events is greater than or equal to a second preset number and there are no preset call events of a first preset number among the recorded call events.
[0087] The second preset quantity in the above steps is a pre-set quantity, preset according to the actual application situation. There is no limit to the specific value of the second preset quantity. The second preset quantity is used to ensure that a sufficient number of call events are detected to construct the target event sequence. For example, the second preset quantity is 200, but it is not limited to this; it could also be 220 or 240.
[0088] In one optional embodiment, the invocation events are monitored and the occurrence time of the invocation events is recorded. If the number of invocation events is greater than or equal to a second preset number, it is checked whether the number of invocation events greater than or equal to the second preset number includes at least a first preset number of preset invocation events. If the number of preset invocation events in the second preset number of invocation events does not reach the first preset number, the invocation time of the already recorded invocation events is deleted, and the steps of monitoring the invocation events of the training task and recording the invocation time of the invocation events are re-executed.
[0089] Using a first preset quantity of 10 and a second preset quantity of 200, and assuming the preset invocation events begin with a call to the `dataloader.next()` function and end with a call to the `optimizer.step()` function, the following explanation will be provided. If at least 200 invocation events have been recorded, but at least 10 event sequences starting with `dataloader.next()` and ending with `optimizer.step()` are not found among these events, it means that the currently monitored event sequences are insufficient to accurately represent a complete training iteration. In this case, the recorded invocation event timestamps need to be cleared, and monitoring of new invocation events needs to begin again.
[0090] If the number of recorded call events is greater than or equal to the second preset number, and there are no preset call events of the first preset number among the recorded call events, the monitoring and recording steps are re-executed to avoid incorrect iteration time calculations due to incomplete or irregular event sequences in the early or unstable phases of the training task, thereby ensuring the accuracy of performance monitoring and iteration time calculation.
[0091] In one embodiment of this disclosure, the iteration time satisfies a first preset condition, including: when the iteration time is determined based on the call times of a first preset number of preset call events, the average iteration time of a preset number of iteration training rounds is greater than a first target iteration time, wherein the preset number of iteration training rounds includes any one iteration training round and historical iteration training prior to any one iteration training round, and the first target iteration time is determined by the shortest iteration time among the average iteration times of the preset number of iteration training rounds; when the iteration time is determined based on the waiting time of the last call event, the iteration time is greater than a second target iteration time, wherein the second target iteration time is determined based on the average iteration time of historical iteration training.
[0092] The preset number of rounds in the above steps is a pre-set number of rounds, which can be set according to the actual application situation. There is no limit to the specific value of the preset number of rounds here.
[0093] The average iteration time of the preset number of iterations in the above steps is the average duration of training iterations within the preset number of iterations.
[0094] The first target iteration time in the above steps is a threshold used to determine whether the current training iteration is too slow. The first target iteration time is typically set based on the average iteration time of a preset number of iterations to identify performance degradation. For example, the first target iteration time can be set to 5% of the shortest iteration time among the average iteration times of a preset number of iterations; or it can be set to 7% of the shortest iteration time among the average iteration times of a preset number of iterations, but is not limited to these settings.
[0095] The second objective iteration time in the above steps is also a threshold used to determine whether training experiences prolonged stagnation or blockage. It is set based on the average iteration time of historical training iterations; for example, the second objective iteration time can be set to 5 times or 3 times the average iteration time of historical training iterations, but is not limited to these settings.
[0096] In one optional embodiment, if the listener detects a first preset number of preset invocation events, and if the average iteration time of the preset number of iterations is greater than the first target iteration time, it is determined that the training iteration time has generally become longer, exceeding the previously set threshold, and there is a possibility that some kind of resource bottleneck or code efficiency has been encountered.
[0097] In another alternative embodiment, if the listener fails to detect a first preset number of preset call events, and if the iteration time is greater than the second target iteration time, it is determined that the current iteration has encountered a serious blockage caused by data loading delay, communication bottleneck, or other hardware failure, and further processing is required.
[0098] In one embodiment of this disclosure, when an abnormality in the performance of a training task is detected, performance analysis data of the training task is obtained from at least one computing node. This includes: when an abnormality in the performance of the training task is detected, starting a daemon process on any computing node, and collecting node performance data of the computing node through the daemon process. The node performance data includes: execution data of multiple functions and hardware indicators of the computing node; and summarizing the node performance data of at least one computing node to obtain performance analysis data.
[0099] The daemon process mentioned above runs in the background of the computer system, providing services to the system or users. It listens for performance monitoring signals from the training task and, upon receiving a signal indicating abnormal performance, initiates a performance analysis data acquisition process.
[0100] The node performance data mentioned above is a collection of data related to the performance of the model training task on the computing node, including execution data of multiple functions and hardware metrics of the computing node. Function execution data involves detailed information about function calls during the training task, including but not limited to the number of calls, duration, and resource usage percentage. Hardware metrics include the status and performance of key hardware components on the computing node, including but not limited to GPU utilization, CPU utilization, and network bandwidth usage. Key hardware components include, but are not limited to, GPUs, CPUs, and network interfaces.
[0101] In one optional embodiment, when the performance monitoring module detects an anomaly in the performance of the training task, it sends a notification to the daemon process on the computing node via a TCP (Transmission Control Protocol) socket. Upon receiving the notification, the daemon process starts a pre-registered handle function to initiate the collection of performance data.
[0102] The daemon calls a performance analysis tool via the `handle` function. This tool collects function execution data and hardware metrics to obtain node performance data for the compute nodes. The node performance data collected from at least one compute node is then aggregated to produce performance analysis data.
[0103] In one embodiment of this disclosure, feature extraction is performed on performance analysis data to obtain the runtime behavior features of at least one objective function. This includes: using a daemon process on any computing node, determining at least one objective function from multiple functions based on the node performance data of that computing node in the performance analysis data; using the daemon process to determine target data in the performance analysis data that is related to any objective function; using the daemon process to extract features from the target data to obtain the node behavior features of any objective function on that computing node; and summarizing the node behavior features of any objective function on at least one computing node to obtain the runtime behavior features of any objective function.
[0104] The objective function in the above steps is the function that directly affects training performance in the model training task. The objective function is a function on the critical path, that is, a function that accounts for a large proportion of execution time or has a significant impact on hardware resource consumption. Objective functions include, but are not limited to, loading functions, model computation functions, optimizer update functions, ensemble communication functions, and hardware resource management functions.
[0105] The target data in the above steps refers to data directly related to the objective function within the performance analysis data, used to measure the behavior of the objective function. Target data includes, but is not limited to, the number of function calls, duration, and hardware resource usage.
[0106] In one optional embodiment, the extraction of function behavior features is performed through a daemon process for any work, i.e., any computing node. Functions involved in model training are categorized and assigned different priorities based on their impact on training performance. For example, GPU kernel functions have higher priority because they directly handle the computation in the training task; while Python (a programming language) functions or data loading functions have lower priority.
[0107] Based on function priority, the function with higher priority is identified as the target function. Then, from the performance analysis data, data directly related to the target function is filtered out, such as the target function's call records, execution time, and resource usage, thus obtaining the target data.
[0108] Then, feature extraction is performed on the target data through statistical analysis, such as the mean, standard deviation, and percentile of the function execution time, as well as the frequency of function calls and the volatility of resource consumption, so as to obtain the node behavior characteristics of the target function on the computing node.
[0109] Finally, the behavioral characteristics of nodes with the same objective function across all work nodes are summarized to obtain the runtime behavioral characteristics of that objective function.
[0110] In one embodiment of this disclosure, at least one target function is determined from multiple functions based on node performance data of the computing node in performance analysis data, including: determining the execution time of any function based on node performance data; determining the priority of any function based on the type of any function; and determining at least one target function from multiple functions based on priority and execution time, wherein the priority of the target function is greater than the priority of a first function executed at the same time, and the first function is any function other than the target function executed at the same time.
[0111] The execution time in the above steps is the time required for the function to complete execution from the start of the call.
[0112] The priorities in the above steps are determined based on the degree of influence of the functions on the performance of the training task. Setting priorities helps to identify key functions that affect the performance of the training task.
[0113] In one alternative embodiment, a performance analysis tool is used to collect function call records and event timestamps, thereby calculating the duration of any function from start to finish based on the function call records and event timestamps, and obtaining the function execution time.
[0114] When determining the priority of any function, the program predefines priority rules based on the function type and the function's impact on the training task performance, and then determines the priority of any function according to the predefined priority rules.
[0115] For example, Figure 3 is a schematic diagram of an optional function priority arrangement according to an embodiment of the present disclosure. As shown in Figure 3, the horizontal axis of the coordinate system represents the time axis of the training task, and the vertical axis represents the function priority. On the vertical axis, along the direction of the arrows, the priorities are arranged from largest to smallest. The broken lines in the coordinate system represent the critical path, and the functions on the critical path at different times on the time axis are shown as broken lines. It can be seen that regardless of the time on the time axis, the priority of the GPU computing kernel function is greater than that of the memory operation function, the priority of the memory operation function is greater than that of the set communication function, and the priority of the set communication function is greater than that of the Python function.
[0116] Based on the execution time and priority of the functions, select the functions located on the critical path, that is, functions that account for a large proportion of the execution time or have a significant impact on hardware resource consumption, as the target functions.
[0117] In one embodiment of this disclosure, feature extraction is performed on target data to obtain node behavior features of any target function on the computing node. This includes: determining the target time period and target hardware resources of the target function based on the target data, wherein the priority of the target function is higher than the priority of a second function executed within the target time period, and the second function is any function other than the target function executed within the target time period; obtaining a first behavior feature of the target function based on the proportion of the target time period within the node performance data collection time period; obtaining a second behavior feature of the target function based on the occupancy rate of the target hardware resources during the execution time of the target function; obtaining a third behavior feature of the target function based on the distribution of the occupancy rate of the target hardware resources during the execution time of the target function; and determining the first, second, and third behavior features as the node behavior features of the target function on the computing node.
[0118] The target time period in the above steps is the active period of the objective function throughout the entire training cycle or a certain round of iterative training, used to evaluate whether the objective function is part of the performance bottleneck.
[0119] The target hardware resources in the above steps are hardware components that directly determine the performance of the target function. For example, if the target function is a GPU computing kernel function, the corresponding target hardware resource is the GPU's computing unit; if it is a memory operation function, the corresponding target hardware resource is the GPU or CPU's memory; if it is a network communication function, the corresponding target hardware resource is the bandwidth of a PCIe or network interface card.
[0120] The time period for collecting node performance data in the above steps represents the total time period for collecting target data. This time period should be set according to the actual application; no specific value is limited here.
[0121] The utilization rate during the execution time of the objective function in the above steps refers to the degree of utilization of the corresponding target hardware resources during the execution of the objective function, usually expressed as a percentage. For example, the average utilization rate of GPU computing units when executing GPU computing kernel functions.
[0122] The distribution of the utilization rate during the execution time of the objective function in the above steps is a statistical characteristic of the utilization rate of the target hardware resources during the execution time of the objective function, such as standard deviation and variance, reflecting the volatility and stability of resource utilization.
[0123] The first behavioral feature in the above steps is used to characterize the ratio of the time when the objective function becomes a performance bottleneck to the total data acquisition time.
[0124] The second behavioral feature in the above steps is used to characterize the average utilization rate of the target hardware resources during the execution of the objective function.
[0125] The third behavioral feature in the above steps is used to characterize the volatility of the target hardware resource utilization rate during the execution of the objective function, and is measured using standard deviation.
[0126] In one optional embodiment, performance analysis data is analyzed to determine the active time periods of the objective function, which are then used as the target time periods. Next, based on the type of the objective function, the main hardware resources it depends on, such as GPU, CPU, and network, are determined.
[0127] Next, the operational behavior characteristics are extracted. The percentage of time the objective function becomes a performance bottleneck within the target time period is calculated, which is also the percentage of the target time period within the total data collection period, yielding the first behavioral characteristic. During the execution time of the objective function, the average utilization rate of the target hardware resources is measured, yielding the second behavioral characteristic. The standard deviation of the target hardware resource utilization rate is calculated, and its distribution is characterized by this standard deviation, yielding the third behavioral characteristic.
[0128] The formula for determining the node behavior characteristics of the objective function at this computation node based on the first, second, and third behavioral characteristics is as follows: P f,w =(β) f,w ,μ f,w ,σ f,w );
[0129] Among them, P f,w β represents the behavioral characteristics of the objective function. f,w As the first behavioral feature, μ f,w As the second behavioral characteristic, σ f,w The third row represents the feature, f is the objective function, and w is the computation node.
[0130] In one embodiment of this disclosure, determining the performance detection result of a training task based on the operational behavior characteristics of at least one objective function includes: determining a first difference metric of the objective function based on the difference between the operational behavior characteristics of the objective function and a first preset behavior characteristic; determining a second difference metric of the objective function based on the difference between the operational behavior characteristics of the objective function and a second preset behavior characteristic, wherein the second preset behavior characteristic is determined based on the operational behavior characteristics of the objective function in at least one computing node; and determining the performance detection result based on the first difference metric and the second difference metric.
[0131] The first preset behavioral characteristic in the above steps represents an expected or standard range for the operational behavior of the objective function, set based on past experience, benchmark tests, or a predefined normal range. The first preset behavioral characteristic reflects the performance level that the objective function should exhibit under normal circumstances.
[0132] The first difference metric in the above steps is used to quantify the degree of deviation between the operational behavior characteristics of the objective function and the first preset behavior characteristics. The first difference metric measures the extent to which the actual behavior characteristics of the objective function deviate from the expected normal range; it can be an absolute distance, a relative percentage deviation, or any other applicable statistical indicator to determine whether the objective function exceeds the limits of normal behavior. For example, if the execution time percentage of the objective function is significantly higher than the time percentage in the first preset behavior characteristics, then the first difference metric will be higher, indicating that the objective function has a performance problem.
[0133] The second pre-defined behavioral feature in the above steps is used to characterize the typical behavior of the objective function in a normal training environment without significant individual differences. It is constructed based on the average running behavior features of the objective function across multiple computing nodes. The second pre-defined behavioral feature is regarded as a consistency template for the expected performance of the objective function on different computing nodes, and is used to evaluate whether the behavior of the objective function is consistent across nodes or whether there are individual differences.
[0134] The second difference metric in the above steps is used to quantify the difference between the objective function's operational behavior characteristics on the computing nodes and the second preset behavior characteristics. The second difference metric identifies whether the objective function's performance is consistent across different computing nodes, thereby determining whether there are specific hardware or code implementation issues on the computing nodes. For example, if the mean and standard deviation of resource utilization for a certain objective function are similar on most computing nodes, but the utilization is significantly higher or fluctuates excessively on a certain computing node, then the second difference metric will highlight this, indicating an individual difference problem on the computing node.
[0135] In one optional embodiment, the operational behavior characteristics of the objective function are compared with first preset behavior characteristics, and a first difference metric is determined based on the difference between the operational behavior characteristics of the objective function and the first preset behavior characteristics by a distance-based function.
[0136] Here, the first preset behavioral characteristic is a predefined range. For example, an empirical range R is defined for the first preset behavioral characteristic. f R f The calculation formula is as follows:
[0137] Among them, R f To characterize the empirical range of the first preset behavioral features, and These are the upper and lower bounds of the first behavioral feature, respectively. and These are the upper and lower limits of the second behavioral feature, respectively. and These are the upper and lower limits of the third behavioral characteristic, respectively.
[0138] The formula for determining the first dissimilarity measure based on the distance metric function is as follows:
[0139] Among them, D f,w R is the first measure of difference. f Let P be the empirical range, and p be the objective function within the empirical range. f,w To define the behavioral characteristics of the objective function, d Manhattan () is the metric function for calculating Manhattan distance.
[0140] When determining the second difference metric, the operational behavior characteristics of the objective function on the computation nodes are normalized using the following formula:
[0141] in, w represents the characteristics of the normalized objective function's behavior on the computing node. ′ Let W be the set of computation nodes, and β be the set of computation nodes. f,w The first behavior of the objective function is a feature, μ f,w The second behavior of the objective function is a characteristic, σ f,w The third row of the objective function is a feature. Let the objective function be defined as the maximum value of the first row of features on at least one computation node. The objective function is defined as the maximum value of the second row of features at at least one computation node. The objective function is the maximum value of the third row of features at at least one computation node.
[0142] Then, the operational behavior characteristics of the objective function are compared with the second preset behavior characteristics, and the statistical distance between the operational behavior characteristics of the objective function and the second preset behavior characteristics is calculated. The formula for calculating the second difference measure is as follows:
[0143] Where, Δ f,w W is the second measure of difference. N w is a subset of size N randomly selected from at least one computing node. ′ For computing nodes, The characteristics of the normalized objective function's behavior on the computing nodes. The normalized objective function represents the behavioral characteristics of the reference computation node, where N is the value of W. N The number of nodes in the calculation, I is used for judgment. and The indicator function is whether the differences between them are significant. The reference computation node is any computation node running the objective function, other than the current computation node where the objective function resides.
[0144] Then, based on the first and second difference metrics, it is determined which objective functions exhibit abnormal performance on the computation nodes, thereby determining the performance test results.
[0145] In one embodiment of this disclosure, determining the performance test result based on a first difference metric and a second difference metric includes: determining whether the objective function meets a second preset condition based on the first difference metric, the second difference metric, and a first behavioral feature of the objective function, wherein the first behavioral feature is determined based on the proportion of the objective function's target time period within the node performance data collection time period, and the second preset condition is used to determine whether the objective function is a function exhibiting abnormal behavior; if the objective function meets the second preset condition, the objective function is determined to be a function exhibiting abnormal behavior; and the functions exhibiting abnormal behavior are summarized to obtain the performance test result.
[0146] The second preset condition in the above steps is pre-set and is used to determine whether the target function is a function exhibiting abnormal behavior. It is preset according to the actual application, and the specific content of the second preset condition is not limited here. For example, the second preset condition is characterized by a logical expression that determines whether the target function exhibits abnormal behavior, but it is not limited to this.
[0147] In one optional embodiment, the logical expression corresponding to the second preset condition is as follows: β f,w >0.01∧(D f,w >0∨Δ f,w >Mf +5MAD f );
[0148] Where, β f,w The first row of the objective function is a feature, D f,w The first measure of difference, Δ f,w It is the second measure of difference, M f It is the median of the second difference measure of the objective function on at least one computation node, MAD f For multiple second difference measures and M f The median of the absolute difference, 0.01, indicates that the objective function accounts for 1% of the time on the critical path.
[0149] If the first difference metric, the second difference metric, and the first behavioral feature of the objective function satisfy the logical expression corresponding to the second preset condition, then the objective function is determined to satisfy the second preset condition and is identified as a function exhibiting anomalous behavior. All objective functions satisfying the second preset condition are summarized to generate performance detection results. The performance detection results include a list of objective functions exhibiting anomalous behavior in the training task, as well as descriptions of the anomalous features of these objective functions.
[0150] In one embodiment of this disclosure, the objective function satisfies a second preset condition, including one of the following: a first behavioral feature is greater than a first preset threshold, and a first difference metric is greater than a second preset threshold; or a first behavioral feature is greater than a first preset threshold, and a second difference metric is greater than a third preset threshold, wherein the third preset threshold is determined based on the second difference metric of at least one objective function.
[0151] The first preset threshold in the above steps is a pre-set threshold, which is preset according to the actual application situation. The specific value of the first preset threshold is not limited here. For example, the first preset threshold is 0.01, but it is not limited to this, and other suitable values can be selected according to the actual situation.
[0152] The second preset threshold in the above steps is a pre-set threshold, preset according to the actual application situation. The specific value of the second preset threshold is not limited here. For example, the second preset threshold is 0, but it is not limited to this; other suitable values can be selected according to the actual situation.
[0153] The third preset threshold in the above steps is a pre-set threshold, which is preset according to the actual application situation. The specific value of the second preset threshold is not limited here.
[0154] In one optional embodiment, an example is given where the first preset threshold is 0.01 and the second preset threshold is 0. The second preset condition is that the first behavioral feature is greater than 0.01 and the first difference metric is greater than 0. Here, the first behavioral feature being greater than 0.01 is understood as the objective function accounting for more than 1% of the time on the critical path within the computation node. The objective function on the critical path is understood as the objective function having a significant proportion of execution time or a substantial impact on hardware resource usage.
[0155] In another optional embodiment, the third preset threshold is determined based on a second difference metric of at least one objective function. Exemplarily, one optional method for calculating the third preset threshold is as follows: Δ f,w ′=M f +5MAD f ;
[0156] Δ f,w ′ represents the third preset threshold, M f It is the median of the second difference measure of the objective function on at least one computation node, MAD f For multiple second difference measures and M f The median of the absolute differences.
[0157] The second preset condition is that the first behavioral feature is greater than the first preset threshold, and the second difference metric is greater than the third preset threshold.
[0158] The following description uses a preferred embodiment as an example. Figure 4 is a schematic diagram of an optional extraction of function execution behavior features according to an embodiment of this disclosure. As shown in Figure 4, for each computing node, online collection of computing node measurement and analysis data is performed, whereby the measurement and analysis data is also the node performance data mentioned above. The measurement and analysis data includes function execution records and hardware indicator sampling. The function execution records are equivalent to the execution data of multiple functions mentioned above, and the hardware indicator sampling is equivalent to the hardware indicators of the computing node mentioned above. The function execution records include, but are not limited to, the execution records of GPU computing functions, memory operation functions, set communication functions, and Python functions. The hardware indicator sampling includes, but is not limited to, the GPU's clock speed, GPU's computing unit occupancy, CPU's memory usage, network indicators, etc. Then, the measurement and analysis data collected from each computing node is summarized to obtain performance analysis data. Function execution behavior features are extracted from the performance analysis data to obtain function execution behavior features. The function execution behavior features include functions and their corresponding behavior features.
[0159] According to embodiments of this disclosure, another performance detection method for a model training task is also provided. Figure 5 is a flowchart of another performance detection method for a model training task according to embodiments of this disclosure. As shown in Figure 5, the method includes:
[0160] Step S502: In response to the model training request sent by the client, determine the target model and training task corresponding to the model training request;
[0161] Step S504: Execute a training task on the target model using at least one computing node and monitor the performance of the training task.
[0162] Step S506: When an abnormality is detected in the performance of the training task, the performance analysis data of the training task is obtained from at least one computing node, and the performance analysis data is used to extract features to obtain the running behavior features of at least one objective function. The at least one objective function is a function among multiple functions that is related to the performance of the training task.
[0163] Step S508: Perform feature extraction on the performance analysis data to obtain the running behavior features of at least one objective function. The at least one objective function is a function among multiple functions that is related to the performance of the training task.
[0164] Step S510: Based on the operational behavior characteristics of at least one objective function, determine the performance detection results of the training task. The performance detection results include: at least one objective function that exhibits abnormal behavior.
[0165] Step S512: Output the performance test results to the client.
[0166] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.
[0167] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, because according to this disclosure, some steps may be performed in a different order or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this disclosure.
[0168] Through the above description of the embodiments, those skilled in the art will clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, also by hardware. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, is embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (such as a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this disclosure.
[0169] According to embodiments of this disclosure, a system for implementing the performance detection method for the above-described model training task is also provided. Figure 6 is a schematic diagram of the performance detection system for the model training task according to an embodiment of this disclosure. As shown in Figure 6, the system includes:
[0170] At least one computing node 60 is configured to perform a training task on the target model.
[0171] The detection device 62, connected to at least one computing node, is configured to monitor whether the performance of the training task is abnormal. When an abnormality is detected in the performance of the training task, it obtains the performance analysis data of the training task from the at least one computing node, performs feature extraction on the performance analysis data to obtain the operational behavior features of at least one objective function, and determines the performance detection result of the training task based on the operational behavior features of at least one objective function. The performance analysis data is used to characterize data that has a correlation with multiple functions contained in the target model. The at least one objective function is a function that affects the performance of the training task among multiple functions. The performance detection result includes: at least one objective function that exhibits abnormal behavior.
[0172] It should be noted that the preferred embodiments involved in the above embodiments of this disclosure are the same as the solutions, application scenarios and implementation processes provided in the above embodiments, but are not limited to the solutions provided in the above embodiments.
[0173] Embodiments of this disclosure provide a computing device. FIG7 is a structural block diagram of a computing device according to an embodiment of this disclosure. As shown in FIG7, the computing device 70 includes: one or more (only one is shown in the figure) processors 702, memory 704, memory controller, and peripheral interfaces.
[0174] The aforementioned computing device is understood as an integrated smart terminal, including but not limited to servers, desktop computers, PCs (Personal Computers), all-in-one model machines, etc., and the computing device is pre-installed with the model described in the above embodiments of this disclosure.
[0175] Specifically, this computing device comes pre-installed with various types of models, including but not limited to models in natural language processing, visual processing, speech processing, code processing, and multimodal task processing, thus providing diverse model selection. In different product forms, this computing device supports one or more ways of using models, including but not limited to model training, model invocation, model fine-tuning, model deployment, model inference, and application. In some product forms, this computing device also supports model management, including but not limited to multi-type model management (supporting the management of discriminative, generative, and other types of models), model version control (supporting the control of different model versions), and model evaluation (evaluating model performance and effectiveness based on model evaluation tools). In other product forms, this computing device also provides API calling capabilities for creating applications based on models, allowing users to call models into created applications through API interfaces, while also providing application management tools to control and manage applications.
[0176] Furthermore, the computing device also includes data management (supporting the creation and management of model tuning datasets), a training center (providing abundant training resources to help users learn and master AI technology), and basic control capabilities (providing enterprise-level basic control capabilities to ensure the security and efficient operation of the system). Through the above functions, it provides a comprehensive and integrated device for AI development, training, deployment, and application.
[0177] The memory can be configured to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this disclosure. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the methods in the above embodiments. The memory may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories are connected to terminal A via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0178] The processor invokes an executable program stored in memory via a transmission device to execute the method described in any of the above embodiments.
[0179] Embodiments of this disclosure provide an electronic device. FIG8 is a structural block diagram of an electronic device according to an embodiment of this disclosure. As shown in FIG8, the electronic device includes: an input / output device 82; a memory 84; and a processor 86, wherein the processor 86 is connected to the input / output device 82 and the memory 84 via a bus 88.
[0180] The memory can be configured to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this disclosure. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the methods in the above embodiments. The memory may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories are connected to terminal A via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0181] The processor invokes an executable program stored in memory via a transmission device to execute the method described in any of the above embodiments.
[0182] Those skilled in the art will understand that the structure shown in the figure is merely illustrative, and the computing device can also be a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile internet device (MID), a tablet computer, or other terminal devices. This figure does not limit the structure of the aforementioned computing device. For example, the computing device 100 may include more or fewer components (such as a network interface, a display device, etc.) than shown in the figure, or may have a different configuration than shown in the figure.
[0183] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments are implemented by a program instructing the hardware related to the terminal device. The program is stored in a computer-readable storage medium, including flash drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0184] Embodiments of this disclosure also provide a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium is configured to store program code executed by the method provided in the above embodiments.
[0185] Optionally, in this embodiment, the storage medium is located in a computing device.
[0186] Optionally, in this embodiment, the computer-readable storage medium is configured to store an executable program, which, when the executable program is running, controls the device where the computer-readable storage medium is located to execute the method described in any of the above embodiments.
[0187] Embodiments of this disclosure also provide a computer program product. Optionally, in this embodiment, the computer program product includes a computer program that, when executed by a processor, implements the methods provided in the above embodiments.
[0188] Embodiments of this disclosure also provide a computer program product. Optionally, the computer program product includes a non-volatile computer-readable storage medium configured to store a computer program that, when executed by a processor, implements the methods provided in the embodiments described above.
[0189] Embodiments of this disclosure also provide a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, it implements the method provided in the above embodiments.
[0190] In the above embodiments of this disclosure, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0191] In the several embodiments provided in this disclosure, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as multiple units or components being combined or integrated into another system, or some features being ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed between units or modules is through some interface, indirect coupling or communication connection, and is electrical or other forms.
[0192] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units, meaning they may be located in one place or distributed across multiple network units. Some or all of these units are selected to achieve the purpose of this embodiment, depending on actual needs.
[0193] Furthermore, in the various embodiments of this disclosure, the functional units are integrated into one processing unit, or each unit exists as a separate physical entity, or two or more units are integrated into one unit. The integrated units described above are implemented either in hardware or as software functional units.
[0194] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it is stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, is embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media for storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.
[0195] The above description is only a preferred embodiment of this disclosure. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles of this disclosure, and these improvements and modifications should also be considered within the scope of protection of this disclosure.
Claims
1. A performance testing method for a model training task, comprising: During the process of performing a training task on the target model using at least one computing node, the performance of the training task is monitored. If an anomaly is detected in the performance of the training task, performance analysis data of the training task is obtained from the at least one computing node, wherein the performance analysis data is used to characterize data that has a correlation with multiple functions contained in the target model; Feature extraction is performed on the performance analysis data to obtain the operational behavior features of at least one objective function, wherein the at least one objective function is one of the plurality of functions that affects the performance of the training task; Based on the operational behavior characteristics of the at least one objective function, the performance detection result of the training task is determined, wherein the performance detection result includes: functions exhibiting abnormal behavior among the at least one objective function.
2. The method of claim 1, wherein, Monitoring the performance of the training task includes: During the execution of multiple rounds of iterative training in the training task using the at least one computing node, the iteration time of any one round of iterative training is determined. If the iteration time meets a first preset condition, it is determined that the performance of the training task is abnormal, wherein the first preset condition is used to determine whether the performance of the training task is abnormal. If the iteration time does not meet the first preset condition, it is determined that the performance of the training task is not abnormal.
3. The method of claim 2, wherein, Determining the iteration time for any round of training includes: Monitor the invocation events of the training task and record the invocation time of the invocation events; If there is a first preset number of preset call events among the recorded call events, the iteration time of any round of iterative training is determined based on the call time of the first preset number of preset call events; If the first preset number of preset call events are not present in the recorded call events, the iteration time for any round of iterative training is determined based on the waiting time of the last call event.
4. The method of claim 3, wherein, The method further includes: If the number of recorded invocation events is greater than or equal to the second preset number, and there is no preset number of invocation events among the recorded invocation events, delete the invocation time of the recorded invocation events, and re-execute the steps of monitoring the invocation events of the training task and recording the invocation time of the invocation events.
5. The method of claim 2, wherein, The iteration time satisfies a first preset condition, including: When the iteration time is determined based on the invocation time of a first preset number of preset invocation events, the average iteration time of the preset number of iteration training rounds is greater than the first target iteration time. The preset number of iteration training rounds includes any iteration training round and historical iteration training before any iteration training round. The first target iteration time is determined based on the shortest iteration time among the iteration times of the preset number of iteration training rounds. When the iteration time is determined based on the waiting time of the last invocation event, the iteration time is greater than the second target iteration time, wherein the second target iteration time is determined based on the average iteration time of the historical iteration training.
6. The method of any one of claims 1 to 5, wherein, When an anomaly is detected in the performance of the training task, the step of obtaining performance analysis data of the training task from the at least one computing node includes: If an abnormality is detected in the performance of the training task, a daemon process on any computing node is started, and the node performance data of the computing node is collected through the daemon process. The node performance data includes: the execution data of the multiple functions and the hardware indicators of the computing node. The performance data of the at least one computing node is summarized to obtain the performance analysis data.
7. The method of claim 6, wherein, The process of collecting node performance data of the computing node through the daemon process includes: Based on the daemon process, a preset function is started, wherein the preset function is used to start the collection of the node performance data; Based on the preset function, the performance analysis tool is invoked; Based on the performance analysis tool, the execution data and hardware metrics are collected; Based on the execution data and the hardware metrics, the node performance data is constructed.
8. The method of any one of claims 1 to 5, wherein, The step of extracting features from the performance analysis data to obtain at least one behavioral feature of the objective function includes: Using a daemon process on any computing node, at least one objective function is determined from the plurality of functions based on the node performance data of that computing node in the performance analysis data; The daemon process determines the target data in the performance analysis data that is related to any objective function. The daemon process extracts features from the target data to obtain the node behavior features of any objective function on the computing node. The node behavior characteristics of any objective function on the at least one computing node are summarized to obtain the runtime behavior characteristics of any objective function.
9. The method of claim 8, wherein, The process of determining the at least one objective function from the plurality of functions based on the node performance data of the computing node in the performance analysis data includes: Based on the node performance data, determine the execution time of any function; The priority of any function can be determined based on its type. Based on the priority and the execution time, at least one target function is determined from the plurality of functions, wherein the priority of the target function is greater than the priority of a first function executed at the same time, and the first function is any function other than the target function executed at the same time.
10. The method of claim 8, wherein, The step of extracting features from the target data to obtain node behavior features of any objective function on the computation node includes: Based on the target data, the target time period and target hardware resources of the target function are determined, wherein the priority of the target function is higher than the priority of the second function executed within the target time period, and the second function is any function other than the target function executed within the target time period; Based on the proportion of the target time period within the data collection period of the node performance data, the first behavioral feature of the objective function is obtained; Based on the occupancy rate of the target hardware resources during the execution time of the target function, the second behavioral characteristics of the target function are obtained; Based on the distribution of the occupancy rate of the target hardware resources during the execution time of the target function, the third behavioral feature of the target function is obtained; The first behavioral feature, the second behavioral feature, and the third behavioral feature are determined as the node behavioral features of the objective function on the computing node.
11. The method of claim 10, wherein, The step of determining the target time period and target hardware resources for the objective function based on the target data includes: The performance analysis data is analyzed to obtain the active time period of the objective function, and the active time period is determined as the target time period; The target hardware resources are determined based on the type of the objective function and the correspondence between the type and hardware resources.
12. The method of any one of claims 1 to 5, wherein, Determining the performance detection result of the training task based on the operational behavior characteristics of the at least one objective function includes: Based on the difference between the operational behavior characteristics of the objective function and the first preset behavior characteristics, a first difference metric value of the objective function is determined; Based on the difference between the operational behavior characteristics of the objective function and the second preset behavior characteristics, a second difference metric value of the objective function is determined, wherein the second preset behavior characteristics are determined based on the operational behavior characteristics of the objective function in the at least one computing node; The performance test result is determined based on the first difference metric and the second difference metric.
13. The method of claim 12, wherein, The determination of a second difference metric for the objective function based on the difference between the operational behavior characteristics of the objective function and a second preset behavior characteristic includes: The objective function is normalized to obtain the normalized objective function; Determine the running behavior characteristics of the normalized objective function on any computing node, and the running behavior characteristics of the normalized objective function on a reference computing node, wherein the reference computing node is any computing node other than the at least one computing node that runs the objective function. The difference between the normalized objective function's operating behavior characteristics on any computing node and the difference between the normalized objective function's operating behavior characteristics on the reference computing node is used to obtain the difference metric value corresponding to any computing node. The second difference metric is constructed based on the difference metric corresponding to the at least one computing node.
14. The method of claim 12, wherein, The step of determining the performance test result based on the first difference metric and the second difference metric includes: Based on the first difference metric, the second difference metric, and the first behavioral feature of the objective function, it is determined whether the objective function meets the second preset condition. The first behavioral feature is determined based on the proportion of the objective function's target time period within the node performance data collection time period. The second preset condition is used to determine whether the objective function is a function exhibiting abnormal behavior. If the objective function satisfies the second preset condition, the objective function is determined to be a function exhibiting abnormal behavior. The performance test results are obtained by summarizing the functions that exhibited abnormal behavior.
15. The method of claim 14, wherein, The objective function satisfies the second preset condition, including one of the following: The first behavioral feature is greater than a first preset threshold, and the first difference metric is greater than a second preset threshold; The first behavioral feature is greater than the first preset threshold, and the second difference metric is greater than the third preset threshold, wherein the third preset threshold is determined based on the second difference metric of the at least one objective function.
16. A performance testing method for a model training task, comprising: In response to a model training request sent by a client, determine the target model and training task corresponding to the model training request; The training task is performed on the target model using at least one computing node, and the performance of the training task is monitored. If an anomaly is detected in the performance of the training task, performance analysis data of the training task is obtained from the at least one computing node, wherein the performance analysis data is used to characterize data that has a correlation with multiple functions contained in the target model; Feature extraction is performed on the performance analysis data to obtain the operational behavior features of at least one objective function, wherein the at least one objective function is one of the plurality of functions that affects the performance of the training task; Based on the operational behavior characteristics of the at least one objective function, the performance detection result of the training task is determined, wherein the performance detection result includes: functions exhibiting abnormal behavior among the at least one objective function; The performance test results are output to the client.
17. A performance testing system for a model training task, comprising: At least one computing node is configured to perform a training task on the target model; A detection device, connected to the at least one computing node, is configured to monitor the performance of the training task. When an abnormality is detected in the performance of the training task, it acquires performance analysis data of the training task from the at least one computing node, extracts features from the performance analysis data to obtain the operational behavior features of at least one objective function, and determines the performance detection result of the training task based on the operational behavior features of the at least one objective function. The performance analysis data is used to characterize data that has a correlation with multiple functions included in the target model. The at least one objective function is a function among the multiple functions that affects the performance of the training task. The performance detection result includes: functions among the at least one objective function that exhibit abnormal behavior.
18. A computing device, comprising: Memory, which stores executable programs; A processor is configured to run the program, wherein the program, when running, performs the method according to any one of claims 1 to 16.
19. An electronic device comprising: Memory, which stores executable programs; A processor, connected to the memory via a bus, is configured to run the program, wherein the program, when running, performs the method according to any one of claims 1 to 16.
20. A computer readable storage medium comprising a stored executable program, wherein, When the executable program is executed, it controls the device containing the computer-readable storage medium to perform the method according to any one of claims 1 to 16.
