Diffusion model inference method and related apparatus

By obtaining the first path and time step weight coefficients of the diffusion model, and using a scheduler and a noise prediction network for denoising, the problem of high inference load in the diffusion model is solved, resulting in faster data generation speed and a better user experience.

WO2026237889A1PCT designated stage Publication Date: 2026-11-19HUAWEI TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2026/096186
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-13
Filing Date
2026-05-11
Publication Date
2026-11-19

AI Technical Summary

Technical Problem

The diffusion model has a large computational and memory access requirements during the inference phase, which increases the latency of image or video generation and limits its practicality in application scenarios.

Method used

By obtaining the first path and weight coefficients of each time step of the diffusion model, denoising is performed using the first scheduler and noise prediction network to generate first output data that meets the preset data accuracy and error threshold, thereby reducing the number of sampling steps and denoising iterations.

Benefits of technology

While ensuring the accuracy of the output data, the inference load of the diffusion model is reduced, the generation speed and user experience are improved, and the practicality of the model is enhanced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2026096186_19112026_PF_FP_ABST
    Figure CN2026096186_19112026_PF_FP_ABST
Patent Text Reader

Abstract

A diffusion model inference method and a related apparatus, which relate to the field of artificial intelligence. The method comprises: acquiring a first path of a diffusion model and a weight coefficient for each time step in the first path; and performing denoising on the basis of the first path and the weight coefficient for each time step in the first path by means of a first scheduler and a noise prediction network of the diffusion model, so as to obtain first output data of the diffusion model. In the embodiments of the present application, fitting can be performed on the basis of a second path of a diffusion model, so as to generate a first path, wherein the first path is a sampling path that meets a preset data precision and has a fewer number of time steps than that in the second path, and / or the first path is a sampling path that has a preset number of time steps, where the error at each time step is less than a first error threshold value, and the preset number of time steps is less than the number of time steps in the second path. The present solution can effectively reduce the number of sampling steps of the diffusion model and the number of denoising iterations while ensuring the precision of output data, thereby achieving inference acceleration for the diffusion model.
Need to check novelty before this filing date? Find Prior Art

Description

Diffusion Model Inference Methods and Related Devices

[0001] This application claims priority to Chinese Patent Application No. 202510612958.4, filed on May 13, 2025, entitled "Diffusion Model Reasoning Method and Related Apparatus", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence, and in particular to a diffusion model reasoning method and related apparatus. Background Technology

[0003] Multimodal large models, by integrating and processing data from different sources (such as text, images, and audio), provide powerful content creation and understanding capabilities for Artificial Intelligence Generated Content (AIGC), pushing the boundaries of creative and personalized content generation.

[0004] Among them, the diffusion model (DM) performs well in visual content generation, but it requires multiple forward computations during the inference phase, significantly increasing the computational and memory usage of the model's inference, leading to increased latency in image or video generation and limiting its practicality in application scenarios. Therefore, reducing the inference load of large multimodal models and improving the speed of visual content generation are crucial for enhancing the model's practicality and user satisfaction. Summary of the Invention

[0005] This application provides a diffusion model inference method and related apparatus, which can solve the problem of reducing the inference load of diffusion models.

[0006] In a first aspect, embodiments of this application provide a diffusion model inference method, which includes: obtaining a first path of the diffusion model and weight coefficients of each time step in the first path; and performing denoising based on the first path and the weight coefficients of each time step in the first path using a first scheduler and a noise prediction network of the diffusion model to obtain first output data of the diffusion model. The first path is a sampling path that satisfies a preset data accuracy, obtained based on a second path of the diffusion model, and / or a sampling path obtained based on the second path with a preset number of time steps and an error of less than a first error threshold at each time step. The second path is the denoising process path of the diffusion model, which consists of several time steps. The number of time steps in the first path is less than the number of time steps in the second path.

[0007] In this embodiment, the second path based on the diffusion model can be fitted to generate a first path. This first path is a sampling path that meets a preset data accuracy and has fewer time steps than the second path, and / or, the first path is a sampling path with a preset number of time steps and an error at each time step less than a first error threshold, where the preset number of time steps is less than the second path's time steps. This solution can effectively reduce the number of sampling steps in the diffusion model while ensuring the accuracy of the output data, reducing the number of denoising iterations, reducing computational load, accelerating diffusion model inference, and solving the problem of reducing the inference load of the diffusion model. Furthermore, this embodiment can generate a first path that meets specified requirements according to the user's actual needs, improving the practicality of the diffusion model and enhancing the user experience.

[0008] Optionally, the error of the aforementioned time step refers to the error between the output latent variable of that time step and the output latent variable of the second path at the time step corresponding to that time step.

[0009] Optionally, the weight coefficients of the above time steps refer to the weights used in denoising, i.e., the projection matrix.

[0010] In one possible implementation, the above-mentioned acquisition of the first path of the diffusion model specifically includes the following steps: S1, fitting a fitted path from the second path based on a second error threshold, wherein the error at each time step in the fitted path is less than the second error threshold. S2, if the number of time steps in the fitted path is not equal to the preset number of time steps, then adjusting the second error threshold and re-executing step S1 until the number of time steps in the fitted path is equal to the preset number of time steps.

[0011] In this embodiment, a fitted path is first obtained from the second path based on a second error threshold. Then, it is determined whether the fitted path meets the specified time step requirement. If not, the second error threshold is adjusted, and a new fitted path is obtained based on the adjusted second error threshold. This process continues until the time step count of the fitted path equals the preset time step count. The fitted path at this point is the first path, and the second error threshold at this point is the aforementioned first error threshold. Since the preset time step count is less than the time step count of the second path, this embodiment can effectively reduce the sampling steps of the diffusion model while ensuring the accuracy of the output data, thereby reducing the number of denoising iterations, decreasing computational load, and accelerating the inference of the diffusion model.

[0012] Optionally, the second error threshold can be a large threshold, and it is continuously reduced when acquiring the first path until the fitted path meets the specified number of time steps. Alternatively, the second error threshold can be a small threshold, and it is continuously increased when acquiring the first path until the fitted path meets the specified number of time steps.

[0013] Optionally, the specific value of the second error threshold can be set according to the actual situation and is not limited.

[0014] In one possible implementation, the first scheduler and noise prediction network of the diffusion model are used to denoise based on the first path and the weight coefficients of each time step in the first path to obtain the first output data of the diffusion model. Specifically, this includes the following steps: obtaining the target noise used for denoising by the diffusion model; repeating the following denoising steps until the number of executions reaches N, where N is a preset number of time steps; and obtaining the first output data based on the output latent variables of the Nth denoising iteration: generating the predicted noise e at the current time step t through the noise prediction network. t Through the first scheduler, based on e t The output latent variable x of the previous time step at the current time step t+1 x t+1 The corresponding weighting coefficient w x,t and e t weighting coefficient w e,t Obtain the output latent variable x at the current time step t In the initial denoising process, t+1 is N, and x... t+1 The target noise.

[0015] In this embodiment, denoising is performed based on the first path and the weight coefficients of each time step in the first path, which can quickly obtain the first output data. The denoising coefficients use a projection matrix (weight coefficients), which only requires first-order linear calculation to denoise, and can ensure the quality of the first output data.

[0016] In one possible implementation, when the target noise is a latent space variable, the first output data is obtained based on the output latent variable of the Nth denoising, specifically including the following steps: performing latent space variable decoding processing on the output latent variable of the Nth denoising to obtain the first output data.

[0017] In this embodiment, when the target noise is a latent space variable, after iterative denoising, the latent variable output by the Nth denoising iteration needs to undergo latent space variable decoding to obtain the first output data. Executing the method of this embodiment in the latent space can reduce computational complexity, increase computational speed, and improve the inference performance of the diffusion model.

[0018] In one possible implementation, the above-mentioned generation of predicted noise for the current time step through a noise prediction network specifically includes the following steps: generating predicted noise based on data generation conditions through a noise prediction network, wherein the data generation conditions are related to the first output data.

[0019] In this embodiment, the predicted noise at each time step in the first path is obtained based on the data generation conditions, which in turn affects the denoising process of the diffusion model, resulting in the first output data related to the data generation conditions. The diffusion model can achieve condition generation.

[0020] Optionally, the data generation conditions include at least one of the following types of data: image data, video data, text data, or voice data.

[0021] In one possible implementation, the above diffusion model inference method further includes the following step: denoising the diffusion model using a second scheduler to obtain a second path and second output data of the diffusion model. The first and second output data are of the same type. The second scheduler is a packaged scheduler.

[0022] The encapsulated scheduler can be understood as a scheduler that abstracts complex sampling logic into a simple interface, and can output parameters such as time step, prediction noise, and output latent variables through the interface.

[0023] In this embodiment, the diffusion model is denoised using a second scheduler to obtain a second path and second output data. The first and second output data are of the same type; in other words, they are obtained using a diffusion model that performs the same task. The first path can then be learned and fitted based on the second path.

[0024] In one possible implementation, the above diffusion model inference method further includes the following steps: acquiring and saving path data of the second path through a second scheduler. This path data includes the output latent variables and prediction noise at each time step. The acquisition of the weight coefficients for each time step in the first path includes: obtaining the weight coefficients for the third time step in the first path based on the output latent variables and prediction noise of the first time step, and the output latent variables and prediction noise of the second time step. The first and second time steps are two time steps in the second path corresponding to the initial state and the target state of the third time step. The third time step can be any time step in the first path.

[0025] In this embodiment, the encapsulated second scheduler can be used to obtain path data such as output latent variables and prediction noise at each time step in the second path, and this path data can be saved for subsequent fitting. Furthermore, based on the path data of the second path, the weight coefficients of the first path and its corresponding time steps can be calculated.

[0026] In one possible implementation, to avoid random errors in the path data of the second path, the diffusion model is denoised multiple times by a second scheduler to obtain multiple sets of path data for the second path. Based on the multiple sets of path data for the second path, the weight coefficients of the first path and its various time steps can be obtained.

[0027] In one possible implementation, the second scheduler can be any scheduler, such as a Denoising Diffusion Implicit Models (DDIM) scheduler, a Denoising Diffusion Probabilistic Model (DDPM) scheduler, or a Pseudo Numerical Methods for Diffusion Models (PNDM) scheduler.

[0028] Secondly, embodiments of this application provide a diffusion model inference apparatus, the apparatus including a unit for performing the method as described in any of the first aspects.

[0029] In one possible design, the device includes:

[0030] The acquisition unit is used to acquire the first path of the diffusion model and the weight coefficients of each time step in the first path.

[0031] The inference unit is used to denoise the first path and the weight coefficients of each time step in the first path through the first scheduler and the noise prediction network of the diffusion model to obtain the first output data of the diffusion model.

[0032] The first path is a sampling path that meets a preset data accuracy, obtained from the second path based on the diffusion model, and / or a sampling path obtained from the second path with a preset number of time steps and an error less than a first error threshold at each time step. The second path is the denoising process path of the diffusion model, and the number of time steps in the first path is less than the number of time steps in the second path.

[0033] The steps performed by the acquisition unit and the reasoning unit described in the second aspect and any possible implementation can be referred to the corresponding implementations in the first aspect.

[0034] For the technical effects of the second aspect and any possible implementation, please refer to the description of the technical effects corresponding to the first aspect and the corresponding implementation.

[0035] Optionally, in the diffusion model inference apparatus described in the second aspect above and any possible implementation thereof:

[0036] In one implementation, the diffusion model inference device is a diffusion model inference apparatus. When the diffusion model inference device is a diffusion model inference apparatus, the acquisition unit can be a transceiver or an input / output interface; the inference unit can be at least one processor. Optionally, the transceiver can be a transceiver circuit. Optionally, the input / output interface can be an input / output circuit.

[0037] In another implementation, the diffusion model inference device is a chip (system) or circuit used in a diffusion model inference device. When the diffusion model inference device is a chip (system) or circuit used in a diffusion model inference device, the acquisition unit can be a communication interface (input / output interface), interface circuit, output circuit, input circuit, pin, or related circuit on the chip (system) or circuit; the inference unit can be at least one processor, processing circuit, or logic circuit.

[0038] Thirdly, embodiments of this application provide a diffusion model inference apparatus, which includes a processor. The processor is coupled to a memory and can be used to execute instructions in the memory to implement the methods described in the first aspect and any of the possible implementations. Optionally, the diffusion model inference apparatus further includes a memory. Optionally, the diffusion model inference apparatus further includes a communication interface, and the processor is coupled to the communication interface.

[0039] Fourthly, embodiments of this application provide a chip, including: logic circuitry and a communication interface. The communication interface is used to receive or send information; the logic circuitry is used to receive or send information through the communication interface, causing the chip to execute the methods described in the first aspect and any of the possible implementations.

[0040] Fifthly, embodiments of this application provide a computer-readable storage medium for storing a computer program (also referred to as code or instructions); when the computer program is run on a computer, the methods described in the first aspect and any possible implementation are implemented.

[0041] Sixthly, embodiments of this application provide a computer program product, the computer program product comprising: a computer program (also referred to as code or instructions); and, when the computer program is run, causing a computer to perform the methods described in the first aspect and any possible implementation thereof.

[0042] Furthermore, in the process of performing the method described in the first aspect and any possible implementation above, the processes related to sending and / or receiving information in the above methods can be understood as the process of the processor outputting information, and / or the process of the processor receiving input information. When outputting information, the processor can output the information to a transceiver (or communication interface, or transmitting module) so that the transceiver can transmit it. After the information is output by the processor, it may need to undergo other processing before reaching the transceiver. Similarly, when the processor receives input information, the transceiver (or communication interface, or transmitting module) receives the information and inputs it to the processor. Furthermore, after the transceiver receives the information, the information may need to undergo other processing before being input to the processor.

[0043] Based on the above principles, for example, the information sent mentioned in the aforementioned method can be understood as information output by the processor. Similarly, the information received can be understood as information received by the processor from input.

[0044] Optionally, unless otherwise specified, or unless they contradict their actual function or internal logic in the relevant description, the operations of the processor, such as transmitting, sending, and receiving, can be more generally understood as processor output and receiving, input, and other operations.

[0045] Optionally, in performing the methods described in the first aspect and any possible implementation above, the processor may be a processor specifically designed to perform these methods, or it may be a processor that performs these methods by executing computer instructions stored in memory, such as a general-purpose processor. The memory may be a non-transitory memory, such as read-only memory (ROM), which may be integrated with the processor on the same chip or disposed on separate chips. This application does not limit the type of memory or the arrangement of the memory and processor.

[0046] In one possible implementation, at least one of the aforementioned memories is located outside the device.

[0047] In yet another possible implementation, at least one of the aforementioned memories is located within the device.

[0048] In another possible implementation, a portion of the memory of the at least one memory is located inside the device, while another portion is located outside the device.

[0049] In this application, the processor and memory may also be integrated into a single device, that is, the processor and memory can be integrated together. Attached Figure Description

[0050] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 is a schematic diagram of a system architecture provided in an embodiment of this application;

[0052] Figure 2 is a flowchart illustrating a diffusion model inference method provided in an embodiment of this application;

[0053] Figure 3A is a schematic diagram of a diffusion model inference method provided in an embodiment of this application;

[0054] Figure 3B is a schematic diagram of an information input interface provided in an embodiment of this application;

[0055] Figure 4A is a schematic flowchart of a diffusion model inference method provided in an embodiment of this application;

[0056] Figure 4B is a schematic flowchart of another diffusion model inference method provided in an embodiment of this application;

[0057] Figure 5 is a denoising diagram provided in an embodiment of this application;

[0058] Figure 6 is a schematic diagram of a diffusion model inference device provided in an embodiment of this application;

[0059] Figure 7 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0060] Figure 8 is a schematic diagram of the structure of a chip provided in an embodiment of this application. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described below with reference to the accompanying drawings.

[0062] The terms "first" and "second," etc., used in the specification, claims, and drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0063] The term "embodiment" as used herein means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that, unless otherwise specified or logically conflicting, the terminology and / or descriptions between the various embodiments of this application are consistent and can be mutually referenced, and technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships.

[0064] It should be understood that in this application, "at least one (item)" means one or more, "more than one" means two or more, "at least two (items)" means two or three or more, and "and / or" is used to describe the relationship between related objects, indicating that there can be three relationships. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the related objects before and after are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0065] This application provides a diffusion model inference method and related apparatus, relating to the field of artificial intelligence. To more clearly describe the solution of this application, some terms used in the embodiments of this application will be explained below.

[0066] Diffusion Model (DM): A generative model based on probability distribution, which generates high-quality data (such as images, audio, etc.) from random noise through a stepwise "denoising" process. Its core idea is to simulate the gradual evolution of data from noise to real samples, which can be divided into forward diffusion and backward generation processes.

[0067] Forward diffusion: A process that gradually transforms data into random noise. It is usually achieved by gradually increasing the noise in the data through multiple steps until the data becomes completely random noise.

[0068] The reverse generation process, also known as the reverse diffusion process, reverse denoising process, or sampling process, is the inverse operation of the diffusion process. It starts with pure random noise and gradually removes the noise through a series of iterative steps, ultimately reconstructing clear data.

[0069] Time Steps (T): These are key parameters that control the forward diffusion and backward denoising processes, and their determination directly affects the generation quality and efficiency of the model.

[0070] During the forward diffusion process, noise is gradually added to the real data until it becomes pure noise. The time step determines the noise intensity at each step.

[0071] In the reverse denoising process, noise is removed gradually, starting from pure noise, to recover the data. Each time step corresponds to a step in the denoising process.

[0072] Time steps (quantity): refers to the number of steps in forward diffusion or backward denoising.

[0073] Scheduler: In the inference phase of the diffusion model, the scheduler plays a crucial role, responsible for controlling the addition and removal of noise. It directly affects the efficiency of the denoising process and the generation quality.

[0074] For example, the scheduler defines the noise intensity (or denoising step size) for each time step, which determines the smoothness and stability of the denoising process. During the inference phase, the model needs to gradually recover data from pure noise based on the noise intensity defined by the scheduler.

[0075] The step size chosen by the scheduler directly affects inference efficiency. For example, the DDIM scheduler significantly speeds up inference by reducing the step size.

[0076] Latent variables: These are the direct objects of manipulation in the diffusion / denoising process. That is, the state data corresponding to each time step in the diffusion / denoising process can be called latent variables.

[0077] Currently, AI-generated content includes text-to-image or text-to-video conversion. By transforming textual descriptions into visual images or videos, it expands the ways humans can express their creativity, bringing revolutionary changes to fields such as art, education, and entertainment, while also promoting the development of human-computer interaction and the digital content industry.

[0078] With advancements in natural language processing and computer vision technologies, text-based image and video generation techniques have achieved significant breakthroughs, encompassing various models such as Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), and diffusion models. Currently, Data Generation (DM) has become the mainstream paradigm for text-based image and video generation due to its ability to effectively generate high-quality, high-resolution images or videos from text descriptions, while also possessing strong versatility and controllability.

[0079] However, while diffusion models generate data through multi-step iterative denoising, their core bottleneck lies in sampling efficiency. The following are two solutions to improve the sampling efficiency of diffusion models:

[0080] The first approach is to optimize the sampler method: by optimizing the mathematical methods of the sampling process, the number of steps required to generate samples can be significantly reduced while maintaining the quality of generation. The core idea is to use higher-order adaptive solution techniques to more efficiently approximate the reverse stochastic differential equation (SDE) or probability flow ordinary differential equation (ODE) of the diffusion process, thereby accelerating convergence. Higher-order Taylor expansions (such as second- or third-order) are used to approximate the inverse process of the diffusion model. Compared to traditional first-order methods (such as DDIM), higher-order methods can more accurately predict the changes in the solution at each sampling step, thus approximating the target distribution faster at the same number of steps.

[0081] Scheme 1 has the following drawbacks: When the number of sampling steps is extremely low (e.g., <10 steps), the accumulation of errors in higher-order methods may lead to a significant decrease in generated quality (e.g., blurred details or structural distortion). For higher-order methods, the gradient computation cost increases at each step, resulting in a longer single-step time. If the hardware parallelism capability is insufficient, the overall time savings may be limited. Higher-order methods rely on multi-step historical information, and storing this historical information increases memory usage, which is not conducive to generating high-resolution images.

[0082] Option two is sampler distillation: the core idea is to compress the multi-step denoising process into fewer steps. By training a student model to mimic the multi-step behavior of the teacher model, the computational load during inference is reduced. Sampler distillation enables efficient inference, significantly reducing the number of sampling steps (e.g., from 1000 steps to 4-8 steps) while maintaining the quality of the generated image, producing a quality close to the original model within a reasonable compression range. For example, LCM-Lora performs consistent distillation in the latent space, generating high-quality images in 1-4 steps. Combined with techniques such as quantization, it can effectively reduce memory usage and computational requirements.

[0083] Scheme 2 has the following drawbacks: the performance of the sampler distillation model is highly dependent on the quality of the teacher model; if the teacher model has biases, the student model is difficult to correct. Furthermore, there are limitations in generalization; the distilled diffusion model may overfit to specific data distributions, resulting in decreased performance in new domains. For extremely low-step distillation sampling (e.g., 1-2 steps), the number of parameters increases explosively, placing high demands on memory size and computational performance.

[0084] In summary, diffusion models mainly face the following challenges in data generation tasks:

[0085] Slow inference speed: Traditional diffusion models require stepwise denoising, a process that requires multiple iterations and is sequential, resulting in a heavy inference load and limiting the model's availability in real-time or near-real-time applications.

[0086] Computational redundancy: In the process of generating data, the diffusion model involves a large number of repetitive inter-layer calculations, which produce similar outputs at different time steps.

[0087] Low-step sampling incurs high training and computational costs: sampler optimization methods require repeated model training, resulting in high computational overhead and unstable training. Low-step sampling may increase the computational cost per step, offsetting the benefits of reducing the number of steps.

[0088] Trade-off between data quality and speed: While reducing iteration steps to speed up the generation process, a trade-off often needs to be made between data quality and generation speed.

[0089] Therefore, a new method is needed to reduce the number of sampling steps in the diffusion model.

[0090] In view of this, embodiments of this application provide a diffusion model inference method and related apparatus, relating to the field of artificial intelligence technology, which can reduce the number of sampling steps and solve the problem of reducing the inference load of diffusion models.

[0091] The following describes a system architecture provided by an embodiment of this application.

[0092] Referring to Figure 1, which is a schematic diagram of a system architecture provided in an embodiment of this application; this embodiment of the application provides a system architecture 100. As shown in the system architecture 100, the data acquisition device 160 is used to acquire training data for the diffusion model, and the training data can be image data, text data, or video data, etc. The data acquisition device 160 stores the training data in the database 130.

[0093] The training device 120 can obtain the diffusion model 101 based on the training data maintained in the database 130. Exemplarily, in this embodiment, after model training is completed, the diffusion model 101 can acquire the ability to generate data. The training device 120 can be a server or cloud service device, or it can be a mobile terminal, desktop computer, tablet computer, laptop computer, or other similar device.

[0094] It should be noted that in practical applications, the training data maintained in the database 130 may not all come from the data acquisition device 160; it may also be received from other devices. Furthermore, it should be noted that the training device 120 may not necessarily train the model entirely based on the training data maintained in the database 130; it may also obtain training data from the cloud or other sources for model training. The above description should not be construed as limiting the embodiments of this application.

[0095] The diffusion model 101 obtained after processing by the training device 120 can be applied to a system or device with the same hardware configuration as the training device 120, such as the electronic device 110 shown in FIG. 1. The electronic device 110 can execute the methods of the embodiments of this application based on the diffusion model 101. The electronic device 110 can be a terminal, such as a mobile terminal, desktop computer, tablet computer, laptop computer, etc., or it can be a server or cloud, etc. The training device 120 and the electronic device 110 can be the same device.

[0096] In Figure 1, the electronic device 110 is equipped with an I / O interface 112 for data interaction with external devices. Users can input data into the I / O interface 112 through the client device 140. In this embodiment, the input data can be data generation conditions; for example, the input data can be prompt text entered by the user. The input data can also be obtained from the database 130.

[0097] Optionally, the preprocessing module 113 is used to preprocess the input data received by the I / O interface 112. In this embodiment, the preprocessing module 113 is used to preprocess the input data received by the I / O interface 112, and the preprocessed data enters the calculation module 111.

[0098] During the preprocessing of input data by electronic device 110, or the calculation module 111 of electronic device 110 performing calculations and other related processes, electronic device 110 can call data, code, etc. in data storage system 150 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into data storage system 150.

[0099] Finally, exemplarily, the I / O interface 112 returns the processing result of the electronic device 110 (e.g., the model result of the diffusion model 101) to the client device 140, thereby providing it to the user. In this case, the client device 140 can be a display. In this embodiment, the electronic device 110 executes the diffusion model inference method of this embodiment.

[0100] In the scenario shown in Figure 1, the user can manually provide input data, which can be done through the interface provided by I / O interface 112. Alternatively, the client device 140 can automatically send input data to I / O interface 112. If user authorization is required for the client device 140 to automatically send input data, the user can set the corresponding permissions in the client device 140. The user can view the output results of the electronic device 110 on the client device 140, which can be presented in various forms such as display, sound, or animation. The client device 140 can also act as a data acquisition terminal, collecting the input data and output results of the input I / O interface 112 as shown in Figure 1 as new sample data and storing them in the database 130. Alternatively, data can be collected directly from the I / O interface 112 without going through the client device 140, using the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130.

[0101] It is understood that Figure 1 is only a schematic diagram of a system architecture provided by an embodiment of this application. The positional relationship between the devices, components, modules, etc. shown in Figure 1 does not constitute any limitation. For example, in Figure 1, the data storage system 150 is an external memory relative to the electronic device 110. In other cases, the data storage system 150 can also be placed in the electronic device 110.

[0102] The diffusion model inference method provided in this application will be described in detail below with reference to the accompanying drawings.

[0103] Please refer to Figure 2, which is a flowchart illustrating a diffusion model inference method provided in an embodiment of this application. This diffusion model inference method is applied in the field of artificial intelligence technology. Specifically, the diffusion model inference method includes, but is not limited to, the following steps:

[0104] S201: The diffusion model inference device obtains the first path of the diffusion model and the weight coefficients of each time step in the first path.

[0105] It is understood that the diffusion model inference device in the embodiments of this application may be a device equipped with a processor / chip that can execute computer execution instructions, or it may be a processor / chip that can execute computer execution instructions. Optionally, the diffusion model inference device may be the aforementioned electronic device, or it may be a processor / chip within an electronic device.

[0106] It is understood that the diffusion model in the embodiments of this application can be any diffusion model, and there are no limitations on the data modality or network structure of the diffusion model. For example, the diffusion model can be a conditional diffusion model, an unconditional diffusion model, or a variational diffusion model, etc. A conditional diffusion model introduces data generation conditions during the generation process, such as text descriptions or category labels, to control specific attributes of the generated content, achieving conditional generation of output data. An unconditional diffusion model does not use any conditional information, relying solely on the distribution of the data itself to generate new samples. A variational diffusion model combines the theoretical framework of a variational autoencoder to improve generation quality and sampling efficiency. The variational diffusion model can support both unconditional and conditional generation modes.

[0107] It is understood that the first path is a sampling path that meets the preset data accuracy, obtained based on the second path of the diffusion model, and / or a sampling path obtained based on the second path with a preset number of time steps and an error of less than a first error threshold at each time step. The second path is the denoising process path of the diffusion model, which consists of several time steps. The number of time steps in the first path is less than the number of time steps in the second path. The preset data accuracy refers to the accuracy requirement of the output data of the diffusion model. The number of time steps refers to the number of steps in the denoising process of the diffusion model. The specific values ​​of the preset data accuracy and the preset number of time steps can be set according to the actual situation and are not limited.

[0108] Optionally, the error of the aforementioned time step refers to the error between the output latent variable of that time step and the output latent variable of the second path at the time step corresponding to that time step.

[0109] Optionally, the weight coefficients of the above time steps refer to the weights used in denoising, i.e., the projection matrix.

[0110] It is understandable that step S201 is to fit the second path with a high number of steps to obtain the first path with a low number of steps while ensuring data accuracy, thus ensuring data accuracy while reducing the number of iterations in the denoising process of the diffusion model.

[0111] S202: The diffusion model inference device uses the first scheduler and noise prediction network of the diffusion model to denoise based on the first path and the weight coefficients of each time step in the first path, so as to obtain the first output data of the diffusion model.

[0112] Understandably, the first scheduler controls the diffusion model to perform denoising based on the first path and the weight coefficients of each time step within the first path. The noise prediction network, on the other hand, predicts the denoised noise at each time step within the first path.

[0113] Optionally, the first output data can be at least one of the following: image data (such as two-dimensional images, three-dimensional model renderings, etc.), audio data, text data, video data, point cloud data, molecular data, or time series data (such as financial data, sensor data, etc.).

[0114] Through the embodiments of this application, the diffusion model inference device, based on an inference method that reduces the number of sampling steps in the diffusion model, can fit a first path based on a second path of the diffusion model according to a preset data precision or a preset number of time steps. This first path is a sampling path that meets the preset data precision and has fewer time steps than the second path, and / or, the first path is a sampling path with a preset number of time steps and an error at each time step less than a first error threshold, where the preset time steps are less than the second path's time steps. This solution can effectively reduce the number of sampling steps in the diffusion model while ensuring the accuracy of the output data, reducing the number of denoising iterations, reducing computational load, accelerating diffusion model inference, and solving the problem of reducing the inference load of the diffusion model. Furthermore, the embodiments of this application can generate a first path that meets specified requirements according to the user's actual needs, improving the practicality of the diffusion model and enhancing the user experience.

[0115] It is understandable that since the first path and weight coefficients are obtained based on the fitting of the second path, the method of this application embodiment can improve the efficiency and speed of data generation while ensuring the quality of the generated data, thereby promoting the innovation of artificial intelligence technology and the rapid development of the industry.

[0116] In one possible embodiment, the above diffusion model inference method may also perform the following steps, including but not limited to:

[0117] The diffusion model inference device acquires a preset number of time steps and / or a preset data precision.

[0118] Understandably, before performing inference on the diffusion model, it is necessary to obtain the weight coefficients of the first path and its time steps. Obtaining the weight coefficients of the first path and its time steps requires a preset number of time steps and / or a preset data precision; therefore, it is necessary to set the preset number of time steps and / or a preset data precision first. Referring to Figure 3A, which is a schematic diagram of a diffusion model inference method provided in an embodiment of this application, exemplarily, the user can set the preset number of time steps and / or the preset data precision through an information input interface.

[0119] Referring to Figure 3B, which is a schematic diagram of an information input interface provided in an embodiment of this application, Figure 3B uses image generation as an example. The "sampling steps" are used to select a preset number of time steps. The image generation requirements can also be set, such as "restore face," "tile," or "high resolution." Exemplarily, the image width, image height, batch count, batch size, and Classifier-Free Guidance Scale (CFG scale) can also be customized.

[0120] In this embodiment of the application, referring to Figure 3A, the diffusion model inference method includes a training phase and an inference phase. The training phase is used to obtain the weight coefficients of the first path and its time steps, while the inference phase performs inference of the diffusion model based on the weight coefficients of the first path and its time steps to obtain output data. The data output by the training phase and the inference phase are of the same data type, that is, the training phase and the inference phase utilize the same diffusion model to complete the same data generation task.

[0121] Optionally, referring to Figure 3A, when the method of this embodiment faces multiple different diffusion models, it is necessary to select the diffusion model for training and inference before the training and inference phases. After the training phase, the corresponding first path and weight coefficients of the diffusion model can be saved. During the inference phase, after selecting the diffusion model, the corresponding first path and weight coefficients of the diffusion model can be loaded for model inference.

[0122] Referring to Figure 3A, during the training phase, it is necessary to first collect data on the (second) path, and then perform path planning based on the second path to obtain the weight coefficients of the first path and its time step.

[0123] In one possible embodiment, the above diffusion model inference method may also perform the following steps, including but not limited to:

[0124] The diffusion model inference device denoises the diffusion model using a second scheduler to obtain a second path and second output data of the diffusion model. The first and second output data are of the same type. This second scheduler is a packaged scheduler.

[0125] Understandably, the encapsulated scheduler is a scheduler that abstracts complex sampling logic into a simple interface, and can output parameters such as time step, prediction noise, and output latent variables through the interface.

[0126] Optionally, the second scheduler can be any scheduler, such as the denoising diffusion implicit model DDIM scheduler, the denoising diffusion probabilistic model DDPM scheduler, or the pseudo-numerical method PNDM scheduler for diffusion models.

[0127] Optionally, the number of time steps for the second path can be set according to the actual situation and is not limited.

[0128] The diffusion model inference device uses a second scheduler to denoise the diffusion model to obtain a denoised path, i.e., a second path. At the same time, it can also obtain second output data. The first output data and the second output data are data of the same type. In other words, the first output data and the second output data are obtained by using a diffusion model that performs the same task.

[0129] In one possible embodiment, the above diffusion model inference method may also perform the following steps, including but not limited to:

[0130] The diffusion model inference device acquires and stores the path data of the second path through the second scheduler. This path data includes the output latent variables and the prediction noise at each time step.

[0131] Optionally, the path data described above may also include the input latent variables for each time step. The input latent variables for the current time step can be understood as the output latent variables for the previous time step. Similarly, the output latent variables for the current time step can be understood as the input latent variables for the next time step.

[0132] Optionally, in the above-mentioned storage of the path data for the second path, to avoid random errors, the diffusion model is denoised multiple times by the second scheduler to obtain multiple sets of path data for the second path. Based on the multiple sets of path data for the second path, the first path and the weight coefficients of each time step in the first path can be obtained.

[0133] After collecting the path data of the second path, step S201 is executed to obtain the weight coefficients of the first path and its various time steps.

[0134] In one possible embodiment, obtaining the first path of the diffusion model in step S201 above can be achieved in ways including but not limited to the following:

[0135] Implementation Method 1:

[0136] S1. The diffusion model inference device obtains a fitted path from the second path based on the second error threshold, wherein the error at each time step in the fitted path is less than the second error threshold.

[0137] It is understandable that the specific value of the second error threshold can be set according to the actual situation and is not limited.

[0138] For example, the next hop time step can be selected from the starting point of the second path based on a second error threshold, such that the error at this time step is less than the second error threshold. This process is repeated until the end of the second path, thus obtaining a fitted path.

[0139] S2. If the number of time steps in the fitted path is not equal to the preset number of time steps, the diffusion model inference device adjusts the second error threshold and re-executes step S1 until the number of time steps in the fitted path is equal to the preset number of time steps.

[0140] Optionally, the second error threshold can be a large threshold, and it is continuously reduced when acquiring the first path until the fitted path meets the specified number of time steps. Alternatively, the second error threshold can be a small threshold, and it is continuously increased when acquiring the first path until the fitted path meets the specified number of time steps.

[0141] In this embodiment, the diffusion model inference device first obtains a fitted path from the second path based on a second error threshold. Then, it determines whether the fitted path meets the specified number of time steps. If not, it adjusts the second error threshold and re-obtains the fitted path based on the adjusted second error threshold until the number of time steps in the fitted path equals the preset number of time steps. The fitted path at this point is the first path, and the second error threshold at this point is the aforementioned first error threshold. Therefore, the diffusion model inference device, by setting an error limit (i.e., the second error threshold) and continuously adjusting the error limit, finds a critical time step path (i.e., the first path) within this error limit. This maximizes the number of non-critical time steps between adjacent critical time steps (i.e., time steps in the first path) within this error limit, ultimately finding a critical time step path that meets the specified number of time steps and has the smallest possible error—the first path.

[0142] In this implementation, the preset number of time steps is less than the number of time steps in the second path, and the error of each time step in the first path is less than the first error threshold. This can effectively reduce the number of sampling steps in the diffusion model, reduce the number of denoising iterations, and reduce the amount of computation while ensuring the accuracy of the output data, thereby accelerating the inference of the diffusion model.

[0143] Implementation Method Two:

[0144] The diffusion model inference device can obtain the error at each time step in the first path using a preset data precision. Based on the error at each time step, the first path can be fitted from the second path. Through the embodiments of this application, a first path that meets the specified data precision requirements and reduces the number of time steps can be obtained from the second path to meet different model usage needs of users.

[0145] In one possible embodiment, obtaining the weight coefficients of each time step in the first path in step S201 can be achieved in ways including but not limited to the following:

[0146] The diffusion model inference device obtains the weight coefficients of the third time step in the first path based on the output latent variables and prediction noise of the first time step and the output latent variables and prediction noise of the second time step.

[0147] It is understandable that the first and second time steps mentioned above are the two time steps in the second path that correspond to the initial state and the target state of the third time step. The third time step, on the other hand, is any time step in the first path.

[0148] For example, the diffusion model inference device uses the least squares method to obtain the minimum regression coefficients, i.e., the weight coefficients of the third time step, based on the output latent variables and prediction noise of the first time step and the output latent variables and prediction noise of the second time step. This minimizes the distance between the first path and the second path in the vector space, reducing the number of iterations while ensuring the data quality of the output data. Furthermore, the diffusion model inference device can obtain the output latent variables of the third time step based on the output latent variables and prediction noise of the first time step and the weight coefficients of the third time step.

[0149] Optionally, when obtaining the first path and its time step weight coefficients based on multiple path data of the second path, the multiple data corresponding to the time step can be concatenated to consider multiple data simultaneously during path planning, thereby avoiding random errors during path planning.

[0150] In this embodiment, the encapsulated second scheduler can be used to obtain path data such as output latent variables and prediction noise at each time step in the second path, and this path data can be saved for subsequent fitting. Furthermore, based on the path data of the second path, the weight coefficients of the first path and its corresponding time steps can be calculated.

[0151] In one possible embodiment, step S202 described above can be implemented in ways including but not limited to the following:

[0152] The diffusion model inference device acquires the target noise for denoising using the diffusion model.

[0153] Optionally, referring to Figure 3A, the target noise described above can be generated using random numbers. Optionally, the target noise can be a latent space variable.

[0154] Optionally, noise can be gradually added to the real data until it is approximately pure noise, thus obtaining the target noise mentioned above.

[0155] The diffusion model inference device repeatedly executes the following denoising steps until the execution count reaches N, where N is a preset time step. The first output data is obtained based on the output latent variables of the Nth denoising step:

[0156] The predicted noise e for the current time step t is generated using a noise prediction network. t .

[0157] Optionally, a noise prediction network is used to generate predictive noise based on data generation conditions. These data generation conditions are related to the first output data and have a semantic or logical connection with it. Optionally, the data generation conditions include at least one of the following types of data: image data, video data, text data, or audio data. For example, the data generation condition could be the text data "Help me generate an image of a boy playing on his phone."

[0158] For example, referring to Figure 3A, after the encoder encodes the data generation conditions, the encoded features are then transformed by the feature transformation module and input into the noise prediction network. This allows the noise prediction network to predict noise based on the input features, thereby generating output data according to the conditions. Optionally, the encoder and feature transformation module can be implemented using various network structures. For example, the feature transformation module can be an embedding network or a feature hash transformation network to map high-dimensional input features to low-dimensional features, thereby improving computational efficiency.

[0159] Through the first scheduler, based on e t The output latent variable x of the previous time step at the current time step t+1 x t+1 The corresponding weighting coefficient w x,t and e t weighting coefficient w e,t Obtain the output latent variable x at the current time step t .

[0160] In the initial denoising, t+1 is N, and x t+1 The target noise.

[0161] For example, x t =w x,t x t+1 +w e,t e t The above formula is used to perform a first-order solution for noise reduction. The first-order solver is repeated until the time step of the first path ends.

[0162] It is understandable that solving for the output latent vector at the current time step is based on: x t =w x,t x t+1 +w e,t et It can be regarded as x t ∈span{x t+1 ,e t The output latent vector at the current time step is considered as a projection of the linear space consisting of the output latent vector of the previous step and the prediction noise at the current time step. This strategy can be mathematically inductively derived as x. t(i+1) ∈span{x t(1) ,e t(1) ,…,e t(i) The output latent variable at any time step can be represented by span{x}. t(1) ,e t(1) ,…,e t(i) Orthogonal projection representation in the vector space of}

[0163] The aforementioned denoising strategy replaces time-step coupled denoising with a vector space projection transformation. The latent variables of the initial time step in the first path are linearly combined with all predicted noise before the current time step in the first path, and the coefficients are the projection matrix (i.e., weighting coefficients). This combination calculates the state (output latent variables) for the next time step. Due to the additivity of vector space, high-quality output data can be generated with very few time steps. The projection matrix minimizes the error between adjacent time steps in the first path. The denoising coefficients (weighting coefficients) are calculated using the projection matrix instead of the product of the noise distribution coefficients of the time steps, requiring only first-order linear computation to denoise while maintaining the quality of the output data. A first-order solver based on vector space projection achieves fast denoising, achieving the denoising effect of hundreds of steps in just a few.

[0164] Optionally, the latent variable output by the Nth denoising step can be used as the first output data, meaning that the output of the iterative denoising is the output data.

[0165] Optionally, when the target noise is a latent space variable, the latent space variable of the Nth denoising step is subjected to latent space variable decoding processing to obtain the first output data. The methods for the latent space variable decoding processing include, but are not limited to, deterministic decoding, random decoding, hierarchical decoding, or adversarial decoding. Taking deterministic decoding as an example, referring to Figure 3A, the latent space variable of the Nth denoising step is decoded using an autoencoder to obtain the first output data.

[0166] In this embodiment, when the target noise is a latent space variable, after iterative denoising, the latent variable output by the Nth denoising iteration needs to undergo latent space variable decoding to obtain the first output data. Executing the method of this embodiment in the latent space can reduce computational complexity, improve computational speed, and enhance the inference performance of the diffusion model.

[0167] Through the embodiments of this application, denoising based on the first path and the weight coefficients of each time step in the first path can quickly obtain the first output data. The denoising coefficients use a projection matrix (weight coefficients), requiring only first-order linear calculations for denoising, while ensuring the quality of the first output data.

[0168] The following uses video / image output data as an example to specifically illustrate the method of this application embodiment:

[0169] Referring to Figure 4A, which is a schematic flowchart of a diffusion model inference method provided in an embodiment of this application, the method specifically includes the following steps:

[0170] A401. Load the diffusion model and select the second scheduler.

[0171] Specifically, a diffusion model for inference is selected and loaded into the diffusion model inference device. Then, a suitable scheduler is chosen as the second scheduler based on the selected diffusion model, such as a high-step, high-quality scheduler. Using the second scheduler to obtain the path data for the second path ensures the quality of the final generated image or video.

[0172] A402. Perform image generation reasoning several times, for example, up to 15 times, and save the reasoning process data, that is, the path data of the second path.

[0173] Specifically, the scheduler selected in step A401 is encapsulated to store the fitted data (i.e., the path data of the second path). Several secondary graph inferences (i.e., denoising processes) are performed using the selected diffusion model. The path of this graph inference process is the second path. The encapsulated scheduler will store the input latent variables (and / or output latent variables) and prediction noise at each time step in the second path, and use them as the data to be fitted.

[0174] A403, Path Planning: Fixing the initial state x of the second path T Take points sequentially backwards, according to... Find the first path with a length of a preset number of time steps, where the error of each point is less than the first error threshold. Calculate the weight coefficient of each time step in the first path.

[0175] Specifically, points with a preset number of time steps are selected from the second path where the data to be fitted is stored as the first path, and the error of each time step in the first path is less than a first error threshold. The specific method for selecting each time step in the first path can be referred to the relevant descriptions of steps S1 and S2 above, and will not be repeated here.

[0176] Next, the weight coefficient of each time step in the first path is calculated based on the path data of the second path. The specific calculation method can be referred to the relevant description of obtaining the weight coefficient of each time step in the first path in step S201 above, and will not be repeated here.

[0177] A404. The first scheduler uses the weight coefficients of the first path and its time step for denoising.

[0178] Specifically, the fitted weight coefficients from step A403 are loaded, and iterative denoising is performed based on the first path. The target noise at the denoising starting point is obtained. In this embodiment, taking the target noise as a latent space variable, the noise prediction network generates the predicted noise e at the current time step. t Based on target noise and prediction noise e t and the weight coefficient (w) of the current time step x,t and w e,t ) to perform noise reduction, x t =w x,t x t+1 +w e,t e t During the initial denoising, x t+1 The target noise is used. Repeating this iteration several times yields the final latent space variables.

[0179] A405, after being decoded, yields the corresponding video / image.

[0180] Specifically, for latent space variables Decode the video or image.

[0181] The method shown in Figure 4A can select a non-uniform time step sequence (i.e., the first path), generate a path with a specified number of time steps and its corresponding weight coefficients, reduce the number of sampling steps in the diffusion model while ensuring the accuracy of the output data, reduce the number of iterations, reduce the amount of computation, and accelerate the inference of the diffusion model.

[0182] Referring to Figure 4B, which is a schematic flowchart of another diffusion model inference method provided in this application embodiment, the method specifically includes the following steps:

[0183] B401. Load the diffusion model and select the second scheduler.

[0184] B402. Perform several image generation and reasoning operations, and save the data from the reasoning process.

[0185] Specifically, the second scheduler is used to perform a denoising process on the diffusion model in order to obtain the second path and its path data.

[0186] B403, Path Planning: Based on the preset data accuracy, select key time steps from the saved second path, requiring the error of each step to be closest to the given accuracy requirement. Continuously loop to find the first path and calculate the weight coefficient of each time step in the first path.

[0187] Specifically, based on a preset data precision, the error of each time step in the first path is derived. Then, based on this error and the path data, a time step point that satisfies the error is selected from the second path as the next hop. This allows the generation of an adaptive-length time step path, i.e., the first path, according to the desired precision.

[0188] The minimum regression coefficient between each two adjacent points in the first path is calculated as the weight coefficient for the time step.

[0189] B404. The first scheduler uses the weight coefficients of the first path and its time step for denoising.

[0190] B405, after being decoded, yields the corresponding video / image.

[0191] Using the method shown in Figure 4B, a non-uniform time step sequence (i.e., the first path) can be selected according to the specified data precision, generating a sampling path that meets the precision requirements and reduces the number of steps, along with its corresponding iteration weights. This reduces the number of sampling steps in the diffusion model while ensuring the precision of the output data, thereby reducing the number of iterations, decreasing the computational load, and accelerating model inference.

[0192] Referring to Figure 5, which is a denoising schematic diagram provided in an embodiment of this application; a first path 502 can be obtained by fitting using the second path 501, and an image 503 can be generated by denoising using the first path 502. In this embodiment, the first path 502 is a sampling path that satisfies a preset number of time steps and the error of each time step is less than a first error threshold. The preset number of time steps is 3 as an example. The second path 501 corresponds to the denoising process of the second scheduler, including time steps x. T →x t Time step x t →x t-1 Time step x t-1 →x t-2 Time step x t-2 →x t-3 Time step x t-3 →x t-4 and time step x t-4 →x t-5 Six time steps. The first path corresponds to the denoising process of the first scheduler, including time step x. T →x t Time step x t →x t-2 and time step x t-2 →xt-5 Three time steps. The first scheduler learns from the denoising process of the second scheduler, and can fit a high-quality short path with only a few training samples. For example, when the data generation condition is the text data "Help me generate an image of a boy playing on his phone", the noise prediction network performs noise prediction based on the above data generation condition to affect the generated output data, thereby realizing conditional generation.

[0193] Referring to Tables 1 and 2, there is a comparison of the acceleration benefits of using the methods of the embodiments of this application with methods that do not use the embodiments of this application (such as the DDIM scheduler, Euler scheduler, DPM scheduler, and SA-Solver scheduler in the table). The methods of the embodiments of this application can be referred to as Vector Projection Scheduler (VPS), such as VPS_10 and VPS_15 in the table, where VPS_10 indicates that the first path includes ten time steps, and VPS_15 indicates that the first path includes fifteen time steps.

[0194] Table 1

[0195] Table 2

[0196] According to Tables 1 and 2, using the method of this application embodiment, compared with the method without using this application embodiment, the number of sampling steps is optimized from 50 steps to 10 steps and 15 steps, and the inference speed of 512*512 images is accelerated by ≈5 times; the accuracy error clip score of the diffusion model SD2.1 on the partiprompt data is verified to be <0.003; using the method of this application embodiment can improve throughput and image quality detail fidelity.

[0197] In scenarios requiring high generation speed and timeliness, this application's embodiments can reduce inference load and significantly improve the efficiency of model-generated data. For example, in the field of image or video generation, such as digital art creation, advertising design, and game development, this technology can achieve real-time or near real-time high-quality image generation. In scenarios such as film and television special effects pre-visualization and automatic generation of short video content, this application's embodiments can efficiently synthesize high frame rate video content. Furthermore, for devices such as smartphones and AR / VR headsets, the diffusion model optimized using the method of this application's embodiments supports localized rapid generation and editing of content. It is particularly suitable for scenarios such as offline tourism guides and educational courseware generation, expanding the inclusiveness of generative artificial intelligence (AI). The method in this application's embodiments can connect and generate multimodal data such as text, images, and audio. In the field of machine learning, the accelerated diffusion model can generate synthetic data on a large scale for training AI models such as vision and speech. In the future, with the synergistic optimization of algorithms and hardware, this technology is expected to achieve large-scale implementation in more fields such as metaverse construction, personalized education, and smart city management. With the deep integration of algorithms and hardware, this technology is expected to give rise to more revolutionary application scenarios in cutting-edge fields such as brain-computer interfaces and quantum computing.

[0198] It is understood that the methods in the embodiments of this application are not limited to the field of image or video generation, but are also applicable to other data generation fields such as text or audio. They can reduce the inference load of the model and improve the efficiency of data generation while ensuring the accuracy of the output data.

[0199] The methods of the embodiments of this application have been described in detail above. The following provides an apparatus for implementing any one of the methods in the embodiments of this application. For example, an apparatus is provided that includes a unit (or means) for implementing the steps performed by the device in any of the above methods.

[0200] Please refer to Figure 6, which is a schematic diagram of the structure of a diffusion model inference device provided in an embodiment of this application.

[0201] As shown in Figure 6, the diffusion model inference device 60 may include an acquisition unit 601 and an inference unit 602. The acquisition unit 601 and the inference unit 602 may be software, hardware, or a combination of software and hardware.

[0202] In one possible design, the diffusion model inference device 60 may correspond to the diffusion model inference device in the method embodiment shown in FIG2 above. For example, the diffusion model inference device 60 may be an electronic device or a chip within an electronic device. The diffusion model inference device 60 may include units for performing the operations performed by the diffusion model inference device in the method embodiment shown in FIG2 above, and each unit in the diffusion model inference device 60 is for implementing the operations performed by the diffusion model inference device in the method embodiment shown in FIG2 above. The descriptions of each unit are as follows:

[0203] The acquisition unit 601 is used to acquire the first path of the diffusion model and the weight coefficients of each time step in the first path.

[0204] The inference unit 602 is used to denoise based on the first path and the weight coefficients of each time step in the first path through the first scheduler and the noise prediction network of the diffusion model, so as to obtain the first output data of the diffusion model.

[0205] The first path is a sampling path that meets a preset data accuracy, obtained from the second path based on the diffusion model, and / or a sampling path obtained from the second path with a preset number of time steps and an error less than a first error threshold at each time step. The second path is the denoising process path of the diffusion model, and the number of time steps in the first path is less than the number of time steps in the second path.

[0206] Regarding the acquisition unit 601 and the inference unit 602 described in this design, the steps they perform can be referred to the implementation method corresponding to the diffusion model inference device in the method embodiment shown in Figure 2 above.

[0207] Regarding the technical effects of the implementation methods performed by the acquisition unit 601 and the inference unit 602 described in this design, please refer to the description of the technical effects corresponding to the method embodiment shown in FIG2 above.

[0208] According to embodiments of this application, the various units in the device shown in FIG6 can be individually or entirely merged into one or more other units, or some of the units can be further divided into multiple functionally smaller units. This achieves the same operation without affecting the technical effect of the embodiments of this application. The above units are based on logical function division. In practical applications, the function of one unit can also be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of this application, the electronic device may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.

[0209] It should be noted that the implementation of each unit can also refer to the corresponding description of the method embodiment shown in Figure 2 above.

[0210] In the diffusion model inference apparatus 60 described in Figure 6, the problem of how to reduce the model inference load can be solved.

[0211] If the diffusion model inference device 60 described above can be an electronic device, please refer to the structural schematic diagram of the electronic device shown in Figure 7.

[0212] It should be understood that the electronic device 70 shown in FIG7 is only an example. The electronic device in the embodiments of this application may also include other components, or include components with functions similar to the various components in FIG7, or may not include all the components in FIG7.

[0213] Electronic device 70 includes a transceiver interface 701 and at least one processor 702.

[0214] The electronic device 70 can correspond to a diffusion model inference device. The transceiver interface 701 is used for transmitting and receiving signals, and at least one processor 702 executes program instructions, causing the electronic device 70 to implement the corresponding flow of the method executed by the corresponding device in the above method embodiments.

[0215] In one possible design, the electronic device 70 may correspond to the diffusion model inference device in the method embodiment shown in FIG2 above. For example, the electronic device 70 may be a diffusion model inference device or a chip within the diffusion model inference device. The electronic device 70 may include components for performing the operations performed by the diffusion model inference device in the method embodiment above, and each component in the electronic device 70 is respectively for implementing the operations performed by the diffusion model inference device in the method embodiment above. Specifically, it may be as follows:

[0216] The transceiver interface 701 is used to obtain the first path of the diffusion model and the weight coefficients of each time step in the first path.

[0217] Processor 702 is used to denoise based on a first path and the weight coefficients of each time step in the first path through a first scheduler and a noise prediction network of the diffusion model to obtain the first output data of the diffusion model.

[0218] The first path is a sampling path that meets a preset data accuracy, obtained from the second path based on the diffusion model, and / or a sampling path obtained from the second path with a preset number of time steps and an error less than a first error threshold at each time step. The second path is the denoising process path of the diffusion model, and the number of time steps in the first path is less than the number of time steps in the second path.

[0219] Regarding the transceiver interface 701 and at least one processor 702 described in this design, the steps they perform can be referred to the implementation corresponding to the diffusion model inference device in the method embodiment shown in Figure 2 above.

[0220] For the technical effects of the implementation methods performed by the transceiver interface 701 and at least one processor 702 described in this design, please refer to the description of the technical effects corresponding to the method embodiment shown in FIG2 above.

[0221] In the electronic device 70 described in Figure 7, the problem of how to reduce the model inference load can be solved.

[0222] For cases where the diffusion model inference device 60 can be a chip or a chip system, please refer to the schematic diagram of the chip structure shown in Figure 8.

[0223] As shown in Figure 8, chip 80 includes a processor 801 and an interface 802. The number of processors 801 can be one or more, and the number of interfaces 802 can be multiple. It should be noted that the functions of the processor 801 and interface 802 can be implemented through hardware design, software design, or a combination of both; no restrictions are placed here.

[0224] Optionally, chip 80 may also include memory 803 for storing necessary program instructions and data.

[0225] In this application, processor 801 can be used to call the implementation program of the diffusion model inference method provided in one or more embodiments of this application in a diffusion model inference device from memory 803, and execute the instructions included in the program. Interface 802 can be used to output the execution result of processor 801. In this application, interface 802 can be specifically used to output various messages or information of processor 801.

[0226] The diffusion model inference method provided by one or more embodiments of this application can be referred to the various embodiments shown in Figure 2 above, and will not be repeated here.

[0227] The processor in this application embodiment can be a central processing unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0228] The memory in this application embodiment is used to provide storage space, in which data such as operating system and computer programs can be stored. The memory includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM).

[0229] According to the method provided in the embodiments of this application, the embodiments of this application also provide a computer-readable storage medium storing a computer program. When the computer program is run on one or more processors, it can implement the method shown in FIG2.

[0230] According to the method provided in the embodiments of this application, the embodiments of this application also provide a computer program product, which includes a computer program. When the computer program runs on a processor, it can implement the method shown in FIG2.

[0231] This application also provides a processing apparatus, including a processor and an interface; the processor is used to execute the method in any of the above method embodiments.

[0232] It should be understood that the above-described processing device can be a chip. The units in the various device embodiments and the electronic devices in the method embodiments correspond completely, with corresponding modules or units executing corresponding steps. For example, the communication unit (transceiver) executes the receiving or sending steps in the method embodiments, while other steps besides sending and receiving can be executed by the processing unit (processor). The specific functions of each unit can be found in the corresponding method embodiments. There can be one or more processors.

[0233] It is understood that in the embodiments of this application, the electronic device may perform some or all of the steps in the embodiments of this application. These steps or operations are merely examples, and the embodiments of this application may also perform other operations or variations thereof. Furthermore, the steps may be performed in different orders as presented in the embodiments of this application, and it is not necessarily necessary to perform all the operations in the embodiments of this application.

[0234] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0235] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0236] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0237] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the contributing part, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0238] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A diffusion model inference method, characterized in that, The method includes the following steps: Obtain the first path of the diffusion model and the weight coefficients of each time step in the first path; the first path is a sampling path that meets a preset data accuracy based on the second path, and / or a sampling path that has a preset number of time steps and the error of each time step is less than a first error threshold based on the second path; the second path is the denoising process path of the diffusion model; the number of time steps in the first path is less than the number of time steps in the second path. The diffusion model's first output data is obtained by denoising based on the first path and the weight coefficients of each time step in the first path using the first scheduler and the noise prediction network.

2. The method of claim 1, wherein, The first path for obtaining the diffusion model includes: S1. A fitted path is obtained by fitting the second path based on the second error threshold, wherein the error at each time step in the fitted path is less than the second error threshold; S2. If the time step of the fitted path is not equal to the preset time step, then adjust the second error threshold and re-execute step S1 until the time step of the fitted path is equal to the preset time step.

3. The method according to claim 1 or 2, characterized in that, The first output data of the diffusion model is obtained by denoising based on the first path and the weight coefficients of each time step in the first path through the first scheduler and the noise prediction network of the diffusion model, including: Obtain the target noise for denoising using the diffusion model; Repeat the following denoising steps until the number of executions reaches N, where N is the preset time step. Obtain the first output data based on the latent variables of the Nth denoising execution: generating, by the noise prediction network, a predicted noise e for a current time step t t ; by the first scheduler, based on the e t , an output latent variable x t+1 of a previous time step of the current time step t+1 , a corresponding weight coefficient w x,t and the e t weight coefficient w e,t of the e t ; Wherein, when first denoising, the t+1 is the N, the x t+1 is the target noise.

4. The method of claim 3, wherein, The target noise is a latent space variable, and the first output data is obtained based on the output latent variable of the Nth denoising step, including: The latent variables of the Nth denoising output are subjected to latent space variable decoding to obtain the first output data.

5. The method according to claim 3 or 4, characterized in that, The step of generating the predicted noise for the current time step through the noise prediction network includes: The predicted noise is generated by the noise prediction network based on data generation conditions, which are related to the first output data.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: The diffusion model is denoised by a second scheduler to obtain the second path and the second output data of the diffusion model. The first output data and the second output data are of the same type. The second scheduler is an encapsulated scheduler.

7. The method of claim 6, wherein, The method further includes: The path data of the second path is obtained and saved by the second scheduler. The path data includes the output latent variables and the prediction noise at each time step. The step of obtaining the weight coefficients for each time step in the first path includes: The weight coefficients of the third time step in the first path are obtained based on the output latent variables and prediction noise of the first time step and the output latent variables and prediction noise of the second time step. The first time step and the second time step are two time steps in the second path that correspond to the initial state and the target state of the third time step. The third time step is any time step in the first path.

8. A diffusion model inference apparatus characterized by comprising: It includes a unit for performing the diffusion model inference method according to any one of claims 1-7.

9. A diffusion model inference apparatus characterized by comprising: Includes a processor for executing the diffusion model inference method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program, which, when executed, performs the diffusion model inference method as described in any one of claims 1-7.

11. A computer program product, characterised in that, The computer program product includes a computer program, which, when executed, performs the diffusion model inference method as described in any one of claims 1-7.