Inference apparatus, inference system, and inference method

Abstract

Disclosed in the present application are an inference apparatus, an inference system, and an inference method. The inference apparatus comprises N AI chips corresponding to N levels of large-model inference, wherein N is greater than 1. AI chips corresponding to adjacent levels are connected in series, and each AI chip stores inference data at a corresponding level. An i-th AI chip is used for receiving input data at an i-th level, processing the input data on the basis of inference data at the i-th level, and obtaining and outputting a processing result, which is at the i-th level and is used for generating an inference result, wherein i is a positive integer greater than or equal to 1 and less than or equal to N, and when i is greater than 1 and less than or equal to N, the input data at the i-th level comprises a processing result for an (i-1)-th level. AI chips execute data processing at corresponding levels and store inference data at the corresponding levels, such that storage resources required by the AI chips are reduced; and the connection relationship between AI chips for performing processing at adjacent levels is simple.

Description

Reasoning apparatus, reasoning system and reasoning method

[0001] This application claims priority to Chinese Patent Application No. 202510208514.4, filed on February 24, 2025, entitled "Reasoning Device, Reasoning System and Reasoning Method", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of computer technology, and in particular to inference devices, inference systems and inference methods. Background Technology

[0003] Large-scale model inference refers to the process of generating corresponding output results based on input data using large-scale pre-trained models, such as large language models with a Transformer architecture. A large model consists of multiple layers. During large-scale model inference, each layer processes its input data based on the inference data used by that layer in the large-scale model inference process, and outputs the processing results, until the final inference result is output.

[0004] Inference devices for large-scale model inference using related technologies include multiple artificial intelligence (AI) chips. Data computation and other processing tasks at each level of the large model are handled collaboratively by these AI chips. Consequently, these AI chips are interconnected through different interfaces, requiring a large number of interfaces, consuming significant connection resources, and resulting in complex networking between them. Furthermore, because different levels process different data, and the inference data used for data processing at different levels also differs, each AI chip must store the inference data used at each level. Therefore, each AI chip requires substantial storage resources, leading to a high overall cost for the inference device. Summary of the Invention

[0005] This application provides a reasoning device, a reasoning system, and a reasoning method to reduce the cost of the reasoning device. The technical solution is as follows.

[0006] Firstly, an inference device is provided, comprising N sets of artificial intelligence (AI) chips, which correspond to N levels of large model inference, where N is greater than 1. AI chips corresponding to adjacent levels are connected in series, and each set of AI chips stores inference data for its corresponding level. The i-th set of AI chips is used to receive input data from the i-th level, process the input data according to the inference data of the i-th level, obtain the processing result of the i-th level, and output the processing result of the i-th level. The processing result of the i-th level is used to generate an inference result, where i includes a positive integer greater than or equal to 1 and less than or equal to N. Wherein, when i is greater than 1 and less than or equal to N, the input data of the i-th level includes the processing result of the (i-1)-th level.

[0007] Each AI chip group performs data processing at its corresponding level. Therefore, each AI chip group only needs to store the inference data used at its corresponding level. Each AI chip requires less storage resources. Furthermore, when each AI chip group performs inference tasks, the inference data stored in each AI chip group is called up, which reduces the idle storage resources and improves the utilization rate of storage resources.

[0008] In this application, each AI chip group only needs to connect to the AI ​​chip group corresponding to the next level and the AI ​​chip group corresponding to the previous level. Therefore, the AI ​​chips in adjacent levels are connected in series. The connection relationship between the AI ​​chips in this application is simple, the number of interfaces required is small, the connection cost is low, and the cost of the inference device is reduced. Moreover, when two adjacent AI chips communicate, they call the connection resources connecting the two adjacent AI chips, which reduces the idle connection resources and improves the utilization rate of connection resources.

[0009] In one possible implementation, the AI ​​chip includes a memory and an AI core, with the memory capacity being less than or equal to a capacity threshold. The memory in the i-th group of AI chips is used to store inference data for the i-th level. The AI ​​core in the i-th group of AI chips is used to read the inference data for the i-th level, receive input data for the i-th level, process the input data according to the inference data for the i-th level, obtain the processing result for the i-th level, and output the processing result for the i-th level. Since the AI ​​chip in this application only needs to store the inference data for the corresponding level of the AI ​​chip, configuring a memory with a capacity less than or equal to the capacity threshold for each AI chip can not only reduce the cost of each AI chip, but also reduce the idle storage resources and improve resource utilization.

[0010] In one possible implementation, the memory in the AI ​​chip is static random access memory (SRAM). SRAM typically has a small capacity. Since the memory in the AI ​​chip of this application is used to store the inference data used at the corresponding level, and the amount of inference data is relatively small, using SRAM as the memory in the AI ​​chip provides sufficient capacity to store the inference data used at the corresponding level, with some capacity remaining, thus reducing idle storage resources and improving the utilization rate of storage resources.

[0011] In one possible implementation, the inference device further includes a first processor connected to a first group of AI chips. The first processor transmits first-level input data to the first group of AI chips. Since there are no other groups of AI chips previously connected to the first group, the input data to the first group is not the processing result output by other groups. Transmitting first-level input data to the first group by the first processor enables the first group of AI chips to obtain accurate input data, thereby performing accurate inference tasks.

[0012] In one possible implementation, the first processor is further configured to acquire M sets of input prompts, segment the M sets of input prompts into words to obtain M sets of segmentation results, and generate M first-level input data including the M sets of segmentation results, where the j-th first-level input data includes the j-th set of segmentation results, and j is a positive integer greater than or equal to 1 and less than or equal to M; the first processor is configured to sequentially transmit the M first-level input data to the first group of AI chips. The sequential transmission of the M sets of segmentation results to the first group of AI chips enables the first group of AI chips to serially execute M inference tasks based on the M sets of segmentation results, reducing the amount of data processed by the first group of AI chips at one time and improving the data throughput of the first group of AI chips.

[0013] In one possible implementation, the i-th group of AI chips is used to process the j-th level-i input data based on the inference data of the i-th level, obtain the j-th level-i processing result, and output the j-th level-i processing result. The i-th group of AI chips is also used, when j is less than M, after outputting the j-th level-i processing result, to process the (j+1)-th level-i input data based on the inference data of the i-th level, obtain the (j+1)-th level-i processing result, and output the (j+1)-th level-i processing result. Each group of AI chips in the inference device executes the M inference tasks serially in a pipelined manner, improving the efficiency of inference task execution.

[0014] Multiple AI chips participate in the execution of each inference task. A group of AI chips at any given level performs data processing for that level of an inference task. The number of AI chips executing each inference task is stable, resulting in low computational and storage requirements for the multiple AI chips. After a group of AI chips at any given level completes data processing for that level of an inference task, it immediately proceeds to complete data processing for the next level of the same inference task, without waiting for one inference task to finish completely before starting the next. This serial execution of multiple inference tasks reduces the response time of each inference task and improves the efficiency of multiple AI chips executing multiple inference tasks.

[0015] In one possible implementation, the first processor is a central processing unit (CPU), an extended 8086 (x86) processor, an advanced reduced instruction set computing machine (ARM) architecture processor, or a digital signal processor (DSP). The first processor in this application can be of various types, enabling the inference device to have greater scenario adaptability and flexibility.

[0016] In one possible implementation, the inference device further includes a second processor connected to one or more of the N groups of AI chips. The second processor transmits corresponding inference data to each of the N groups of AI chips. This transmission of inference data by the second processor enables each group of AI chips to store its corresponding inference data and prevents the reception of inference data unrelated to its corresponding level, thereby reducing idle transmission resources and improving resource utilization.

[0017] In one possible implementation, the second processor is a CPU, an x86 processor, an ARM architecture processor, or a DSP. The second processor in this application can be of various types, enabling the inference device to have greater scenario adaptability and flexibility.

[0018] Secondly, a reasoning system is provided, which includes a plurality of reasoning devices described in the first aspect above and any possible implementation thereof.

[0019] Thirdly, a reasoning method is provided, which is applied to a reasoning device. The reasoning device includes N sets of artificial intelligence (AI) chips, which correspond to N levels of large model reasoning, where N is greater than 1. The AI ​​chips in the N sets of AI chips that correspond to adjacent levels are connected in series, and each set of AI chips stores the reasoning data of the corresponding level. The method includes: receiving input data of the i-th level through the i-th AI chip, processing the input data according to the reasoning data of the i-th level, obtaining the processing result of the i-th level, and outputting the processing result of the i-th level. The processing result of the i-th level is used to generate the reasoning result, where i includes a positive integer greater than or equal to 1 and less than or equal to N. Wherein, when i is greater than 1 and less than or equal to N, the input data of the i-th level includes the processing result of the (i-1)-th level.

[0020] In one possible implementation, the AI ​​chip includes a memory and an AI core, the capacity of which is less than or equal to a capacity threshold; the method further includes: storing inference data of the i-th level through the memory in the i-th group of AI chips; reading the inference data of the i-th level through the AI ​​core in the i-th group of AI chips; receiving input data of the i-th level through the i-th group of AI chips, processing the input data according to the inference data of the i-th level, obtaining the processing result of the i-th level, and outputting the processing result of the i-th level, including: receiving input data of the i-th level through the AI ​​core in the i-th group of AI chips, processing the input data according to the inference data of the i-th level, obtaining the processing result of the i-th level, and outputting the processing result of the i-th level.

[0021] In one possible implementation, the inference device further includes a first processor connected to a first group of AI chips; the method further includes: transmitting first-level input data to the first group of AI chips via the first processor.

[0022] In one possible implementation, the method further includes: acquiring M sets of input prompts through a first processor, segmenting the M sets of input prompts into words to obtain M sets of word segmentation results, and generating M first-level input data including the M sets of word segmentation results, wherein the j-th first-level input data includes the j-th set of word segmentation results, and j is a positive integer greater than or equal to 1 and less than or equal to M; transmitting the first-level input data to the first group of AI chips through the first processor, including: transmitting the M first-level input data to the first group of AI chips sequentially through the first processor.

[0023] In one possible implementation, processing the input data based on the inference data of the i-th level to obtain the processing result of the i-th level and outputting the processing result of the i-th level includes: processing the j-th i-th level input data based on the inference data of the i-th level through the i-th group of AI chips to obtain the j-th i-th level processing result and outputting the j-th i-th level processing result; the method further includes: using the i-th group of AI chips, when j is less than M, after outputting the j-th i-th level processing result, processing the (j+1)-th i-th level input data based on the inference data of the i-th level to obtain the (j+1)-th i-th level processing result and outputting the (j+1)-th i-th level processing result.

[0024] In one possible implementation, the inference device further includes a second processor connected to one or more of the N AI chips; the method further includes transmitting corresponding inference data to the N AI chips via the second processor.

[0025] It should be understood that the beneficial effects of the technical solutions and corresponding possible implementations of the second to third aspects of this application can be found in the above description of the technical effects of the first aspect and its corresponding possible implementations, and will not be repeated here. Attached Figure Description

[0026] Figure 1 is a schematic diagram of the structure of an AI server provided by related technologies;

[0027] Figure 2 is a schematic diagram of the structure of a reasoning device provided in an embodiment of this application;

[0028] Figure 3 is a schematic diagram of a hierarchical division provided in an embodiment of this application;

[0029] Figure 4 is a partial schematic diagram of a reasoning device provided in an embodiment of this application;

[0030] Figure 5 is a partial schematic diagram of another inference device provided in an embodiment of this application;

[0031] Figure 6 is a partial schematic diagram of another inference device provided in an embodiment of this application;

[0032] Figure 7 is a partial schematic diagram of another inference device provided in an embodiment of this application;

[0033] Figure 8 is a schematic diagram of the structure of an AI chip provided in an embodiment of this application;

[0034] Figure 9 is a schematic diagram of another inference device provided in an embodiment of this application;

[0035] Figure 10 is a schematic diagram of another inference device provided in an embodiment of this application;

[0036] Figure 11 is a flowchart illustrating the parallel execution of multiple inference tasks according to an embodiment of this application;

[0037] Figure 12 is a schematic diagram of parallel execution of multiple inference tasks provided in an embodiment of this application;

[0038] Figure 13 is a schematic diagram of another parallel execution of multiple inference tasks provided by an embodiment of this application;

[0039] Figure 14 is a schematic diagram of another parallel execution of multiple inference tasks provided in an embodiment of this application;

[0040] Figure 15 is a flowchart illustrating a reasoning method provided in an embodiment of this application. Detailed Implementation

[0041] The terminology used in the implementation section of this application is for the purpose of explaining specific embodiments of this application only, and is not intended to limit this application.

[0042] Large-scale inference based on the Transformer model structure typically consists of two phases: a prefill phase and a decoding phase. In the inference process of a single inference task, the user's prompt is first tokenized. The tokenized result is then input into the large model for the prefill phase inference task. This generates the first token (the smallest unit of text processing, often simply called a text unit) in the expanded text, completing the prefill phase inference task. The generated token is appended to the input token sequence, resulting in a new token sequence. Next, the decoding phase inference task is performed. Based on the new token sequence, inference is performed to generate the second token in the expanded text. This second token is then appended to the input token sequence, resulting in a new token sequence, and so on. This process is repeated until a stop sequence is generated or the length of the generated token sequence reaches the configured maximum sequence length, completing the decoding phase inference task. Generating a stop sequence means generating a specific flag or condition indicating the end of generation, thus stopping further content generation. For example, during the inference phase, the large model predicts the next token based on the input and the generated token sequence. When the model predicts that the next token is a special stop marker, or a token that meets certain pre-defined stopping conditions, it means that a stopping sequence has been generated, and the generation of subsequent content has stopped.

[0043] In related technologies, inference tasks are executed through AI servers. Figure 1 illustrates a schematic diagram of an AI server provided by related technologies. This AI server includes multiple CPUs and multiple AI chips. The AI ​​chips are connected to the CPUs via a high-speed peripheral component interconnect express (PCIe) bus, and the AI ​​chips are connected to each other via a high-speed bus, forming a full-mesh network. Each AI chip includes an AI core (DIE) and high-bandwidth memory (HBM) with large capacity.

[0044] In the AI ​​server provided by the related technology, during the inference task, the CPU determines the strategy for parallel execution of the inference task by multiple AI chips based on the number of model parameters, text (context) length, and AI chip resources. The CPU loads the large model parameters (or large model weights) from the determined strategy onto each AI chip and performs word segmentation on the prompt, obtaining a token sequence, which can be understood as a tensor containing one or more tokens. The CPU transmits the token sequence obtained from word segmentation to multiple AI chips according to the determined strategy. Multiple AI chips collaboratively complete the inference tasks in the pre-filling stage and the decoding stage based on the received token sequence, and generate all output tokens. The CPU detokenizes the tokens generated by each AI chip, that is, it synthesizes the output tokens into coherent text to obtain the generated complete text.

[0045] Because inference tasks in related technologies require the collaborative execution of multiple AI chips, each AI chip needs to store the entire large model's weights and user data, necessitating substantial storage resources. Furthermore, the inference task during the decoding phase is memory-bound, demanding extremely high memory access bandwidth; therefore, AI chips in these technologies are typically configured with large-capacity, high-bandwidth memory. Additionally, since inference tasks require the collaborative execution of multiple AI chips, each chip needs to frequently transmit data with other AI chips during execution. This necessitates configuring and connecting numerous interfaces for each AI chip, resulting in complex networking among multiple AI chips and requiring significant connection resources.

[0046] This application provides an inference device that reduces the consumption of connection resources and the complexity of connections, and achieves efficient large model inference. Referring to Figure 2, a schematic diagram of the structure of an inference device provided in this application is shown. The inference device shown in Figure 2 can be an inference server, specifically an inference server deployed with an inference software stack. The inference device includes N groups of AI chips, namely, AI chip 11 (Group 1), AI chip 12 (Group 2), ..., AI chip 1N (Group N). Here, N is a positive integer greater than 1. The AI ​​chips can include processors with different architectures, such as graphics processing units (GPUs), tensor processing units (TPUs), or neural network processing units (NPUs). The N groups of AI chips correspond to N layers in large model inference. The N layers refer to N neural network layers within the pre-filled or decoding domain during the large model inference process. These N neural network layers work collaboratively through ordered connections and information transmission to complete the inference task. Taking the Transformer large model as an example, the functions and data (such as data size, data format, data flow direction, etc.) of a well-trained Transformer large model are precisely defined. Therefore, during the execution of inference tasks, data processing and transmission can be carried out between different levels according to the defined data flow direction.

[0047] Referring to Figure 3, a schematic diagram of a hierarchical partitioning provided by an embodiment of this application is shown. The pre-filling domain is the domain of the pre-filling stage, and the decoding domain is the domain of the decoding stage. Both the pre-filling and decoding domains include multiple layers. A layer may include a multi-head attention layer (also simply called a multi-head layer), and may also include network layers for residual connections and layer normalization, as well as feed-forward network layers, and other types of network layers. During the execution of the inference task, layer 1 in the pre-filling or decoding domain processes the input data, and the processing result is transmitted to layer 2. Layer 2 further processes the processing result of layer 1 and outputs its own processing result, and so on, until all processing results are obtained, thus realizing the execution of the inference task.

[0048] Referring again to Figure 2, the N groups of AI chips in the inference device provided in this embodiment correspond to the N levels of large model inference. One group of chips corresponds to one level, or one group of chips can correspond to multiple levels, and multiple groups of chips can correspond to one level. Since a group of chips can include one chip or multiple chips, the correspondence between chips and levels can be one-to-one, many-to-one, or one-to-many. For example, the first group of AI chips 11 corresponds to the first level, that is, the first group of AI chips 11 is used to perform data processing of the first level of the pre-filled domain or decoding domain, the second group of AI chips 12 corresponds to the second level, ..., the Nth group of AI chips 1N corresponds to the Nth level. As another example, AI chip a is used to perform data processing of the first level. AI chip a belongs to the first AI chip group 11. In addition, AI chip a can also be used to perform data processing of the second level, so AI chip a also belongs to the second AI chip group. In other words, the AI ​​chips included in N AI chipsets may overlap, but each level has its own corresponding AI chipset. Therefore, when the number of levels is N, the multiple AI chips included in the inference device can be divided into N AI chipsets at different times or at different stages of data processing.

[0049] In N groups of AI chips, AI chips in adjacent layers are connected in series. For example, AI chip 11 in the first layer is connected in series with AI chip 12 in the second layer, and AI chip 12 in the second layer is also connected in series with AI chip 12 in the third layer, and so on. As another example, if AI chip 11 and AI chip 12 both correspond to the first layer, and AI chip 12 in the third layer corresponds to the second layer, then AI chip 11 and AI chip 12 in the first layer are both connected in series with AI chip 12 in the third layer.

[0050] Optionally, a group of AI chips may include one or more AI chips, and the number of AI chips in different groups may be the same or different. The number of AI chips in a group can be a power of 2, where n is an integer greater than or equal to 0. For example, a group of AI chips may include 1 AI chip, 2 AI chips, 4 AI chips, or 8 AI chips, etc. Alternatively, in some cases, the number of AI chips in a group may also be a number other than a power of 2, such as 3, 6, or 9. A group of AI chips may be connected in series with another group of AI chips in an adjacent layer, meaning there may be one or more connections between them.

[0051] If a group of AI chips includes multiple AI chips, these AI chips can be connected in series or interconnected. If multiple AI chips in a group are connected in series, the group can be connected to AI chips in adjacent groups through the AI ​​chips at both ends, thus achieving interconnection between adjacent groups of AI chips. For example, referring to a partial schematic diagram of an inference device shown in Figure 4, the second group of AI chips 12 includes four AI chips: AI chip 121, AI chip 122, AI chip 123, and AI chip 124. These four AI chips are connected in series sequentially. AI chip 121 and AI chip 124 are the AI ​​chips at both ends of the second group of AI chips 12. AI chip 121 is connected to one AI chip in the first group of AI chips 11, and AI chip 124 is connected to one AI chip in the third group of AI chips 13. Figure 4 only shows one AI chip in the first group of AI chips 11 and one AI chip in the third group of AI chips 13 as examples. The first group of AI chips 11 and the third group of AI chips 13 can also include other chips besides those shown in Figure 4.

[0052] If multiple AI chips in a group of AI chips are interconnected, then any one or more AI chips in that group can be connected to one or more AI chips in the corresponding groups of AI chips in adjacent layers, thus achieving series connection between two groups of AI chips in corresponding adjacent layers. For example, referring to a partial schematic diagram of another inference device shown in Figure 5, the fifth group of AI chips includes four AI chips: AI chip 51, AI chip 52, AI chip 53, and AI chip 54. AI chips 51, 52, 53, and 54 are interconnected. Since the fifth group of AI chips is connected to the fourth and sixth groups of AI chips respectively, one AI chip in the fifth group (e.g., AI chip 52) is connected to one AI chip in the fourth group, and another AI chip in the fifth group (e.g., AI chip 53) is connected to one AI chip in the sixth group. In the case shown in Figure 5, the fourth and fifth groups of AI chips, as well as the fifth and sixth groups of AI chips, are connected in series through a single connection.

[0053] Alternatively, referring to a partial schematic diagram of another inference device shown in Figure 6, AI chips 51, 52, 53, and 54 in the fifth group of AI chips are interconnected. One AI chip in the fourth group of AI chips is connected to AI chips 52 and 51 in the fifth group of AI chips, and one AI chip in the sixth group of AI chips is connected to AI chips 53 and 54 in the sixth group of AI chips. Figures 5 and 6 only show examples of one AI chip in the fourth group and one AI chip in the sixth group of AI chips. Both the fourth and sixth groups of AI chips may also include other chips besides those shown in Figures 5 and 6. In the case shown in Figure 6, the fourth and fifth groups of AI chips, as well as the fifth and sixth groups of AI chips, are connected in series through two connections.

[0054] For example, referring to a partial schematic diagram of another inference device shown in Figure 7, the fourth group of AI chips includes interconnected AI chips 41, 42, 43, and 44, and the fifth group of AI chips includes interconnected AI chips 51, 52, 53, and 54. When both groups of AI chips include multiple AI chips, multiple AI chips in one group can be connected to one or more AI chips in the other group. For example, in the partial schematic diagram of the inference device shown in Figure 7, AI chip 42 in the fourth group is connected to AI chip 52 in the fifth group, and AI chip 44 in the fourth group is connected to AI chip 51 in the fifth group. In the case shown in Figure 7, the fourth and fifth groups of AI chips are connected in series via two connections.

[0055] Therefore, it can be seen that one AI chip in a group of AI chips in this application embodiment can be connected to one or more AI chips in an adjacent group of AI chips, thereby realizing the series connection of two adjacent groups of AI chips.

[0056] Optionally, an AI chip can belong to a group of AI chips or multiple groups of AI chips. For example, AI chip 'a' belongs to both group 1 AI chip 11 and group 2 AI chip 12. Group 1 AI chip 11 does not include any other AI chips besides AI chip 'a', and similarly, group 2 AI chip 12 does not include any other AI chips besides AI chip 'a'.

[0057] In large-scale model inference, different levels require different inference data. Inference data for any level refers to the data used to complete the corresponding inference task at that level. This inference data can include, for example, weights (or parameters) indicating the connection strength between neurons at that level, bias parameters or activation functions used to adjust the activation thresholds of neurons at that level, and user data used to process the input data.

[0058] Since the N AI chips in the inference device provided in this application correspond to N levels respectively, and each group of AI chips performs the inference task of the corresponding level, each group of AI chips can store the inference data of the corresponding level, but not the inference data of other levels, thereby reducing the storage resources configured for each AI chip and further reducing the cost of the inference device.

[0059] Each AI chip stores inference data for the level corresponding to its group. If an AI chip belongs to one group, it stores inference data for the level corresponding to that group. If an AI chip belongs to multiple groups, it stores inference data for the levels corresponding to all of those groups.

[0060] During the inference task, the i-th AI chip group receives the input data from the i-th level, processes the input data based on the inference data from the i-th level, obtains the processing result from the i-th level, and outputs the processing result from the i-th level. The processing result from the i-th level is used to generate the inference result, where i includes a positive integer greater than or equal to 1 and less than or equal to N. Where i is greater than 1 and less than or equal to N, the input data from the i-th level includes the processing result from the (i-1)-th level.

[0061] For example, the first AI chip group 11 receives the first-level input data, processes it according to the stored first-level inference data, obtains the first-level processing result, and outputs the first-level processing result. The first-level processing result output by the first AI chip group 11 is transmitted to the second AI chip group 12. The second AI chip group 12 receives the first-level processing result and uses it as the second-level input data. The second AI chip group 12 processes the second-level input data according to the stored second-level inference data, obtains the second-level processing result, and outputs the second-level processing result. This process continues until the processing result of the (N-1)th AI chip group is transmitted to the Nth AI chip group. The Nth AI chip group uses the (N-1)th-level processing result as the Nth-level input data, processes the Nth-level input data according to the stored Nth-level inference data, and obtains the Nth-level processing result. This application does not limit the processing method of each AI chip for the input data in the embodiments. The processing method may include one or more of the following: linear transformation, nonlinear transformation, normalization, feature extraction or feature combination.

[0062] In this embodiment, each group of AI chips performs data processing at its corresponding level, thus each group of AI chips only needs to store the inference data used at its corresponding level. Compared to related technologies where multiple AI chips store inference data used at each level, each AI chip in this application requires less storage resources. Furthermore, in related technologies, multiple AI chips use inference data at different levels at different times, resulting in low overall memory utilization and idle resources used to store inference data at other levels, thus reducing resource utilization.

[0063] In this embodiment, each AI chip group only needs to connect to the AI ​​chip group corresponding to the next lower level and the AI ​​chip group corresponding to the previous level. Therefore, AI chips at adjacent levels are connected in series. Compared to interconnecting multiple AI chips, the connection relationship between AI chips in this embodiment is simpler, requires fewer interfaces, and has lower connection costs, thus reducing the cost of the inference device. Furthermore, in related technologies, when multiple AI chips perform data processing at any level, only some AI chips may perform that level of data processing. Therefore, in related technologies, after obtaining the processing result, the AI ​​chips performing data processing at any level transmit the processing result to the AI ​​chips performing data processing at the next level. The connection resources between the AI ​​chips performing data processing at that level and the AI ​​chips not performing data processing at the next level are not utilized, resulting in idle connection resources. In this embodiment of the application, after completing the data processing at the corresponding level, any group of AI chips transmits the processing result to a group of AI chips that are performing the data processing at the next level, and calls the connection resources between the group of AI chips and the group of AI chips corresponding to the next level. Since there is no connection between the group of AI chips and other groups of AI chips, the connection resources will not be idle, thereby improving the resource utilization rate.

[0064] In one possible implementation, referring to Figure 8, the AI ​​chip includes a memory and an AI core. The memory in the i-th group of AI chips is used to store inference data for the i-th level. Since the memory in each AI chip needs to store the inference data for its corresponding level, in this embodiment, the capacity of the memory in each AI chip is less than or equal to a capacity threshold to reduce the cost of the inference device. The capacity threshold can be set based on experience or by the user. For example, the memory in the AI ​​chip is static random access memory (SRAM). SRAM typically has a small capacity. Since the memory in the AI ​​chip of this application is used to store the inference data used for the corresponding level, and the amount of inference data is relatively small, using SRAM as the memory in the AI ​​chip provides sufficient capacity to store the inference data used for the corresponding level, with some capacity remaining, reducing idle storage resources and improving storage resource utilization.

[0065] The AI ​​core in the i-th group of AI chips is used to read the inference data of the i-th level, receive the input data of the i-th level, process the input data according to the inference data of the i-th level, obtain the processing result of the i-th level, and output the processing result of the i-th level.

[0066] When i is greater than 1, the input data of the i-th AI chip includes the processing result of the previous AI chip. However, since there is no processing result of other AI chips before the first AI chip 11 as the input data of the first AI chip 11, the input data of the first AI chip 11 needs to be obtained through other means.

[0067] In one possible implementation, as shown in Figure 9, the inference device further includes a first processor 21, which is connected to the first group of AI chips 11. The first processor 21 can be various types of processors, such as a central processing unit (CPU), x86 processor, ARM architecture processor, or DSP. The first processor in this application can be of multiple types, giving the inference device greater adaptability and flexibility in different scenarios. When the first processor 21 is a CPU, because the CPU has higher system coordination or task scheduling control capabilities, using the CPU as the first processor can accurately transmit input data to the first group of AI chips, or segment the input prompts to obtain accurate input data. It can then sequentially transmit the segmentation results of multiple input prompts to the first group of AI chips, controlling the first group of AI chips to sequentially execute the first level of data processing for multiple inference tasks, thereby accurately controlling multiple groups of AI chips to serially execute multiple inference tasks.

[0068] The first processor 21 is used to transmit first-level input data to the first AI chip group 11. The first-level input data may include data obtained based on user input, or inference results from other domains of the large model. For example, the first-level input data may include the word segmentation results of user-input prompts, or if the first level is the first level of the decoding domain, the first-level input data may also include the processing results of pre-filled domains, such as key-value cache (KV Cache) data.

[0069] Taking the input data of the first level, which includes the word segmentation result of the prompt input by the user, as an example, the first processor 21 can segment the prompt input by the user, obtain the word segmentation result, transmit the word segmentation result to the first AI chip 11, and use the word segmentation result as the input data of the first level.

[0070] Optionally, the inference device can also execute multiple inference tasks in parallel via a pipeline. For example, the first processor 21 is further configured to acquire M sets of input prompts, segment the M sets of input prompts into words to obtain M sets of word segmentation results, and generate M first-level input data including the M sets of word segmentation results. The j-th first-level input data includes the j-th set of word segmentation results, where j is a positive integer greater than or equal to 1 and less than or equal to M. For example, the first first-level input data includes the word segmentation results for the first set of input prompts, i.e., it includes the first set of word segmentation results.

[0071] A set of input prompts can correspond to one inference task or multiple inference tasks. Optionally, a set of input prompts can correspond to a user group; that is, a set of input prompts includes the prompts entered by users within a user group. For example, the prompts entered by users in user group 1 are the first set of input prompts, the prompts entered by users in user group 2 are the second set of input prompts, and so on. The number, type, and content of the input prompts included in different sets of input prompts can be the same or different.

[0072] For each input prompt, the first processor 21 performs word segmentation to obtain the segmentation result for each group, which includes one or more tokens. The first processor 21 sequentially transmits M first-level input data to the first group of AI chips 11. Optionally, during the transmission of the M first-level input data to the first group of AI chips 11, the first processor 21 can transmit the M first-level input data at specified time intervals. For example, the first group of word segmentation results is transmitted to the first group of AI chips 11 at time t1, the second group of word segmentation results is transmitted to the first group of AI chips 11 at time t2, and the third group of word segmentation results is transmitted to the first group of AI chips 11 at time t3. The time interval between time t1 and time t2 can be based on empirical settings or user settings, and the time interval between time t2 and time t3 can also be based on empirical settings or user settings. The time interval between transmitting different adjacent groups of word segmentation results can be the same or different.

[0073] Alternatively, during the process of transmitting M first-level input data to the first group of AI chips 11, the first processor 21 can also transmit data based on the processing status of each group of word results by the first group of AI chips 11. For example, the first processor 21 transmits the first group of word results to the first group of AI chips 11, and after the first group of AI chips 11 completes the processing of the first group of word results, the first processor 21 transmits the second group of word results to the first group of AI chips 11. The method by which the first processor 21 determines that the first group of AI chips has completed the processing of a group of word results is not limited in this embodiment. Exemplarily, after completing a group of word results, the first group of AI chips 11 can transmit processing completion information to the first processor 21, so that the first processor 21 determines that the first group of AI chips 11 has completed the processing of a group of word results, and then transmits the next group of word results to the first group of AI chips 11.

[0074] During the process of the inference device serially executing multiple inference tasks through a pipeline, the i-th group of AI chips is used to process the j-th level i-th input data based on the inference data of the i-th level, obtain the j-th level i-th processing result, and output the j-th level i-th processing result; the i-th group of AI chips is also used, when j is less than M, after outputting the j-th level i-th processing result, to process the (j+1)-th level i-th input data based on the inference data of the i-th level, obtain the (j+1)-th level i-th processing result, and output the (j+1)-th level i-th processing result.

[0075] In related technologies, when multiple AI chips have multiple inference tasks to execute, a portion of the AI ​​chips is allocated to execute one inference task, another portion to execute a different inference task, and so on. Multiple AI chips execute their respective inference tasks, allowing multiple inference tasks to be executed in parallel. The limited number of AI chips executing each inference task results in high computational and storage requirements for the multiple AI chips. Dedicated to a small number of AI chips for a single inference task may lead to reduced execution efficiency for each inference task. Furthermore, when the number of inference tasks is large, the computational power and number of multiple AI chips are insufficient to execute them in parallel. Therefore, multiple inference tasks must be processed in batches. For a period of time, multiple AI chips execute a portion of the inference tasks in parallel, completing one portion before starting another. This results in longer response times for later inference tasks, impacting user experience.

[0076] In this embodiment, multiple AI chips participate in the execution of each inference task. A group of AI chips at any given level executes the data processing for that level of an inference task. The number of AI chips executing each inference task is stable, resulting in lower computational and storage requirements for the multiple AI chips. After a group of AI chips at any given level completes the data processing for that level of an inference task, it immediately proceeds to complete the data processing for the next level of an inference task, without waiting for one inference task to finish completely before executing the next. This serial execution of multiple inference tasks reduces the response time of each inference task and improves the efficiency of multiple AI chips executing multiple inference tasks.

[0077] In one possible implementation, referring to Figure 10, the inference device further includes a second processor 22, which is connected to one or more of the N groups of AI chips. The second processor 22 is used to transmit corresponding inference data to the N groups of AI chips. This application embodiment does not limit the type of the second processor 22. For example, the second processor 22 can be a CPU, x86 processor, ARM architecture processor, or DSP, etc. The second processor in this application can be of various types, making the inference device more adaptable and flexible to different scenarios. Optionally, the second processor 22 and the first processor 21 can be of the same or different types. When the second processor 22 and the first processor 21 are of the same type, they can be the same processor or different processors. This application embodiment does not limit the method by which the second processor 22 obtains inference data at each level. For example, the second processor 22 can read inference data at each level from the knowledge base of a large model inference, or receive inference data at each level input by the user. If a CPU is used as the second processor, the second processor can accurately transmit the corresponding level of inference data to each group of AI chips through the system coordination capability of the CPU.

[0078] Referring to Figure 11, a flowchart illustrating the execution of multiple inference tasks according to an embodiment of this application is shown. Each group of AI chips processes input data at a fixed level in either the Prefill or Decode domain. The CPU loads the inference data for each level of the large model inference onto the corresponding group of AI chips. The CPU corresponds to the first processor 21 and the second processor 22 mentioned above; that is, both the first processor 21 and the second processor 22 are CPUs. The CPU orchestrates the inference tasks to be executed according to the concept of data flow, determining different user groups and input prompts for different user groups. The CPU can orchestrate tasks according to different requirements, such as maximum throughput and minimum latency. Maximum throughput refers to the maximum total number of tokens obtained from the segmentation of user input prompts processed by the entire inference device within a certain period. Therefore, when orchestrating tasks based on maximum throughput, more users (e.g., up to 256 users) should be packaged into a batch, i.e., packaged into user groups. The segmentation results of each user's input prompts are distributed to each user group, ensuring continuous data flow and enabling parallel execution of the inference tasks corresponding to the user groups. Minimum latency means that the overall latency of single-user inference is less than the specified requirement. When orchestrating tasks based on minimum latency, it is generally handled as a single user, that is, one user's input prompt is sent at a time, and one user's inference task is executed. Or it can be understood as a user group including one user. In addition, there is maximum throughput under the specified latency requirement, which requires estimating the batch size based on the latency (for example, it can be 32, that is, a user group includes 32 users), and packaging users to realize the distribution of inference tasks. The CPU performs word segmentation on the orchestrated input prompt of user group 1 to obtain the first group of word segmentation results, as shown in Figure 12. The CPU sends the word segmentation results of the first group corresponding to user group 1 to the first group of AI chips 11. The first group of AI chips 11 executes the inference task of user group 1, processes the first group of word segmentation results based on the stored first-level inference data, and obtains the first-level processing results corresponding to user group 1.

[0079] Next, as shown in Figures 11 and 13, the CPU segments the input prompts of user group 2 and sends them to the first AI chip 11. The first AI chip 11 then passes the processing results of user group 1 to the second AI chip 12. The first AI chip 11 executes the inference task of user group 2, and the second AI chip 12 executes the inference task of user group 1. This process continues until, as shown in Figures 11 and 14, the CPU segments the input prompts of user group N and sends them to the first AI chip 11. The first AI chip 11 then passes the processing results of user group N-1 to the second AI chip 12... The Nth AI chip 1N reports the processing results of user group 1 to the CPU. The CPU then detokenizes the processing results of user group 1 to obtain the generated text.

[0080] In summary, this application distributes the entire inference process, with each AI chip group handling a fixed level of inference tasks. This reduces the data storage capacity requirements and cost of each AI chip, thereby lowering the overall cost of the inference device. The AI ​​chips in the inference device are serially connected to complete the entire inference process, reducing the number of high-speed interfaces, simplifying connections, and minimizing connection resources, further reducing the cost of the inference device. Furthermore, data stream orchestration technology efficiently handles inference tasks for a large number of users, improving performance and the data throughput of the inference device.

[0081] In an exemplary embodiment, an inference system is provided, which includes any of the possible inference devices described in the above-described inference device embodiments. For example, the inference system includes two of the above-described inference devices, one inference device for performing inference tasks in the pre-filling stage and the other inference device for performing inference tasks in the decoding stage.

[0082] In an exemplary embodiment, an inference method is provided, which is applied to an inference device. The inference device includes N sets of artificial intelligence (AI) chips, which correspond to N levels of large model inference, where N is greater than 1. The AI ​​chips in the N sets of AI chips that correspond to adjacent levels are connected in series, and each set of AI chips stores the inference data of the corresponding level. As shown in FIG15, the method includes: S1101, receiving input data of the i-th level through the i-th set of AI chips, processing the input data according to the inference data of the i-th level, obtaining the processing result of the i-th level, and outputting the processing result of the i-th level. The processing result of the i-th level is used to generate an inference result, where i includes a positive integer greater than or equal to 1 and less than or equal to N. Wherein, when i is greater than 1 and less than or equal to N, the input data of the i-th level includes the processing result of the (i-1)-th level.

[0083] In one possible implementation, the AI ​​chip includes a memory and an AI core, the capacity of which is less than or equal to a capacity threshold; the method further includes: storing inference data of the i-th level through the memory in the i-th group of AI chips; reading the inference data of the i-th level through the AI ​​core in the i-th group of AI chips; receiving input data of the i-th level through the i-th group of AI chips, processing the input data according to the inference data of the i-th level, obtaining the processing result of the i-th level, and outputting the processing result of the i-th level, including: receiving input data of the i-th level through the AI ​​core in the i-th group of AI chips, processing the input data according to the inference data of the i-th level, obtaining the processing result of the i-th level, and outputting the processing result of the i-th level.

[0084] In one possible implementation, the inference device further includes a first processor connected to a first group of AI chips; the method further includes: transmitting first-level input data to the first group of AI chips via the first processor.

[0085] In one possible implementation, the method further includes: acquiring M sets of input prompts through a first processor, segmenting the M sets of input prompts into words to obtain M sets of word segmentation results, and generating M first-level input data including the M sets of word segmentation results, wherein the j-th first-level input data includes the j-th set of word segmentation results, and j is a positive integer greater than or equal to 1 and less than or equal to M; transmitting the first-level input data to the first group of AI chips through the first processor, including: transmitting the M first-level input data to the first group of AI chips sequentially through the first processor.

[0086] In one possible implementation, processing the input data based on the inference data of the i-th level to obtain the processing result of the i-th level and outputting the processing result of the i-th level includes: processing the j-th i-th level input data based on the inference data of the i-th level through the i-th group of AI chips to obtain the j-th i-th level processing result and outputting the j-th i-th level processing result; the method further includes: using the i-th group of AI chips, when j is less than M, after outputting the j-th i-th level processing result, processing the (j+1)-th i-th level input data based on the inference data of the i-th level to obtain the (j+1)-th i-th level processing result and outputting the (j+1)-th i-th level processing result.

[0087] In one possible implementation, the inference device further includes a second processor connected to one or more of the N AI chips; the method further includes transmitting corresponding inference data to the N AI chips via the second processor.

[0088] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk).

[0089] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items with substantially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor does it limit the quantity or order of execution. It should also be understood that although the following description uses the terms "first," "second," etc., to describe various elements, these elements should not be limited by the terms. These terms are merely used to distinguish one element from another.

[0090] It should also be understood that, in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0091] In this application, the term "at least one" means one or more, and the term "multiple" means two or more. For example, multiple second devices means two or more second devices. The terms "system" and "network" are often used interchangeably herein.

[0092] It should be understood that the terminology used in the description of the various examples herein is for the purpose of describing particular examples only and is not intended to be limiting. As used in the description of the various examples and the appended claims, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0093] It should also be understood that the term "and / or" as used herein refers to and covers any and all possible combinations of one or more of the associated listed items. The term "and / or" describes an association between related objects, indicating that three relationships can exist; for example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " in this application generally indicates that the preceding and following related objects are in an "or" relationship.

[0094] It should also be understood that the terms “if” and “if” can be interpreted as meaning “when” or “upon”, or “in response to determination” or “in response to detection”. Similarly, depending on the context, the phrases “if determination…” or “if detection [the stated condition or event]” can be interpreted as meaning “when determination…”, or “in response to determination…”, or “when detection [the stated condition or event]” or “in response to detection [the stated condition or event]”.

[0095] The above description is merely an embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A reasoning device, characterized in that, The inference device includes N sets of artificial intelligence (AI) chips, which correspond to N levels of large model inference, where N is greater than 1. The AI ​​chips in the N sets of AI chips that correspond to adjacent levels are connected in series, and each set of AI chips stores the inference data of the corresponding level. The i-th AI chip is used to receive input data from the i-th level, process the input data according to the inference data of the i-th level, obtain the processing result of the i-th level, and output the processing result of the i-th level. The processing result of the i-th level is used to generate inference results. The i-th level includes a positive integer greater than or equal to 1 and less than or equal to N. Where i is greater than 1 and less than or equal to N, the input data of the i-th level includes the processing results of the (i-1)-th level.

2. The reasoning device according to claim 1, characterized in that, The AI ​​chip includes a memory and an AI core, wherein the capacity of the memory is less than or equal to a capacity threshold. The memory in the i-th AI chip group is used to store the inference data of the i-th level; The AI ​​core in the i-th group of AI chips is used to read the inference data of the i-th level, receive the input data of the i-th level, process the input data according to the inference data of the i-th level, obtain the processing result of the i-th level, and output the processing result of the i-th level.

3. The reasoning device according to claim 2, characterized in that, The memory in the AI ​​chip is static random access memory (SRAM).

4. The inference device according to any one of claims 1-3, characterized in that, The inference device further includes a first processor, which is connected to a first group of AI chips; The first processor is used to transmit the first-level input data to the first group of AI chips.

5. The reasoning device according to claim 4, characterized in that, The first processor is further configured to acquire M sets of input prompts, segment the M sets of input prompts into words to obtain the M sets of word segmentation results, and generate M first-level input data including the M sets of word segmentation results, wherein the j-th first-level input data includes the j-th set of word segmentation results, and j is a positive integer greater than or equal to 1 and less than or equal to M. The first processor is used to sequentially transmit the M first-level input data to the first group of AI chips.

6. The reasoning device according to claim 5, characterized in that, The i-th AI chip is used to process the j-th i-th level input data based on the inference data of the i-th level, obtain the j-th i-th level processing result, and output the j-th i-th level processing result. The i-th AI chip is further configured to, when j is less than M, after outputting the processing result of the j-th i-th level, process the input data of the (j+1)-th i-th level based on the inference data of the i-th level to obtain the processing result of the (j+1)-th i-th level, and output the processing result of the (j+1)-th i-th level.

7. The reasoning device according to any one of claims 4-6, characterized in that, The first processor is a central processing unit (CPU), an extended 8086 processor, an ARM architecture processor, or a digital signal processor (DSP).

8. The inference device according to any one of claims 1-7, characterized in that, The inference device further includes a second processor, which is connected to one or more of the N AI chips. The second processor is used to transmit corresponding inference data to the N groups of AI chips.

9. The reasoning device according to claim 8, characterized in that, The second processor is a CPU, an extended 8086 processor, an ARM architecture processor, or a DSP.

10. A reasoning system, characterized in that, The inference system includes a plurality of inference devices as described in any one of claims 1-9.

11. A reasoning method, characterized in that, The inference method is applied to an inference device, which includes N sets of artificial intelligence (AI) chips. The N sets of AI chips correspond to N levels of large model inference, where N is greater than 1. AI chips at adjacent levels within the N sets of AI chips are connected. The number of connections between the N sets of AI chips is less than the number of connections between the N sets of AI chips themselves. Each set of AI chips stores the inference data for its corresponding level. The method includes: The i-th AI chip receives input data from the i-th level, processes the input data based on the inference data from the i-th level, obtains the processing result from the i-th level, and outputs the processing result from the i-th level. The processing result from the i-th level is used to generate the inference result, where i includes a positive integer greater than or equal to 1 and less than or equal to N. Where i is greater than 1 and less than or equal to N, the input data of the i-th level includes the processing results of the (i-1)-th level.

12. The reasoning method according to claim 11, characterized in that, The AI ​​chip includes a memory and an AI core, wherein the capacity of the memory is less than or equal to a capacity threshold; the method further includes: The inference data of the i-th level is stored in the memory of the i-th AI chip group; The inference data of the i-th level is read through the AI ​​core in the i-th group of AI chips; The process of receiving input data from the i-th level through the i-th group of AI chips, processing the input data based on the inference data of the i-th level, obtaining the processing result of the i-th level, and outputting the processing result of the i-th level includes: The AI ​​core in the i-th group of AI chips receives the input data of the i-th level, processes the input data according to the inference data of the i-th level, obtains the processing result of the i-th level, and outputs the processing result of the i-th level.

13. The reasoning method according to claim 11 or 12, characterized in that, The inference device further includes a first processor, which is connected to a first group of AI chips; the method further includes: The first processor transmits the first-level input data to the first group of AI chips.

14. The reasoning method according to claim 13, characterized in that, The method further includes: The first processor acquires M sets of input prompts, segments the M sets of input prompts into words, obtains the M sets of word segmentation results, and generates M first-level input data including the M sets of word segmentation results. The j-th first-level input data includes the j-th set of word segmentation results, where j is a positive integer greater than or equal to 1 and less than or equal to M. The step of transmitting the first-level input data to the first group of AI chips through the first processor includes: The first processor sequentially transmits the M first-level input data to the first group of AI chips.

15. The reasoning method according to claim 14, characterized in that, The step of processing the input data based on the inference data of the i-th level to obtain the processing result of the i-th level and outputting the processing result of the i-th level includes: The i-th AI chip processes the j-th i-th level input data based on the inference data of the i-th level to obtain the j-th i-th level processing result and output the j-th i-th level processing result. The method further includes: Using the i-th AI chip, when j is less than M, after outputting the processing result of the j-th i-th level, the input data of the (j+1)-th i-th level is processed according to the inference data of the i-th level to obtain the processing result of the (j+1)-th i-th level, and the processing result of the (j+1)-th i-th level is output.

16. The reasoning method according to any one of claims 11-15, characterized in that, The inference device further includes a second processor, which is connected to one or more of the N groups of AI chips; the method further includes: The second processor transmits the corresponding inference data to the N AI chips.

Images