Large language model inference apparatus based on compute-in-memory, inference system, and electronic device

Abstract

The present application relates to the field of artificial intelligence, and discloses a large language model (LLM) inference apparatus based on compute-in-memory, an inference system, and an electronic device. The inference apparatus comprises: a storage layer at least used for storage; and a computation layer at least used for computation. The computation layer and the storage layer are stacked by means of hybrid bonding. The computation layer comprises a neural network accelerator based on compute-in-memory. The neural network accelerator comprises an in-memory computing matrix. The in-memory computing matrix is used for performing neural network computation on input feature data and weights from the storage layer. The computation layer is further used for being electrically connected to a main control chip that controls the inference apparatus. The computation layer is further used for performing prefill processing of LLM inference and transmitting data, which is obtained after the prefill processing, to the main control chip for decoding processing of LLM inference, such that the prefill processing is separated from the decoding processing. The inference apparatus provided by the present application supports high bandwidth, has high computational power and low power consumption, and can also resolve the heat dissipation problem of existing LLM inference apparatuses.

Description

In-store computing-based large language model reasoning device, reasoning system and electronic equipment

[0001] This application claims priority to Chinese Patent Application No. 202411873249.3, filed on December 18, 2024, entitled "In-store Computing Large Language Model Reasoning Device, Reasoning System and Electronic Device", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence technology, and in particular to a large language model reasoning device, reasoning system and electronic device based on in-memory computing. Background Technology

[0003] Large Language Model (LLM) inference devices are a major direction for the future application of AI, especially edge-side large language model inference chips, which are important devices for the future application of AI and have a very broad development prospect in fields such as AI laptops, AI mobile phones, and intelligent robots.

[0004] Large language models are natural language processing models based on deep learning technology. With the rapid development of AI large language model inference devices and the increasing societal demand for them, the deployment of edge AI large language model inference devices faces numerous difficulties and challenges. For example, as edge large models rapidly develop, their parameter count increases with iteration, leading to a continuous increase in bandwidth and computing power requirements. Furthermore, processing large amounts of data generates significant power consumption and heat, necessitating increasingly lower power consumption and urgent solutions to heat dissipation issues. This ensures that edge AI large language model inference devices can meet the growing functional demands of users.

[0005] In summary, with the development of artificial intelligence, the inference devices of AI large language models increasingly require high bandwidth, high computing power, low power consumption, and good heat dissipation. Summary of the Invention

[0006] The purpose of this application is to provide a large language model reasoning device, reasoning system, and electronic device based on in-memory computing to solve some or all of the above-mentioned technical problems.

[0007] To achieve the above objectives, this application provides a large language model inference device based on in-memory computing, comprising: a storage layer for at least storage; a computation layer for at least computation, wherein the computation layer is stacked with the storage layer via hybrid bonding; the computation layer includes an in-memory computing-based neural network accelerator, the neural network accelerator including an in-memory computation matrix, the in-memory computation matrix being used to perform neural network computation on input feature data and weights from the storage layer; the computation layer is also used to be electrically connected to a main control chip controlling the inference device, and the computation layer is also used to perform pre-filling processing for large language model inference and to transmit the pre-filled data to the main control chip for decoding processing of large language model inference, thereby separating the pre-filling processing and the decoding processing.

[0008] In one embodiment, the in-memory computation matrix includes: N column storage operator modules, each column storage operator module including M rows of storage units for storing and performing computations on the stored data and input feature data.

[0009] In one embodiment, each computing unit includes: an SRAM memory for storing weights from the storage layer, and a logic unit for computing disposed adjacent to the SRAM memory.

[0010] In one embodiment, the in-memory computation matrix includes: a weight storage array, a bit multiplier, a storage readout circuit, and a logic operation unit; the weight storage array is used to store weights; the bit multiplier is used to receive a weight readout enable signal and input feature data, and when the weight readout enable signal is enabled, it selects a weight in the weight storage array according to the weight readout enable signal, and multiplies the bit of the input feature data with the selected weight; the storage readout circuit is used to read the product of the multiplication to the logic operation unit when the bit of the input feature data is 1, and does not perform a readout operation when the bit of the input feature data is 0; the logic operation unit is used to accumulate the product read out by the storage readout circuit when the bit of the input feature data is 1, so as to realize the multiplication and accumulation of the input feature data and the weight.

[0011] In one embodiment, the bit multiplier includes N AND gates that are respectively connected to the weight storage array; one input of each AND gate is used to receive the input feature data, another input is used to receive the weight read enable signal, and the output is used to output a control signal.

[0012] In one embodiment, the storage layer is also used for computation to perform a first-level computation of the neural network computation, and the computation layer completes the neural network computation based on the result of the first-level computation.

[0013] In one embodiment, the storage readout circuit further includes an AND gate; one input terminal of the AND gate is used to receive the input feature data, the other input terminal is used to receive the weight readout enable signal, and the output terminal is used to output a first control signal.

[0014] In one embodiment, the storage layer includes at least two DRAM memory layers, and the at least two DRAM memory layers are connected via TSV.

[0015] This application also provides an inference system, which includes the large language model inference device described in any of the above claims, and a main control chip electrically connected to the large language model inference device. The main control chip is used to perform large language model inference decoding processing on the data after the computation layer pre-filling processing in the large language model inference device.

[0016] This application also provides an electronic device, including the large language model reasoning device described in any of the above claims.

[0017] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0018] The large language model inference device based on in-memory computing provided in this application stacks its storage layer and computation layer, which includes an in-memory computing-based neural network accelerator, using a hybrid bonding method. This reduces data movement, energy loss during signal transmission, and time delay during signal propagation, resulting in lower power consumption, less heat generation, and higher computing power. Furthermore, it performs pre-filling and decoding processes in LLM inference separately in the aforementioned computation layer and the main control chip. This fully utilizes the high-computing-power neural network accelerator of this invention, allowing the computationally intensive pre-filling process to be executed in the computation layer. This supports large language model inference devices requiring high bandwidth and improves the efficiency of large language model inference. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 is a schematic diagram of the structure of a large language model reasoning device and a large language model reasoning system provided in an embodiment of this application.

[0021] Figure 2 is a schematic diagram of a neural network accelerator module provided in an embodiment of this application.

[0022] Figure 3 is a schematic diagram of a column of stored operator modules in an in-memory computation matrix in one embodiment of this application.

[0023] Figure 4 is a schematic diagram of the vertical structure of a column of stored operator modules in an in-memory computation matrix in one embodiment of this application.

[0024] Figure 5 is a schematic diagram illustrating the principle of in-memory computation matrix calculation in one embodiment of this application.

[0025] Figure 6 is a schematic diagram of the in-memory matrix calculation module in one embodiment of this application.

[0026] Figure 7 is a schematic diagram of the structure of an in-memory computation matrix according to an embodiment of this application. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] The purpose of this application is to provide a large language model reasoning device, reasoning system, and electronic device based on in-memory computing to solve some or all of the above-mentioned technical problems.

[0029] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0030] Inference in a large language model mainly consists of two phases: the prefill phase, which processes input tokens in parallel; and the decoding phase, which generates the next token sequentially. These two phases are repeated until an EOS (End-Of-Sequence) token is generated or a user-defined stopping condition is met. In other words, prefilling provides the model with the user's input as a prompt (initial information). This prompt can be seen as the model's initial information, guiding subsequent generation. Because the prefill phase can be processed in parallel, multiple inputs can be processed simultaneously, improving inference efficiency. Parallel computation allows for faster prefilling operations, reducing waiting time.

[0031] The applicant of this application discovered that the pre-filling stage in large language model inference has a large computational load, with the computational power requirement exceeding the bandwidth requirement; while the computational power requirement of the decoding stage in large language model inference is significantly reduced, its bandwidth requirement is significantly increased. Taking a typical edge application scenario as an example: the LIama3-8B model, int8 quantization, first token time = 1s, decoding stage 100 tokens / s. The computational power and bandwidth requirements of its different stages are shown in Table 1:

[0032] Table 1. Computing power and bandwidth requirements of the LIama3-8B model (int8) at different stages.

[0033] As shown in Table 1, in the application of edge-side Large Language Model (LLM) inference chips, the prefill stage has a large computational load, with the computational power requirement exceeding the bandwidth requirement. Conversely, in the decode stage, the computational power requirement decreases significantly, but the bandwidth requirement increases dramatically to 800GB / s. This is because the input in the decode stage becomes a vector, and the calculation changes from matrix multiplication to vector-matrix multiplication. Furthermore, since edge-side LLM inference spends over 90% of its time in the decode stage, it is memory-intensive, and its inference efficiency is limited by bandwidth rather than computational power. Due to the huge bandwidth requirement of the decode stage, existing GPUs and LPDDR-based edge chips cannot meet the deployment bandwidth requirements of edge-side LLM. Moreover, the amount of data that LLM inference devices need to infer is increasing, leading to higher power consumption. In existing technologies, the computational components in LLM inference devices are far from the heat sink, making it difficult to effectively dissipate the heat generated by these components. This presents a challenge for LLM inference devices in terms of heat dissipation. Existing technologies typically improve the heat dissipation capacity of the heat sink, but the improvement effect is still unsatisfactory.

[0034] The large language model inference device based on in-memory computing provided in this application can solve the problems of computing power, bandwidth, power consumption and heat dissipation.

[0035] To illustrate the technical solutions described in this application, specific embodiments are provided below, which only show the parts related to the embodiments of this application.

[0036] Referring to Figure 1, the large language model inference device based on in-memory computing provided in this application includes: a stacked storage layer (illustrated as DRAM Layers in the figure) and a computation layer (i.e., the bottom layer in the example shown in the figure). The storage layer is used for storage at least, and can be used to store parameters of a large model, such as storing 8GB of large model parameters, and storing weights used for neural network computation in the computation layer; the storage layer can also be used to store the output feature data after neural network computation. The computation layer is used for computation at least, and is stacked with the storage layer via hybrid bonding, and the computation layer and storage layer are electrically connected to realize data interaction between the storage layer and the computation layer. The computation layer includes an in-memory computing-based neural network accelerator (i.e., the NPU acceleration chip in the figure) for neural network computation. The computation layer may include multiple sub-NPU clusters. The neural network accelerator includes an in-memory computation matrix, which is used to perform neural network computation on the input feature data and weights from the storage layer. The computation layer is also used to electrically connect to the main control chip of the control inference device. The computation layer is also used to perform pre-filling processing for large language model inference and to transmit the pre-filled data to the main control chip for decoding processing of large language model inference, so as to separate the pre-filling processing and the decoding processing.

[0037] The storage layer and computing layer provided in this application can take many forms, such as chips or wafers.

[0038] The neural network accelerator included in the computation layer of this application achieves in-memory computing through an in-memory computation matrix, enabling neural network computation by combining input feature data and weights from the storage layer. In-memory computing is an innovative architecture designed to overcome the "memory wall" problem by integrating storage and computation functions onto a single chip. This technology embeds computational power into memory and employs a new operational architecture to perform matrix-vector multiplication and accumulation operations. Therefore, it reduces the frequent data transfer between memory and the computing chip in existing technologies, significantly reducing transfer time, power consumption, and heat generation, and improving the efficiency of parallel data processing. It achieves in-situ computation, eliminating bandwidth limitations and data movement costs. This fundamentally eliminates unnecessary data transfer latency and power consumption, improving AI computation efficiency by hundreds or thousands of times, reducing costs, and breaking down the "memory wall" and "power wall." Compared to existing devices performing computations under the same conditions, the computation layer and large language model inference device can have greater computing power, consume less power, and generate less heat, thus addressing existing heat dissipation problems at their source.

[0039] Furthermore, this application utilizes hybrid bonding to achieve a 3D stacked structure of the memory and compute layers in an LLM inference device. Hybrid bonding enables high-density, high-performance interconnects between different chips. Through hybrid bonding, the metal layers (typically copper layers) of two or more chips can be precisely aligned and directly pressed together to form direct electrical contacts. This allows for sub-micron and even nanometer-level interconnect spacing, enabling more connection points to be placed in a smaller area, significantly increasing the data communication bandwidth between chips. Its contact density can reach 10K-1MM / mm. 2 Hybrid bonding eliminates intermediate materials such as solder, resulting in lower resistance in direct copper-to-copper connections, reducing energy loss during signal transmission and minimizing signal propagation time delay. The compact structure and direct conductive path formed by hybrid bonding help improve thermal management and reduce heat generation, thus enabling higher integration, faster data transmission speeds, greater bandwidth, and lower power consumption.

[0040] Furthermore, the inference device provided in this application combines the advantage of its high computing power based on in-memory computing layer by separating prefill and decoding. That is, it completes the prefill, which requires high computing power, in the computing layer, and transmits the data processed by the computing layer through prefill to the main control chip electrically connected to it. The main control chip then decodes the prefilled data. This reduces the bandwidth requirement of the inference device, enabling it to perform large language model inference work faster and better. It also allows the large language model inference device with the same computing power to support greater bandwidth for large language model inference work.

[0041] In summary, the inference device provided in this application can not only solve the existing bandwidth bottleneck to support large bandwidth and large computing power, but also reduce power consumption in many ways, thus solving the existing heat dissipation problem.

[0042] Referring to the embodiment shown in Figure 2, the neural network accelerator includes: a preprocessing module, an in-memory computation matrix and vector processing module, and a shared memory electrically connected in sequence to the preprocessing module, the in-memory computation matrix and vector processing module, respectively; it also includes a chip controller electrically connected to the shared memory. The NPU is also electrically connected to an external memory, which in this application is the external memory shown in Figure 2. Data stored in the external memory is transferred to the NPU's shared memory in batches. For example, weights are read from the external memory in batches to the shared memory. One possible working method is briefly described as follows: The preprocessing module retrieves input feature data from shared memory, processes the input feature data into matrix data in in-memory computation matrix format, and inputs the matrix data into the in-memory computation matrix. Specifically, if the input feature data includes at least two input feature maps, and these at least two input feature maps are stored in shared memory using a skip-address method, then the preprocessing module retrieves the at least two input feature maps from the shared memory in address-sequential reading; or, if the input feature data includes at least two input feature maps, and these at least two input feature maps are stored in shared memory in a contiguous storage manner, then the preprocessing module retrieves the at least two input feature maps from the shared memory using a skip-address reading method; weight data is then obtained through the in-memory computation matrix. The system performs convolution calculations based on matrix data and weight data using an in-memory computation matrix to obtain the calculation result, which is then input into a vector processing module. The vector processing module performs vector processing on the calculation result to obtain output feature map data, which is then written to shared memory and / or external memory. If the output feature map data is an intermediate calculation result, it is used as input data for the next level of calculation. Before the preprocessing module obtains the input feature data and / or after the vector processing module processes the calculation result to obtain the output feature map data, the chip controller controls the data interaction between the shared memory and external memory. The data interacted includes the input feature data and / or the output feature map data.

[0043] In another embodiment of this application, the in-memory computation matrix includes: N columns of storage operator modules, each column of storage operator module including: M rows of storage units for storing and calculating the stored data with the input feature data, and capacitors electrically connected to the storage units, the charges output by the capacitors in the M rows are pooled together to achieve charge accumulation.

[0044] Referring to Figure 3, which shows a schematic diagram of a column of in-memory operator sub-modules in an embodiment of this application, in this embodiment, the in-memory operator unit can store and perform multiplication calculations on weights and input feature data. The corresponding product is accumulated through a capacitor connected to the in-memory operator unit to realize multiply-accumulate (MAC) calculations in neural network computation. That is, matrix calculations can be performed within the in-memory operator matrix, which avoids the frequent data transfer between two different and geographically distant memories and computing chips as in the prior art. The power consumption of multiply-accumulate calculations usually accounts for most of the power consumption of the entire computation, for example, often 50-70%. Based on an in-memory computing architecture, this application sets up a neural network accelerator with an in-memory operator matrix in the computing layer to realize two-dimensional and three-dimensional matrix multiplication / addition operations through a new computing architecture, thereby greatly reducing the power consumption of neural network computation. Moreover, it can make the area of ​​the neural network accelerator smaller, and can support greater computing power and generate less heat under the same area and power consumption, thus solving the heat dissipation problem.

[0045] Referring to Figures 4 and 5, Figure 4 shows a vertical structural diagram of a column of in-memory computation submodules in an embodiment of this application. Each in-memory computation unit includes: an SRAM memory (i.e., SRAM Memory in Figure 4) for storing weights from the memory layer, and a logic unit for computation arranged adjacent to the SRAM memory. Multiple in-memory computation units are stacked to form a column, and several columns of in-memory computation units form an in-memory computation matrix. The digital logic, i.e., the aforementioned logic unit, can be used to perform multiplication and addition calculations on the weights and input feature data in the SRAM memory. Figure 4 also shows a typical architecture block diagram of Integrated In-Memory Computing (CIMD). Specifically, it is based on SRAM memory, and model weights are stored in multiple on-chip SRAM Memory Banks. The multiply-accumulate (MAC) computation logic circuits required by CIMD are deployed near each SRAM Memory Bank to significantly reduce the latency and power consumption caused by weight handling. Compared to traditional von Neumann architecture AI chips, integrated in-memory computing (CIMD) significantly reduces power consumption during data transfer while dramatically increasing the speed of parallel multiply-accumulate (MAC) computations, effectively alleviating the two major challenges of the 'memory wall' and the 'power wall'. SRAM memory and the logic units used for computation are stacked adjacently, allowing for a more compact in-memory computing matrix structure, greater computing power, lower power consumption, and faster and more accurate computation when combined with digital logic units. As shown in Table 2, at the 12nm process node, the matrix multiplication energy efficiency ratio of the CIM in-memory computing unit can reach 24.7 TOPS / W, representing an order-of-magnitude improvement in energy efficiency compared to traditional digital architectures (i.e., general-purpose computing units, such as GPUs and TPUs). Integrated in-memory computing (CIMD) significantly reduces power consumption during data transfer while dramatically increasing the speed of parallel multiply-accumulate (MAC) computations. Thanks to this innovative underlying computing unit architecture, in-memory computing (CIMD) technology can significantly improve the energy efficiency of edge AI large model (LLM) inference chips, meaning that the computing power remains the same while the power consumption requirement is greatly reduced.

[0046] Table 2 Performance Comparison of Different Digital Architectures

[0047] Figure 5 is a schematic diagram illustrating the principle of in-memory computation matrix calculation in one embodiment of this application. The in-memory computation unit performs multiplication calculations on the input feature data and weights, and then accumulates the product through a multi-level addition tree to obtain the multiply-accumulated result.

[0048] Referring to Figure 6, in this embodiment, the in-memory computation matrix includes a weight storage array 10 (also called a weight parameter storage array), a bit multiplier 12, a storage readout circuit 21, and a logic operation unit 22. One input of the bit multiplier 12 is connected to a weight readout enable signal, and the other input is connected to input feature data; the weight acquisition end is connected to the weight storage array 10. The input of the storage readout circuit 21 is connected to the weight storage array 10, and its output is connected to the logic operation unit 22. The output of the logic operation unit 22 is used to output the convolution operation result. The weight storage array 10 is used to store weights.

[0049] Bit multiplier 12 receives the weight read enable signal and input feature data (in convolutional neural networks, the input feature map usually needs to be processed). When the weight read enable signal is enabled, it selects the weights in the weight storage array according to the weight read enable signal and multiplies the bits of the input feature data with the selected weights. The input feature data is used to represent the data output by the previous layer during the convolution operation and may include at least one bit, where the bit value is 1 or 0. For input feature data, a bit value of 1 represents that the input feature data is 1, and a bit value of 0 represents that the input feature data is 0. For example, when the bit of the input feature data is 1, the product of the bit of the input feature data and the corresponding weight is the corresponding weight. At this time, the storage readout circuit 21 can read the corresponding weight from the weight storage array 10 to realize the corresponding product readout. When the bit of the input feature data is 0, the storage readout circuit 21 does not perform a readout operation at this time, so the storage readout circuit 21 can directly perform a readout operation on the weight storage array 10 to realize the corresponding product readout. The storage readout circuit 21 reads the product of the multiplication to the logic operation unit 22 when the bits of the input feature data are 1, and does not perform the readout operation when the bits of the input feature data are 0, so that the storage readout circuit 21 does not generate power when the bits of the input feature data are 0, thereby reducing the power consumption of the storage readout circuit 21. Specifically, the storage readout circuit 21 can read the corresponding weight to the logic operation unit 22 when the bits of the input feature data are 1. Specifically, the storage medium of the storage readout circuit 21 includes any one of SRAM, DRAM and RRAM, and the storage medium can be a volatile storage medium or a non-volatile storage medium. The logic operation unit 22 is used to accumulate the product read out by the storage readout circuit 21 when the bits of the input feature data are 1, so as to realize the multiplication and accumulation of the input feature data and weights. The above-mentioned logic operation unit 22 only needs to output the final convolution operation result, without transmitting intermediate data in the convolution operation process, which can reduce the requirements of data output and data transmission bandwidth. In other words, it ensures that the storage read circuit generates no power when the input feature data bits are 0, thereby reducing the power consumption of the storage read circuit. The output data of the logic operation unit is the accumulation result, eliminating the need to transmit intermediate data during the convolution operation process, thus reducing the data bit width of the output data. If more complex accumulation operations are implemented in the logic operation unit, the output data bit width can be further compressed. Therefore, this application can reduce the power consumption of reading large amounts of concurrent data and lower the requirements for data output volume and data transmission bandwidth.

[0050] Referring to Figure 7, the bit multiplier includes N AND gates connected to the weight storage array respectively. One input of the AND gate is used to input the feature data fmi, the other input is used to input the weight read enable signal, and the output is used to output the control signal. The weight storage array 10 includes K rows and N columns of parameter storage units. Each parameter storage unit is used to store a weight or an electrical signal representing the corresponding weight. The h-th row and j-th column parameter storage unit stores the weight W(h, j), 1 ≤ h ≤ K, 1 ≤ j ≤ N, where K is the number of rows of weights and N is the number of columns of weights. The K rows of parameter storage units are correspondingly set on the K row bit lines; the h-th row bit line is denoted as bitline, h. The aforementioned weight read enable signal is used to enable one column of parameter storage units in the N columns of parameter storage units, that is, one weight read enable signal corresponds to one column of parameter storage units. For example, the weight read enable signal corresponding to the j-th column of parameter storage units can be denoted as wordline, j. When the corresponding weight read enable signal wordline (j=1) is received by the j-th column parameter storage unit, the electrical signal representing the weight is output to the corresponding bit line, so that the storage readout circuit 21 can read the corresponding weight signal through each row bit line. For example, the probability of an N-bit input vector being all zeros is much lower than the probability of some bits in an N-dimensional input vector being zero. The bit-level sparsity of data is very high. However, the smaller the granularity of data sparsity, the more difficult it is to use, mainly because it requires real-time calculation and judgment on each bit of the input data. Through the above method, this application can solve the problem of data sparsity. For the case where the data sparse matrix contains a large number of zero values, it can greatly reduce resource waste, reduce unnecessary computation, and improve storage space utilization.

[0051] In another embodiment of this application, the storage readout circuit further includes an AND gate; one input terminal of the AND gate is used to receive input feature data, the other input terminal is used to receive a weight readout enable signal, and the output terminal is used to output a first control signal; for the weight readout enable signal wordline,j and the input feature data fmi,j corresponding to the j-th column, when both the weight readout enable signal wordline,j and the input feature data fmi,j are 1, the first control signal is 1; when at least one of the weight readout enable signal wordline,j and the input feature data fmi,j is 0, the first control signal is 0. Its circuit is simple and can accurately control the read operation, reducing the power consumption of the storage readout circuit and the power consumption of the large language model inference device.

[0052] In another embodiment of this application, the storage layer is also used for computation to implement the first-level computation of the neural network. The computation layer completes the neural network computation based on the result of the first-level computation. That is, simple computations can also be performed in the storage layer, such as the first multiplication after multiplication parameters are calculated, i.e., the first-level computation. The result after the first-level computation is transmitted to the NPU for accumulation computation to complete the neural network computation. This can make full use of the storage layer, reduce the computational load of the computation layer, and thus reduce the power consumption of the computation layer. Moreover, after the first-level computation processing, the data transfer from the storage layer to the computation layer can be reduced, which can correspondingly reduce the demand for storage capacity and high bandwidth. The storage layer preferably includes DRAM memory, which has low cost and high density. The storage layer includes at least two layers of DRAM memory, and the at least two layers of DRAM memory are connected through silicon vias (TSVs). This allows multiple DRAM chips to be stacked vertically, significantly improving memory bandwidth and reducing power consumption; achieving large-capacity, high-bandwidth storage, and meeting the stringent memory requirements of high-performance computing, artificial intelligence, and other fields.

[0053] As shown in Table 3, the 3D stacked DRAM with integrated Hybrid Bonding provided in this application has a bandwidth of 1-60TB, which is a significant improvement over other advanced GDDR6 and HBM3E technologies. At the same time, its energy requirement for transmitting a single bit of data is also greatly reduced to 0.5pJ / bit, which greatly reduces the power consumption of data transfer. It can solve the bandwidth bottleneck problem encountered in the deployment of edge AI large model (LLM) chips, and at the same time, it significantly reduces the power consumption of data transfer.

[0054] Table 3 Comparison of bandwidth and power consumption of different technologies

[0055] In summary, this application provides a novel LLM inference device that combines 3D stacked DRAM, hybrid bonding technology, and high-efficiency SRAM in-memory computing, among other technologies. It addresses the current bottlenecks in bandwidth, power consumption, computing power, and heat dissipation encountered by edge AI large model (LLM) inference devices (which can take various forms such as chips or modules). It significantly improves memory bandwidth, reduces power consumption, and achieves large-capacity, high-bandwidth storage, meeting the stringent memory requirements of high-performance computing, artificial intelligence, and other fields. It can be widely applied to various edge devices, such as mobile phones, computers, and robots.

[0056] This application also provides an inference system (see Figure 1), which includes any of the aforementioned large language model inference devices and a main control chip electrically connected to the large language model inference device. The main control chip is used to perform decoding processing on the data after pre-filling processing in the computation layer of the large language model inference device for large language model inference. The structure of the NPU included in the computation layer is also briefly illustrated in Figure 1. The inference system provided by this application separates the pre-filling processing and decoding processing into different devices, and allows the pre-filling processing, which requires high computing power, to be processed in an SRAM-based in-memory computing structure, while the main control chip and other devices with relatively lower computing power perform the decoding. This allows the large language model inference system to significantly solve the existing bandwidth bottleneck problem, greatly reduce its power consumption, and alleviate heat dissipation issues. The main control chip can be various types of chips, such as GPUs, SOCs, etc.

[0057] This application also provides an electronic device including the aforementioned large language model inference device or inference system. It can significantly reduce the power consumption and cost of electronic devices, and improve the efficiency and experience of AI inference. It can be widely used in smartphones, tablets, wearable electronic devices, smart home electronic products, and so on.

[0058] The above are merely preferred embodiments of this application. Those skilled in the art will understand that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of this application, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of this application. Therefore, this application is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of this invention.

Claims

1. A memory-compute integrated-based large language model inference apparatus, characterized in that, The large language model reasoning device includes: At least a storage layer used for storage; At least a computing layer for computation, the computing layer being stacked with the storage layer via hybrid bonding, the computing layer being electrically connected to the storage layer for data interaction; The computing layer includes a memory-based neural network accelerator, which includes an in-memory computing matrix. The in-memory computing matrix is ​​used to perform neural network calculations on the input feature data and the weights from the storage layer. The computing layer is also used to electrically connect to the main control chip that controls the inference device. The computing layer is also used to perform pre-filling processing for large language model inference and to transmit the pre-filled data to the main control chip for decoding processing of large language model inference, so that the pre-filling processing and the decoding processing are separated.

2. The large language model inference apparatus according to claim 1, wherein The in-memory computation matrix includes: N columns of in-memory operator modules, each column of which includes M rows of in-memory units for storing and performing computations on the stored data and input feature data.

3. The large language model inference apparatus according to claim 2, wherein, Each of the memory units includes: an SRAM memory for storing weights from the memory layer, and a logic unit for computation disposed adjacent to the SRAM memory.

4. The large language model inference apparatus of claim 1, wherein, The in-memory computation matrix includes: a weight storage array for storing weights, a bit multiplier, a storage readout circuit, and a logic operation unit. The bit multiplier receives a weight readout enable signal and input feature data. When the weight readout enable signal is enabled, it selects a weight from the weight storage array according to the weight readout enable signal and multiplies the bit of the input feature data with the selected weight. The storage readout circuit reads the product of the multiplication to the logic operation unit when the bit of the input feature data is 1, and does not perform a readout operation when the bit of the input feature data is 0. The logic operation unit accumulates the product read out by the storage readout circuit when the bit of the input feature data is 1, so as to realize the multiplication and accumulation of the input feature data and the weight.

5. The large language model inference apparatus of claim 4, wherein, The bit multiplier includes N AND gates that are respectively connected to the weight storage array; one input of each AND gate is used to receive the input feature data, the other input is used to receive the weight read enable signal, and the output is used to output a control signal.

6. The large language model inference apparatus of claim 4, wherein, The storage readout circuit further includes an AND gate; one input terminal of the AND gate is used to receive the input feature data, the other input terminal is used to receive the weight readout enable signal, and the output terminal is used to output the first control signal.

7. The large language model inference apparatus according to any one of claims 1 to 5, characterized in that, The storage layer is also used for computation to implement the first-level computation of the neural network, and the computation layer completes the neural network computation based on the result of the first-level computation.

8. The large language model inference apparatus according to any one of claims 1 to 5, characterized by, The storage layer includes at least two DRAM memory layers, and the at least two DRAM memory layers are connected via TSV.

9. An inference system, characterized by The device includes a large language model inference apparatus as described in any one of claims 1-8, and a main control chip electrically connected to the large language model inference apparatus. The main control chip is used to perform large language model inference decoding processing on the data after the computational layer pre-filling processing in the large language model inference apparatus.

10. An electronic device, comprising: The large language model reasoning device includes any one of claims 1-8 or the reasoning system includes the one described in claim 9.

Images