56 results about "Cache compression" patented technology

The invention discloses a question and answer reasoning method and device based on key value cache compression, equipment and a medium, and relates to the technical field of natural language processing, and the method comprises the steps: segmenting a cue word in a current question and answer task into a lexical sequence, and generating an initial key value cache of the lexical sequence; dividing the lexical element sequence into context lexical elements and tail end lexical elements corresponding to each layer based on a preset tail end window size of each attention layer of the target large language model; screening out keyword elements of each attention layer from the context lexical elements according to importance scores between key matrixes of the context lexical elements and query matrix mean values of the tail end lexical elements; removing key value pairs of lexical elements except the keyword elements in the initial key value cache to obtain a compressed key value cache; and generating a reasoning result corresponding to the compressed key value cache by using the target large language model. The high computing power consumption of the large language model caused by key value cache data increase is reduced, and the dependence of an existing key value cache compression method on a complete attention weight matrix is broken through.

The invention relates to the technical field of artificial intelligence and natural language processing, in particular to a key value cache compression and sparse attention calculation method and system for large language model reasoning, and the method comprises the steps: an offline calibration stage; the online reasoning stage comprises the following steps: a pre-filling step; an autoregression generation step: for each newly generated lexical element, projecting a current query vector Q and a key vector K in a key cache to a low-dimensional space to obtain Q'and K '; calculating an approximate attention score based on Q'and K ', and selecting an index I of the first k most relevant lexical elements which are ranked from high to low; and calculating an accurate attention score based on Q and K [I], and calculating with the value vector V [I] to obtain the output of the current lexical element. According to the scheme, the memory and calculation bottleneck of large model reasoning in a scene of long text sequence input are solved, and the method has the advantages of reducing video memory occupation and calculation complexity at the same time.

The invention belongs to the technical field of large language models, and particularly relates to a large language model KV cache compression method based on interlayer fusion. The method comprises the following steps: inputting a to-be-processed text into a large language model to obtain a KV cache; the compression ratio of each layer of the large model is calculated, and preliminary SVD compression is carried out on KV cache of different layers of the large model; the KV cache subjected to preliminary compression is subjected to block division, and a plurality of blocks are obtained; calculating the attention score of each block; s blocks with the highest attention score are selected, and SVD reconstruction is carried out on the selected blocks; constructing a cache pool to store the reconstructed S blocks; dynamically updating the cache pool; splicing the S blocks in the cache pool to obtain a KV cache, and completing the compression of the KV cache; according to the method, the model is ensured to maintain stable prediction accuracy under diversified task scenes such as text generation and question and answer systems, and balance between storage efficiency and performance is realized.

The invention relates to the technical field of artificial intelligence and big language model reasoning optimization, and discloses a big language model-oriented adaptive KV cache compression method and system, and the method comprises the steps: constructing a lexical element importance measurement mechanism; analyzing an attention head distribution structure in large language model reasoning, and constructing a plurality of pruning strategies; based on a lexical element importance measurement mechanism and the attention head distribution structure, designing a self-adaptive key value cache compression hybrid strategy set based on a pruning strategy; constructing a static self-adaptive key value cache compression method, and automatically distributing a key value cache compression strategy in a pre-filling stage of large language model reasoning; and in a decoding stage, performing adaptive compression on the key value cache based on the allocated key value cache compression strategy. According to the method, the key value cache can be efficiently compressed on the premise of not depending on explicit attention score calculation, a system-level reasoning optimization framework is compatible, the generation performance is kept, meanwhile, the video memory consumption is remarkably reduced, and the long context reasoning capability is enhanced.

The invention relates to the technical field of artificial intelligence and natural language processing, in particular to a KV cache compression and eviction lexical element recovery method and system for large language model reasoning, and the method comprises the steps: for a query vector of an ith lexical element calculated by a current Transform layer, calculating attention scores of the query vector and all key vectors; executing a V cache dynamic updating operation based on the attention score; and for the ith lexical element, after the V cache dynamic updating operation is completely completed, performing attention calculation by using the updated value vector set in the V cache storage pool and the pre-calculated attention score partial product P. According to the technical scheme, the technical problem that in the prior art, collaborative optimization of video memory occupancy and calculation efficiency is difficult is solved, and the method has the advantages that KV cache video memory occupancy is dynamically managed, the model reasoning quality is kept, and the calculation efficiency is improved.

The invention provides a key value cache compression method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining an attention score between an input current lexical element and a target lexical element set, and the target lexical element set comprises all historical lexical elements from a sequence starting position to a current position and the current lexical element; performing aggregation processing on the plurality of attention scores historically accumulated by each target lexical element to generate an aggregation attention value corresponding to each target lexical element; dynamically determining a cache merging threshold according to the sorting result of the aggregation attention values of the target lexical units; recognizing the lexical element fragments with the aggregation attention values continuously lower than the cache merging threshold value and adjacent positions as target cache fragments; and executing fusion compression on the key value cache corresponding to the target cache fragment and updating the original cache. According to the method and the device, occupation of key value cache on video memory resources can be effectively reduced, and meanwhile, the semantic retention capability is remarkably improved.

The invention discloses a cascade speculation inference method and system based on hierarchical decline KV cache compression, and the method comprises the steps: firstly inputting a context prompt text into a target model for coding, generating a KV cache, and calculating an attention score between tokens; secondly, descending sorting is carried out based on the attention scores of the final input token, KV cache blocks corresponding to the first k attention scores are selected as cascading middle layer cache, a lightweight large language model is loaded to serve as a draft model, and a hierarchical decline KV cache compression strategy is adopted to maintain draft model cache. Then, based on all the caches, a double-layer cascade speculation reasoning framework is constructed, a target reasoning path is obtained, and the caches are updated; and finally, repeating the operation until target response data corresponding to the context prompt text is output according to the target reasoning path. According to the method, the KV cache proportion is reduced, meanwhile, the draft token acceptance rate of the target model of the full KV cache is improved, and reduction of precision is reduced.

The invention discloses a KV Cache compression and hierarchical management method and device for RAG acceleration and a medium, and belongs to the technical field of large model reasoning acceleration. In order to solve the problems that in an existing RAG KV Cache management technology, storage occupation is too large, and long-term importance and instant accessibility of data cannot be considered at the same time, the invention provides a hierarchical management strategy, and according to the method, a multi-dimensional popularity vector containing global popularity, session popularity and context popularity is defined for a KV Cache block. The method is characterized in that the compression precision of a cache block is independently determined by using global popularity so as to balance fidelity and performance; and meanwhile, the session heat and the context heat are used for independently determining the placement positions in heterogeneous hierarchies such as a GPU (Graphics Processing Unit), a CPU (Central Processing Unit) fixed memory and a CPU paging memory. According to the method, placement is guided through the real-time dimension, compression is guided through the importance dimension, context intelligent prefetching is achieved, and the reasoning performance of an RAG system, especially in a session and in a context pursuit scene is remarkably improved.

The invention relates to a KV cache compression method based on attention alignment, and belongs to the technical field of large language models. Comprising the following steps: a target model self-generates dialogue data reply, and constructs training data; adding a soft token into a word list of the target model, and carrying out soft token random initialization; finely adjusting specified parameters, splicing soft tokens at the tail of an original input sequence, and transmitting the soft tokens to the model; and respectively calculating the soft token and the attention distribution of the self-generated reply, calculating the mean square error of the soft token and the attention distribution of the self-generated reply as a loss function, and completing training. According to the method, a limited number of soft tokens playing an auxiliary role are introduced into original input, and importance discrimination of KV cache elements is realized and expelling is completed by aligning attention distribution of the soft tokens and real generated content; according to the method, better compromise is achieved between sequence length compression and performance loss, and the loss of model performance is better controlled while it is guaranteed that KV cache video memory space occupation is reduced.

Provided is an accelerator for accelerating batch large language model (LLM) inference via key-value cache compression. An accelerator, according to one embodiment, may comprise a plurality of compute cores. Here, each of the plurality of compute cores may comprise a plurality of processing units for processing an LLM inference operation on a per-token basis, and a direct memory access unit for managing operations of reading weights from a memory and writing key-value activation data back to the memory. In addition, the direct memory access unit may comprise a compression engine for managing online compression of the key-value activation data when the key-value activation data is written to the memory, a decompression engine for decompressing the compressed key-value activation data retrieved from the memory, and a memory management unit for managing reading and writing of the compressed key-value activation data in the memory.

The invention discloses a mixed precision parallel compression method for optimizing large model key value cache, which combines the advantages of mixed precision key value cache compression with an advanced system optimization technology. Based on the characteristic that key value pairs needing high-precision retention in mixed precision compression are the same as key value pairs used for attention calculation in a prefetching strategy, low-precision KV caches are stored in a GPU memory, and meanwhile predicted high-precision important KV caches are dynamically prefetched from a CPU memory according to needs. The technical problems that an existing multi-head attention mechanism-based method is incompatible with an existing pre-training model and cannot be directly applied to a closed source or a fine-tuned large model, and generalization of the method is reduced can be solved, and the situation that an existing pruning-based method is prone to deleting unimportant marks in the current stage can be solved. And contextual information loss is caused.

The invention provides a cache compression method and system, electronic equipment and a storage medium. The method comprises the following steps: generating an observation token sequence and a candidate set according to a first text in a buffer area and a second text in a cache; generating first attention mechanism information of the observation token and second attention mechanism information of the candidate token; determining an attention importance score according to the first attention mechanism information of the observation token and the second attention mechanism information of the candidate token; determining redundancy scores of the candidate tokens according to the second attention mechanism information of the candidate tokens; a target candidate token is screened out from all the candidate tokens according to a comprehensive score calculated by the attention importance score and the redundancy score of the candidate tokens, and a to-be-compressed token sequence is generated according to the target candidate token and the observation token; and performing compression caching according to the to-be-compressed token sequence to obtain a compressed text.

The invention relates to a large language model long context cache compression method, system and device and a medium, and the method comprises the steps: obtaining a long text data set as verification data, inputting a large language model, calculating key value cache, executing Fourier projection and expansion operation, and calculating data reconstruction loss of each feature dimension, identifying key value dimensions insensitive to long text context information; in a text data pre-filling reasoning stage, finite-order Fourier projection compression is performed on key value cache data in a GPU memory, and marked data with insensitive dimensions are compressed from an original storage mode of linearly increasing along with the text length to a fixed-length frequency spectrum for representation; in the decoding reasoning stage, inverse projection expansion is directly carried out on the compressed Fourier projection coefficient in a GPU memory to restore the insensitive dimension by customizing a kernel operator in the GPU, and the attention weight is calculated. Compared with the prior art, the method has the advantages of reducing the storage overhead of the large language model and the like.

The present disclosure relates to the technical field of artificial intelligence and natural language processing, in particular to a key-value cache compression and sparse attention calculation method and system for large language model inference. The key-value cache compression and sparse attention calculation method for large language model inference comprises: an offline calibration stage; an online inference stage, comprising: a pre-filling step; an autoregressive generation step, for each newly generated word element: projecting the current query vector Q and the key vector K in the key cache into a low-dimensional space to obtain Q' and K'; calculating the approximate attention score based on Q' and K', and selecting the top k most relevant word element indexes I in descending order; calculating the accurate attention score based on Q and K[I], and calculating the output of the current word element with the value vector V[I]. The above scheme solves the memory and calculation bottleneck of large model inference in the scenario of long text sequence input, and has the advantages of reducing the memory occupation and the calculation complexity at the same time.

The invention provides a low-rank-quantization mixed key value cache compression method and system, and belongs to the technical field of large language models, and the method comprises the steps: initializing a reasoning process and a KV cache structure, and loading low-rank projection parameters and quantization configuration of each layer / header; if calibration is not completed, off-line calibration is triggered to determine a low-rank base Pr, and a calibration result is solidified for subsequent reasoning; the Key / Value of the new token is received; acquiring kt and vt of the current layer when a new token is generated each time; whether the token falls into a recent window L or not is judged, and if yes, kt and vt of the token are directly written into a recent area with high precision; and if not, carrying out asymmetric compression. According to the invention, the end-to-end throughput is improved; the two-area cache reduces the memory access overhead in the decoding period, has good adaptability to multi-layer multi-head and different models / hardware, does not need to change the model structure, and can be smoothly integrated with the existing reasoning parallelization / scheduling strategy.

The invention discloses an inference model key value cache compression method based on supervised training and reinforcement learning, and the method comprises the following steps: S1, building a dynamic key value cache elimination mechanism in long text generation: judging the importance of each key value pair in an inference process through a key value pair scoring device, and selecting an important key value pair in each step for retention; s2, supervised training: enabling the scoring model to have the capability of identifying key value pairs; and S3, reinforcement learning: eliminating and modeling the key value cache in the reasoning process into a Markov decision process, and optimizing an elimination strategy by adopting a reinforcement learning algorithm. According to the application, the currently reserved key value cache is regarded as a state, the discarding of the key value cache is regarded as an action, and the loss of the model on the sequence is regarded as an award. And optimization is carried out through a GRPO algorithm to maximize rewards, so that a large model and a key value cache elimination strategy are adapted to each other, and better performance is obtained on a long reasoning task.

Systems and methods for tensor cache compression and / or decompression are disclosed. An example method includes receiving a tensor or dynamically generated data. The example method includes compressing the tensor (or the dynamically generated data) by applying a compression scheme to values of the tensor (or dynamically generated data). The example method also includes storing a compressed tensor into a tensor cache (or compressed dynamically generated data into a respective cache). The example method includes reading the compressed tensor from the tensor cache (or the compressed dynamically generated data from the respective cache), and decompressing the compressed tensor (or compressed dynamically generated data) by applying a decompression scheme to values of the compressed tensor (or compressed dynamically generated data). The example method further includes forwarding a decompressed tensor (or decompressed dynamically generated data) to a compute unit.

Key-value (KV) caching accelerates inference in large language models (LLMs) by allowing the attention operation to scale linearly rather than quadratically with the overall sequence length. Due to large context lengths in modern LLMs, the KV cache size can exceed the model size, negatively impacting throughput. To address this issue, a multi-granular clustering-based solution for KV cache compression can be implemented. Key tensors and value tensors corresponding to unimportant tokens can be approximated using clusters generated at different clustering levels with varying accuracy. Accuracy loss can be mitigated by using proxies generated at a finer clustering level for a subset of attention heads that are more significant.More significant attention heads can have a greater impact on model accuracy than less significant attention heads. Latency is improved by retrieving proxies from a faster memory for a subset of less significant attention heads when the impact on accuracy is smaller.

This invention relates to the field of artificial intelligence and large language model inference optimization technology, and discloses an adaptive key-value cache compression method and system for large language models. The method includes: constructing a lexical importance measurement mechanism; analyzing the attention head distribution structure in large language model inference and constructing multiple pruning strategies; designing an adaptive key-value cache compression hybrid strategy set based on the lexical importance measurement mechanism and the attention head distribution structure, and based on the pruning strategies; constructing a static adaptive key-value cache compression method to automatically allocate key-value cache compression strategies during the pre-filling stage of large language model inference; and adaptively compressing the key-value cache based on the allocated key-value cache compression strategies during the decoding stage. This invention can achieve efficient compression of the key-value cache without relying on explicit attention score calculation, and is compatible with system-level inference optimization frameworks, maintaining generation performance while significantly reducing memory consumption and enhancing long-context inference capabilities.

A method for operating a transformer model includes algorithmically allocating a fixed budget for a key-value cache between multiple decoding layers per an allocation scheme that ensures progressively higher decoding layers in the transformer model are allocated progressively smaller quantities of cache memory. The method further includes configuring each of the multiple decoding layers of the transformer model to retain no more than a maximum number of key-value vector pairs in the key-value cache during a token decoding operation, the maximum number of key-value vector pairs being independently determined for each decoding layer of the multiple decoding layers based on the cache memory that is allocated to the decoding layer.

The invention discloses a data caching device and method, and the device comprises a caching module which is used for caching compressed data; the compression processing module comprises a compression unit and a decompression unit, and the compression unit is used for performing compression processing on the original data to obtain compressed data when the original data passes the comparison verification and outputting the compressed data to the cache module for caching; the decompression unit is used for decompressing the compressed data acquired from the cache module; the decompression unit further comprises a verification subunit, the verification subunit is used for comparing and verifying original data to be written into the first position of the cache module with first decompressed data, and the first decompressed data is obtained by decompressing existing data obtained from the first position in the cache module through the decompression unit.

The invention discloses a rapid decoding method for an Activity-Bit code. Key value cache compression is a technology for optimizing memory usage and computational efficiency of a large language model in an inference process. With the increase of the sequence length, the key value cache is linearly increased, resulting in significant memory consumption and computational complexity. A simple and effective data compression technology is Active-Bit coding, matrix data is compressed before being written into a memory, then the compressed data is decoded before being loaded into a tensor multiplier, and a decoding method comprises the following steps: comparing coding with a limited number of constant arrays, and then, decoding the compressed data after the comparison with the limited number of constant arrays; a first part of high-activity bits E which are decoded through a full subtractor and through the highest borrow bit output of each subtractor; a proper difference value is selected from a first multiplexer based on a first part of high-activity bit E variables, then a corresponding low-activity bit is selected from a second multiplexer according to the difference value, and decoding is completed bit by bit or logically to obtain original data.

The present disclosure relates to the technical field of artificial intelligence and natural language processing, in particular to a KV cache compression and word piece recovery method and system for large language model inference, the method comprising: calculating the attention score of the query vector of the i-th word piece calculated by the current Transformer layer with all key vectors; performing a V cache dynamic update operation based on the attention score; for the i-th word piece, after the V cache dynamic update operation is completed, performing attention calculation using the updated value vector set in the V cache storage pool and the pre-calculated attention score partial product P. The above technical solution solves the technical problem that the existing technology is difficult to optimize the memory occupation and the calculation efficiency, and has the advantages of dynamically managing the KV cache memory occupation, maintaining the model inference quality, and improving the calculation efficiency.

The invention provides a large model multi-round dialogue fast state recovery method based on dynamic interlayer KV cache compression. The method comprises KV cache compression strategy search based on an attention mode, attention similarity calculation based on heterogeneous calculation and a KV cache recovery method based on calculation and storage fusion. The method has the advantages that the average precision loss of the dialogue is less than 1%, the precision loss of some specific tasks is reduced from 40% to 5%, meanwhile, the time delay of the first spitting of the model is reduced by 2 times, and the storage overhead of the KV cache is reduced by 2.5 times.

The invention relates to an adaptive budget allocation KV cache optimization method based on similarity, and the method comprises the steps: firstly providing an adaptive budget allocation KV cache expelling strategy, distributing a large budget for a head with uniform attention distribution, and strictly controlling the budget scale of a sparse head, so as to achieve the optimal budget allocation of the head; the overall budget constraint is effectively maintained, the reduction of the generation quality of a large language model is remarkably reduced, and on this basis, a similarity-based KV cache merging strategy is implemented to deal with the problem that the existing cache expelling strategy only retains part of high-importance KV cache pairs in the expelling process and does not retain part of high-importance KV cache pairs in the expelling process. And the problem of serious loss of context detail information caused by other secondary important KV cache pairs is solved, so that more reliable information restoration and retrieval capability is provided in a long context application scene, and the performance of a large language model on related tasks is improved. According to the method, under the condition that the cache budget is small, the KV cache compression loss of the model can be effectively reduced, the memory size needed by KV cache is reduced to the maximum degree, and the requirement for the environment is lower.

The invention discloses a large model KV cache compression method, system, product and equipment, and belongs to the technical field of artificial intelligence. The method comprises the following steps: S1, global low-rank approximation: capturing the overall redundancy of a KV matrix, and reducing the global storage overhead through low-rank decomposition; s2, local residual compensation: compensating local detail loss caused by global low-rank approximation, and accurately retaining details of a high-information-density region; s3, global-local complementary compensation: integrating redundant compression of global low-rank approximation and detail reservation of local residual compensation to form a complementary optimized KV cache; and S4, dynamic complementary management: dynamically adjusting parameters to optimize performance according to different tasks, sequence lengths or hardware environments. According to the method, the balance of global redundancy efficient compression and local detail accurate reservation can be realized.

The invention provides a cache structure, a cache method, a chip and electronic equipment, and relates to the technical field of cache. The cache structure comprises N sectors, wherein N is a positive integer; each sector comprises M cache lines; wherein the storage area of each cache line is divided into a first area and a second area, each cache line is associated with two groups of cache line status bits in 2M groups of cache line status bits, and the two groups of cache line status bits associated with each cache line are respectively used for indicating the states of the first area and the second area; a first sector address label used for indicating the first area, a second sector address label used for indicating the second area and 2M groups of cache line status bits are stored in the storage unit, and M is an integer greater than 1. In this way, the extra time overhead caused by cache data compression and decompression and the complexity of accessing the cache compressed data can be reduced, and therefore the overall performance of the cache compressed and decompressed data can be improved.

The invention discloses an electric energy data security storage method based on power failure detection and cache compression, and belongs to the technical field of intelligent single-chip microcomputer systems. The method specifically comprises the following steps: after electrification, detecting keg data, reading and storing all electric energy data, further embedding a pre-trained model, and constructing a transaction log; based on the electric energy data accumulated in real time, creating a timed task and setting a trigger condition, when the condition is triggered, executing a specific task and updating a transaction log; on the basis of a software-hardware combination mechanism, it is ensured that data can be stored in time and a transaction log can be updated when power failure is abnormal; and when the power is recovered, executing an initialization stage to recover data, further predicting a power failure reason by using a pre-trained model, and updating a transaction log. Through a software and hardware fusion algorithm technology, the problem of loss of the electric energy data under an abnormal condition is solved, and the safety of the electric energy data is ensured.

The key-value (LV) cache in this application accelerates inference in large language models (LLMs) by allowing attention operations to scale linearly, rather than quadratically, with the total sequence length. Since context lengths are long in modern LLMs, the KV cache size may exceed the model size, potentially negatively impacting throughput. To address this issue, a multi-granularity clustering-based scheme for KV cache compression is implemented. Clusters created at different clustering levels with variable precision approximate the key and value tensors corresponding to less important terms. Precision loss is reduced by using proxies generated at finer-grained clustering levels of subsets of more salient attention heads. More salient attention heads have a greater impact on model accuracy than less salient attention heads. When the impact on accuracy is low, latency is improved by retrieving proxies from a subset of less salient attention heads from faster memory.

The present application relates to a high-fidelity key cache compression method and system based on spectral quantization, comprising: step S1: spectral transformation and feature analysis process; receiving original key cache data from the model, mapping the original key cache data from the time domain to the frequency domain, and deeply analyzing the signal structure by using the energy concentration characteristics unique to the discrete cosine transform (DCT); step S2: frequency domain hybrid quantization encoding process; by using the main frequency extraction mechanism, the high frequency pre-emphasis technology and the hybrid bit width allocation strategy, the frequency domain coefficient is compressed into a sparse representation with extremely low bit; step S3: fusion decoding and sparse accumulation process; by using the hardware-aware fusion operator and the delay inverse transform technology, the attention score is calculated directly in the compression domain. The present application realizes the limit compression and efficient inference under the lossless precision.