42 results about "Cache compression" patented technology

The invention relates to the technical field of artificial intelligence and big language model reasoning optimization, and discloses a big language model-oriented adaptive KV cache compression method and system, and the method comprises the steps: constructing a lexical element importance measurement mechanism; analyzing an attention head distribution structure in large language model reasoning, and constructing a plurality of pruning strategies; based on a lexical element importance measurement mechanism and the attention head distribution structure, designing a self-adaptive key value cache compression hybrid strategy set based on a pruning strategy; constructing a static self-adaptive key value cache compression method, and automatically distributing a key value cache compression strategy in a pre-filling stage of large language model reasoning; and in a decoding stage, performing adaptive compression on the key value cache based on the allocated key value cache compression strategy. According to the method, the key value cache can be efficiently compressed on the premise of not depending on explicit attention score calculation, a system-level reasoning optimization framework is compatible, the generation performance is kept, meanwhile, the video memory consumption is remarkably reduced, and the long context reasoning capability is enhanced.

The invention provides a key value cache compression method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining an attention score between an input current lexical element and a target lexical element set, and the target lexical element set comprises all historical lexical elements from a sequence starting position to a current position and the current lexical element; performing aggregation processing on the plurality of attention scores historically accumulated by each target lexical element to generate an aggregation attention value corresponding to each target lexical element; dynamically determining a cache merging threshold according to the sorting result of the aggregation attention values of the target lexical units; recognizing the lexical element fragments with the aggregation attention values continuously lower than the cache merging threshold value and adjacent positions as target cache fragments; and executing fusion compression on the key value cache corresponding to the target cache fragment and updating the original cache. According to the method and the device, occupation of key value cache on video memory resources can be effectively reduced, and meanwhile, the semantic retention capability is remarkably improved.

The invention discloses a KV Cache compression and hierarchical management method and device for RAG acceleration and a medium, and belongs to the technical field of large model reasoning acceleration. In order to solve the problems that in an existing RAG KV Cache management technology, storage occupation is too large, and long-term importance and instant accessibility of data cannot be considered at the same time, the invention provides a hierarchical management strategy, and according to the method, a multi-dimensional popularity vector containing global popularity, session popularity and context popularity is defined for a KV Cache block. The method is characterized in that the compression precision of a cache block is independently determined by using global popularity so as to balance fidelity and performance; and meanwhile, the session heat and the context heat are used for independently determining the placement positions in heterogeneous hierarchies such as a GPU (Graphics Processing Unit), a CPU (Central Processing Unit) fixed memory and a CPU paging memory. According to the method, placement is guided through the real-time dimension, compression is guided through the importance dimension, context intelligent prefetching is achieved, and the reasoning performance of an RAG system, especially in a session and in a context pursuit scene is remarkably improved.

The invention relates to a KV cache compression method based on attention alignment, and belongs to the technical field of large language models. Comprising the following steps: a target model self-generates dialogue data reply, and constructs training data; adding a soft token into a word list of the target model, and carrying out soft token random initialization; finely adjusting specified parameters, splicing soft tokens at the tail of an original input sequence, and transmitting the soft tokens to the model; and respectively calculating the soft token and the attention distribution of the self-generated reply, calculating the mean square error of the soft token and the attention distribution of the self-generated reply as a loss function, and completing training. According to the method, a limited number of soft tokens playing an auxiliary role are introduced into original input, and importance discrimination of KV cache elements is realized and expelling is completed by aligning attention distribution of the soft tokens and real generated content; according to the method, better compromise is achieved between sequence length compression and performance loss, and the loss of model performance is better controlled while it is guaranteed that KV cache video memory space occupation is reduced.

Provided is an accelerator for accelerating batch large language model (LLM) inference via key-value cache compression. An accelerator, according to one embodiment, may comprise a plurality of compute cores. Here, each of the plurality of compute cores may comprise a plurality of processing units for processing an LLM inference operation on a per-token basis, and a direct memory access unit for managing operations of reading weights from a memory and writing key-value activation data back to the memory. In addition, the direct memory access unit may comprise a compression engine for managing online compression of the key-value activation data when the key-value activation data is written to the memory, a decompression engine for decompressing the compressed key-value activation data retrieved from the memory, and a memory management unit for managing reading and writing of the compressed key-value activation data in the memory.

The invention provides a cache compression method and system, electronic equipment and a storage medium. The method comprises the following steps: generating an observation token sequence and a candidate set according to a first text in a buffer area and a second text in a cache; generating first attention mechanism information of the observation token and second attention mechanism information of the candidate token; determining an attention importance score according to the first attention mechanism information of the observation token and the second attention mechanism information of the candidate token; determining redundancy scores of the candidate tokens according to the second attention mechanism information of the candidate tokens; a target candidate token is screened out from all the candidate tokens according to a comprehensive score calculated by the attention importance score and the redundancy score of the candidate tokens, and a to-be-compressed token sequence is generated according to the target candidate token and the observation token; and performing compression caching according to the to-be-compressed token sequence to obtain a compressed text.

The invention relates to a large language model long context cache compression method, system and device and a medium, and the method comprises the steps: obtaining a long text data set as verification data, inputting a large language model, calculating key value cache, executing Fourier projection and expansion operation, and calculating data reconstruction loss of each feature dimension, identifying key value dimensions insensitive to long text context information; in a text data pre-filling reasoning stage, finite-order Fourier projection compression is performed on key value cache data in a GPU memory, and marked data with insensitive dimensions are compressed from an original storage mode of linearly increasing along with the text length to a fixed-length frequency spectrum for representation; in the decoding reasoning stage, inverse projection expansion is directly carried out on the compressed Fourier projection coefficient in a GPU memory to restore the insensitive dimension by customizing a kernel operator in the GPU, and the attention weight is calculated. Compared with the prior art, the method has the advantages of reducing the storage overhead of the large language model and the like.

The present disclosure relates to the technical field of artificial intelligence and natural language processing, in particular to a key-value cache compression and sparse attention calculation method and system for large language model inference. The key-value cache compression and sparse attention calculation method for large language model inference comprises: an offline calibration stage; an online inference stage, comprising: a pre-filling step; an autoregressive generation step, for each newly generated word element: projecting the current query vector Q and the key vector K in the key cache into a low-dimensional space to obtain Q' and K'; calculating the approximate attention score based on Q' and K', and selecting the top k most relevant word element indexes I in descending order; calculating the accurate attention score based on Q and K[I], and calculating the output of the current word element with the value vector V[I]. The above scheme solves the memory and calculation bottleneck of large model inference in the scenario of long text sequence input, and has the advantages of reducing the memory occupation and the calculation complexity at the same time.

The invention provides a low-rank-quantization mixed key value cache compression method and system, and belongs to the technical field of large language models, and the method comprises the steps: initializing a reasoning process and a KV cache structure, and loading low-rank projection parameters and quantization configuration of each layer / header; if calibration is not completed, off-line calibration is triggered to determine a low-rank base Pr, and a calibration result is solidified for subsequent reasoning; the Key / Value of the new token is received; acquiring kt and vt of the current layer when a new token is generated each time; whether the token falls into a recent window L or not is judged, and if yes, kt and vt of the token are directly written into a recent area with high precision; and if not, carrying out asymmetric compression. According to the invention, the end-to-end throughput is improved; the two-area cache reduces the memory access overhead in the decoding period, has good adaptability to multi-layer multi-head and different models / hardware, does not need to change the model structure, and can be smoothly integrated with the existing reasoning parallelization / scheduling strategy.

The invention discloses an inference model key value cache compression method based on supervised training and reinforcement learning, and the method comprises the following steps: S1, building a dynamic key value cache elimination mechanism in long text generation: judging the importance of each key value pair in an inference process through a key value pair scoring device, and selecting an important key value pair in each step for retention; s2, supervised training: enabling the scoring model to have the capability of identifying key value pairs; and S3, reinforcement learning: eliminating and modeling the key value cache in the reasoning process into a Markov decision process, and optimizing an elimination strategy by adopting a reinforcement learning algorithm. According to the application, the currently reserved key value cache is regarded as a state, the discarding of the key value cache is regarded as an action, and the loss of the model on the sequence is regarded as an award. And optimization is carried out through a GRPO algorithm to maximize rewards, so that a large model and a key value cache elimination strategy are adapted to each other, and better performance is obtained on a long reasoning task.

Systems and methods for tensor cache compression and / or decompression are disclosed. An example method includes receiving a tensor or dynamically generated data. The example method includes compressing the tensor (or the dynamically generated data) by applying a compression scheme to values of the tensor (or dynamically generated data). The example method also includes storing a compressed tensor into a tensor cache (or compressed dynamically generated data into a respective cache). The example method includes reading the compressed tensor from the tensor cache (or the compressed dynamically generated data from the respective cache), and decompressing the compressed tensor (or compressed dynamically generated data) by applying a decompression scheme to values of the compressed tensor (or compressed dynamically generated data). The example method further includes forwarding a decompressed tensor (or decompressed dynamically generated data) to a compute unit.

Key-value (KV) caching accelerates inference in large language models (LLMs) by allowing the attention operation to scale linearly rather than quadratically with the overall sequence length. Due to large context lengths in modern LLMs, the KV cache size can exceed the model size, negatively impacting throughput. To address this issue, a multi-granular clustering-based solution for KV cache compression can be implemented. Key tensors and value tensors corresponding to unimportant tokens can be approximated using clusters generated at different clustering levels with varying accuracy. Accuracy loss can be mitigated by using proxies generated at a finer clustering level for a subset of attention heads that are more significant.More significant attention heads can have a greater impact on model accuracy than less significant attention heads. Latency is improved by retrieving proxies from a faster memory for a subset of less significant attention heads when the impact on accuracy is smaller.

This invention relates to the field of artificial intelligence and large language model inference optimization technology, and discloses an adaptive key-value cache compression method and system for large language models. The method includes: constructing a lexical importance measurement mechanism; analyzing the attention head distribution structure in large language model inference and constructing multiple pruning strategies; designing an adaptive key-value cache compression hybrid strategy set based on the lexical importance measurement mechanism and the attention head distribution structure, and based on the pruning strategies; constructing a static adaptive key-value cache compression method to automatically allocate key-value cache compression strategies during the pre-filling stage of large language model inference; and adaptively compressing the key-value cache based on the allocated key-value cache compression strategies during the decoding stage. This invention can achieve efficient compression of the key-value cache without relying on explicit attention score calculation, and is compatible with system-level inference optimization frameworks, maintaining generation performance while significantly reducing memory consumption and enhancing long-context inference capabilities.

A method for operating a transformer model includes algorithmically allocating a fixed budget for a key-value cache between multiple decoding layers per an allocation scheme that ensures progressively higher decoding layers in the transformer model are allocated progressively smaller quantities of cache memory. The method further includes configuring each of the multiple decoding layers of the transformer model to retain no more than a maximum number of key-value vector pairs in the key-value cache during a token decoding operation, the maximum number of key-value vector pairs being independently determined for each decoding layer of the multiple decoding layers based on the cache memory that is allocated to the decoding layer.

The present disclosure relates to the technical field of artificial intelligence and natural language processing, in particular to a KV cache compression and word piece recovery method and system for large language model inference, the method comprising: calculating the attention score of the query vector of the i-th word piece calculated by the current Transformer layer with all key vectors; performing a V cache dynamic update operation based on the attention score; for the i-th word piece, after the V cache dynamic update operation is completed, performing attention calculation using the updated value vector set in the V cache storage pool and the pre-calculated attention score partial product P. The above technical solution solves the technical problem that the existing technology is difficult to optimize the memory occupation and the calculation efficiency, and has the advantages of dynamically managing the KV cache memory occupation, maintaining the model inference quality, and improving the calculation efficiency.

The invention provides a large model multi-round dialogue fast state recovery method based on dynamic interlayer KV cache compression. The method comprises KV cache compression strategy search based on an attention mode, attention similarity calculation based on heterogeneous calculation and a KV cache recovery method based on calculation and storage fusion. The method has the advantages that the average precision loss of the dialogue is less than 1%, the precision loss of some specific tasks is reduced from 40% to 5%, meanwhile, the time delay of the first spitting of the model is reduced by 2 times, and the storage overhead of the KV cache is reduced by 2.5 times.

The invention relates to an adaptive budget allocation KV cache optimization method based on similarity, and the method comprises the steps: firstly providing an adaptive budget allocation KV cache expelling strategy, distributing a large budget for a head with uniform attention distribution, and strictly controlling the budget scale of a sparse head, so as to achieve the optimal budget allocation of the head; the overall budget constraint is effectively maintained, the reduction of the generation quality of a large language model is remarkably reduced, and on this basis, a similarity-based KV cache merging strategy is implemented to deal with the problem that the existing cache expelling strategy only retains part of high-importance KV cache pairs in the expelling process and does not retain part of high-importance KV cache pairs in the expelling process. And the problem of serious loss of context detail information caused by other secondary important KV cache pairs is solved, so that more reliable information restoration and retrieval capability is provided in a long context application scene, and the performance of a large language model on related tasks is improved. According to the method, under the condition that the cache budget is small, the KV cache compression loss of the model can be effectively reduced, the memory size needed by KV cache is reduced to the maximum degree, and the requirement for the environment is lower.

The invention discloses a large model KV cache compression method, system, product and equipment, and belongs to the technical field of artificial intelligence. The method comprises the following steps: S1, global low-rank approximation: capturing the overall redundancy of a KV matrix, and reducing the global storage overhead through low-rank decomposition; s2, local residual compensation: compensating local detail loss caused by global low-rank approximation, and accurately retaining details of a high-information-density region; s3, global-local complementary compensation: integrating redundant compression of global low-rank approximation and detail reservation of local residual compensation to form a complementary optimized KV cache; and S4, dynamic complementary management: dynamically adjusting parameters to optimize performance according to different tasks, sequence lengths or hardware environments. According to the method, the balance of global redundancy efficient compression and local detail accurate reservation can be realized.

The invention provides a cache structure, a cache method, a chip and electronic equipment, and relates to the technical field of cache. The cache structure comprises N sectors, wherein N is a positive integer; each sector comprises M cache lines; wherein the storage area of each cache line is divided into a first area and a second area, each cache line is associated with two groups of cache line status bits in 2M groups of cache line status bits, and the two groups of cache line status bits associated with each cache line are respectively used for indicating the states of the first area and the second area; a first sector address label used for indicating the first area, a second sector address label used for indicating the second area and 2M groups of cache line status bits are stored in the storage unit, and M is an integer greater than 1. In this way, the extra time overhead caused by cache data compression and decompression and the complexity of accessing the cache compressed data can be reduced, and therefore the overall performance of the cache compressed and decompressed data can be improved.

The invention discloses an electric energy data security storage method based on power failure detection and cache compression, and belongs to the technical field of intelligent single-chip microcomputer systems. The method specifically comprises the following steps: after electrification, detecting keg data, reading and storing all electric energy data, further embedding a pre-trained model, and constructing a transaction log; based on the electric energy data accumulated in real time, creating a timed task and setting a trigger condition, when the condition is triggered, executing a specific task and updating a transaction log; on the basis of a software-hardware combination mechanism, it is ensured that data can be stored in time and a transaction log can be updated when power failure is abnormal; and when the power is recovered, executing an initialization stage to recover data, further predicting a power failure reason by using a pre-trained model, and updating a transaction log. Through a software and hardware fusion algorithm technology, the problem of loss of the electric energy data under an abnormal condition is solved, and the safety of the electric energy data is ensured.

The key-value (LV) cache in this application accelerates inference in large language models (LLMs) by allowing attention operations to scale linearly, rather than quadratically, with the total sequence length. Since context lengths are long in modern LLMs, the KV cache size may exceed the model size, potentially negatively impacting throughput. To address this issue, a multi-granularity clustering-based scheme for KV cache compression is implemented. Clusters created at different clustering levels with variable precision approximate the key and value tensors corresponding to less important terms. Precision loss is reduced by using proxies generated at finer-grained clustering levels of subsets of more salient attention heads. More salient attention heads have a greater impact on model accuracy than less salient attention heads. When the impact on accuracy is low, latency is improved by retrieving proxies from a subset of less salient attention heads from faster memory.

The present application relates to a high-fidelity key cache compression method and system based on spectral quantization, comprising: step S1: spectral transformation and feature analysis process; receiving original key cache data from the model, mapping the original key cache data from the time domain to the frequency domain, and deeply analyzing the signal structure by using the energy concentration characteristics unique to the discrete cosine transform (DCT); step S2: frequency domain hybrid quantization encoding process; by using the main frequency extraction mechanism, the high frequency pre-emphasis technology and the hybrid bit width allocation strategy, the frequency domain coefficient is compressed into a sparse representation with extremely low bit; step S3: fusion decoding and sparse accumulation process; by using the hardware-aware fusion operator and the delay inverse transform technology, the attention score is calculated directly in the compression domain. The present application realizes the limit compression and efficient inference under the lossless precision.

A method, system, and computer program product configured to perform operations including: performing, during a decoding phase performed by a large language model (LLM), multi-head compression of key-value data in a cache; and performing partial decompression of the key-value data during the decoding phase, wherein the LLM is a decoder-based transformer model that is configured to generate an output based on an input by performing a tokenization phase, a prefill phase, the decoding phase, and a detokenization phase.

The present application relates to the cross field of artificial intelligence and computer architecture, in particular to an adaptive KV cache compression method, system and device for an audio-text multi-modal large model. The method comprises opening a full-precision hot table and a low-bit cold table in the video memory; writing the newly generated audio or text Token in the FP16 format into the hot table, and when the number of Tokens in the hot table reaches a preset threshold, gradually migrating the Tokens in the hot table to the cold table; identifying the modal type of the Token during the migration process, selecting a corresponding quantization grouping strategy according to the identified modal type, and performing INT2 compression storage to the cold table after online calculation of the quantization parameters of the Token to be migrated. Through the cooperation of the above steps, the present application realizes the improvement of the model inference throughput and the natural full-duplex human-computer interaction under the premise of ensuring the audio synthesis quality.

This invention discloses a semantically aware KV Cache management method. It constructs a semantically aware KV Cache management model for the reasoning process of large-scale Transformer architecture models. Based on deepest-layer attention drift detection, it determines the segmentation point and synchronizes across all layers after triggering semantic segmentation. Combined with four-region caching, it performs cache compression, transforming the cache management process from fixed-position truncation to a dynamic process based on semantic changes. Through the collaborative updating of initial blocks, Top-K blocks, temporary blocks, and active blocks, this method can balance the preservation of historical key information, maintenance of current semantic continuity, and cache occupancy control during long-context reasoning, thereby improving cache management efficiency and reasoning stability. It solves the problems of rapid growth in KV Cache memory usage with increasing sequence length during long-context reasoning of large language models, the tendency of existing compression methods to truncate the context at fixed physical positions and cause semantic fragmentation, and the impact of inconsistent multi-layer cache updates on reasoning stability.

The application provides a KV cache compression processing method, device and equipment for generating investment strategy code by a large model, and a medium, and relates to the field of large models. The method comprises the following steps: obtaining a token sequence of investment strategy code generated through multiple iterations of a large model, wherein the token sequence comprises multiple tokens; performing importance evaluation on each token to obtain an importance score of each token; dividing the multiple tokens in the token sequence into different branches based on the importance score of each token; and performing different KV cache compression strategies on different branches.

The application discloses a kind of compression method, device, equipment and storage medium of time series data.The method comprises: obtaining the set of service model to be compressed, determines compression step;Based on temporary service model and the compression step, traverse the set of service model to be compressed, and update the temporary service model in the process of traversal, and record parent class time data, wherein the temporary service model is used to cache temporary time series data in compression process, and the parent class time data is used to indicate that the original time data corresponding to the original index data meeting the preset characteristic requirement in the unit compression range corresponding to the compression step;According to the traversal result, determine the parent class time data set and determine the service model compression result set.The technical scheme of the embodiment of the application orderly compresses a large amount of time series data, not only guarantees the compression efficiency of time series data, but also records the parent class time data, to provide user to consult original time data.

The invention relates to the technical field of data processing and the field of smart medical treatment and financial science and technology, and provides a cache compression method and device, electronic equipment and a computer readable storage medium, and the method comprises the steps: obtaining reasoning input information; determining the number of target key value pairs according to the reasoning input information; determining a budget cache according to the number of the target key value pairs; carrying out preliminary screening processing on the budget cache based on a preset first scale factor to obtain a coarsened cache; performing pooling operation on the coarsened cache to obtain a first cache; performing refined screening processing on the first cache based on a preset second scale factor to obtain a refined cache; and when the attention importance score is greater than or equal to a preset level threshold value, performing propagation processing on the token corresponding to the key value pair. According to the technical scheme, the minimum cache budget can be automatically matched in different input scenes, and excessive or insufficient compression is avoided.

This invention relates to a key-value caching compression method and system for large language models based on semantic cluster center offset, belonging to the field of large language model inference acceleration technology. The method includes: a pre-filling stage, where semantic clustering is performed on the key matrix and cluster center vectors are extracted; the offset between the key vectors and the cluster centers is calculated; unstructured channel pruning and low-bit quantization are performed on the offsets; compressed data is stored in off-chip video memory; and cluster centers are stored in on-chip shared memory. In the decoding stage, compressed data is read from off-chip video memory; the offsets are dequantized to recover the data; approximate key vectors are reconstructed using the on-chip cluster centers; and attention calculation is performed. This invention maintains semantic integrity under high sparsity by transforming the compressed object from the original vector into an offset from the semantic cluster centers, eliminating the truncation and destruction of low-bit quantization by large numerical outliers, and achieving lossless superposition of pruning and quantization.

The application relates to an enhanced shallow KV Cache compression method and system based on temporary storage screening and residual compensation. A client (such as a user terminal device) is responsible for acquiring user inputted long text query data and sending the long text query data to a server end. The server end is provided with a large language model with a Transformer architecture. The application is executed in a computing device of the server end. Through differential and dynamic compression scheduling of shallow and deep KV Cache (key value cache) in an LLM inference process, high throughput and high precision long text inference can be completed under the condition of limited physical display memory of hardware, and finally, the generated text result is returned to the client.