68 results about "Cache compression" patented technology

The invention discloses a question and answer reasoning method and device based on key value cache compression, equipment and a medium, and relates to the technical field of natural language processing, and the method comprises the steps: segmenting a cue word in a current question and answer task into a lexical sequence, and generating an initial key value cache of the lexical sequence; dividing the lexical element sequence into context lexical elements and tail end lexical elements corresponding to each layer based on a preset tail end window size of each attention layer of the target large language model; screening out keyword elements of each attention layer from the context lexical elements according to importance scores between key matrixes of the context lexical elements and query matrix mean values of the tail end lexical elements; removing key value pairs of lexical elements except the keyword elements in the initial key value cache to obtain a compressed key value cache; and generating a reasoning result corresponding to the compressed key value cache by using the target large language model. The high computing power consumption of the large language model caused by key value cache data increase is reduced, and the dependence of an existing key value cache compression method on a complete attention weight matrix is broken through.

The invention relates to the technical field of artificial intelligence and natural language processing, in particular to a key value cache compression and sparse attention calculation method and system for large language model reasoning, and the method comprises the steps: an offline calibration stage; the online reasoning stage comprises the following steps: a pre-filling step; an autoregression generation step: for each newly generated lexical element, projecting a current query vector Q and a key vector K in a key cache to a low-dimensional space to obtain Q'and K '; calculating an approximate attention score based on Q'and K ', and selecting an index I of the first k most relevant lexical elements which are ranked from high to low; and calculating an accurate attention score based on Q and K [I], and calculating with the value vector V [I] to obtain the output of the current lexical element. According to the scheme, the memory and calculation bottleneck of large model reasoning in a scene of long text sequence input are solved, and the method has the advantages of reducing video memory occupation and calculation complexity at the same time.

The invention discloses a data processing method and system, electronic equipment, a storage medium and a computer program product, and relates to the technical field of large model technology and key value caching. The method comprises the following steps: acquiring a plurality of visual marks, a plurality of text marks and initial key value data corresponding to a visual language model; determining a key text mark in the plurality of text marks by utilizing the initial key value data; according to the initial key value data and the distribution positions of the key text marks in the multiple text marks, importance evaluation is conducted on the multiple visual marks, an evaluation result is obtained, and the evaluation result is used for representing cross-modal attention weight distribution between the multiple visual marks and the key text marks; and according to an evaluation result, performing cache compression processing on the initial key value data to obtain target key value data. According to the method and the device, the technical problem that the model reasoning efficiency is influenced due to high key value data caching overhead of a visual language model in related technologies is solved.

The invention provides a self-adaptive prefix key value cache compression method and device and electronic equipment, relates to the technical field of computers, and aims to solve the defects of an existing key value cache compression method in the aspect of ensuring high reasoning performance and high task precision. The method comprises the steps that mark sequences corresponding to image information and text information are input into a large model for multi-layer attention interaction, and the attention score of each mark in each layer is obtained; according to the attention score of each mark, determining the importance proportion of each key value cache vector of each layer; taking the importance proportion of the key value cache vectors as a priority numerical value, and performing descending sort on the key value cache vectors in each layer to obtain a priority sequence of the key value cache vectors of each layer; determining a target information retention ratio based on the target compression ratio budget; and according to the target information retention ratio, determining a retained key value cache vector from the priority sequence of the key value cache vector of each layer.

The invention discloses a computing device, method, equipment, chip and system, and relates to the field of computers. The calculation device provided by the invention performs calculation (such as a matrix inner product and a matrix outer product) of the compressed sparse format matrix based on the compression identifiers corresponding to the element values in the compressed sparse format matrix, and outputs the matrix inner product result or the matrix outer product result of the compressed sparse format matrix, so that the calculation efficiency of the compressed sparse format matrix is improved by utilizing the large calculation power of the calculation unit supporting the matrix calculation. The sparse format matrix is efficiently calculated and compressed, so that the calculation unit expands and supports the matrix calculation of the unstructured sparse matrix, and the calculation efficiency of the sparse matrix is improved. Besides, a merging circuit is arranged in the computing device to realize matrix accumulation of an intermediate result of a matrix outer product in the computing device, and a register is arranged in the computing device to cache and compress the intermediate result of the sparse format matrix, so that the storage space and the access bandwidth required by sparse matrix computing are reduced, and the computing efficiency of the sparse matrix is further improved.

The invention relates to the technical field of computers, in particular to a fault report generation method based on the size of a dynamic window in an edge cloud scene, which comprises the following steps of: S1, performing data processing according to acquired data to obtain preprocessed data, and inputting the preprocessed data into a pre-training model for training to obtain an adjustment model; s2, multi-modal input data generates a multi-modal KV cache according to the adjustment model, encoding is carried out to obtain an encoding set, KV key value pairs are screened according to the encoding set, decoding is carried out after the KV key value pairs are combined, KV cache compression is carried out on the decoded data through Page Attelation to obtain matched compressed data, and a window is dynamically adjusted according to the compressed data; and S3, receiving second real-time operation state data of the edge equipment, generating a second historical fault report, and judging whether the second historical fault report meets the model requirements or not. Through LOOK-M and paged attention technologies, KV cache of a large model is reduced, memory use and calculation efficiency are optimized, model fine tuning is introduced, and it is ensured that a generated fault analysis report has high accuracy and real-time performance.

The invention belongs to the technical field of large language models, and particularly relates to a large language model KV cache compression method based on interlayer fusion. The method comprises the following steps: inputting a to-be-processed text into a large language model to obtain a KV cache; the compression ratio of each layer of the large model is calculated, and preliminary SVD compression is carried out on KV cache of different layers of the large model; the KV cache subjected to preliminary compression is subjected to block division, and a plurality of blocks are obtained; calculating the attention score of each block; s blocks with the highest attention score are selected, and SVD reconstruction is carried out on the selected blocks; constructing a cache pool to store the reconstructed S blocks; dynamically updating the cache pool; splicing the S blocks in the cache pool to obtain a KV cache, and completing the compression of the KV cache; according to the method, the model is ensured to maintain stable prediction accuracy under diversified task scenes such as text generation and question and answer systems, and balance between storage efficiency and performance is realized.

The invention relates to the technical field of artificial intelligence and big language model reasoning optimization, and discloses a big language model-oriented adaptive KV cache compression method and system, and the method comprises the steps: constructing a lexical element importance measurement mechanism; analyzing an attention head distribution structure in large language model reasoning, and constructing a plurality of pruning strategies; based on a lexical element importance measurement mechanism and the attention head distribution structure, designing a self-adaptive key value cache compression hybrid strategy set based on a pruning strategy; constructing a static self-adaptive key value cache compression method, and automatically distributing a key value cache compression strategy in a pre-filling stage of large language model reasoning; and in a decoding stage, performing adaptive compression on the key value cache based on the allocated key value cache compression strategy. According to the method, the key value cache can be efficiently compressed on the premise of not depending on explicit attention score calculation, a system-level reasoning optimization framework is compatible, the generation performance is kept, meanwhile, the video memory consumption is remarkably reduced, and the long context reasoning capability is enhanced.

The invention relates to the technical field of artificial intelligence and natural language processing, in particular to a KV cache compression and eviction lexical element recovery method and system for large language model reasoning, and the method comprises the steps: for a query vector of an ith lexical element calculated by a current Transform layer, calculating attention scores of the query vector and all key vectors; executing a V cache dynamic updating operation based on the attention score; and for the ith lexical element, after the V cache dynamic updating operation is completely completed, performing attention calculation by using the updated value vector set in the V cache storage pool and the pre-calculated attention score partial product P. According to the technical scheme, the technical problem that in the prior art, collaborative optimization of video memory occupancy and calculation efficiency is difficult is solved, and the method has the advantages that KV cache video memory occupancy is dynamically managed, the model reasoning quality is kept, and the calculation efficiency is improved.

The invention discloses a memory page processing method, equipment and device and a storage medium, and belongs to the technical field of communication. According to the method, after the memory page is allocated to the virtual address causing the missing page interruption, the allocated memory page is filled with the first data in advance, then the memory page filled with the first data is accessed, and if the data in the memory page is migrated to the cache subsequently, the first data filled in the memory page can also be migrated to the cache, so that the data in the memory page cannot be migrated to the cache. If the data of the memory page in the cache is eliminated to the memory, the eliminated cache data may also include the first data, and because the first data is the data in the data sequence corresponding to the cache compression mode, when the cache data including the first data is eliminated back to the memory, the probability that the cache data is matched with the cache compression mode can be increased, and the cache compression mode is matched with the cache data. And thus, the compression rate of the cached data can be improved.

The invention provides a key value cache compression method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining an attention score between an input current lexical element and a target lexical element set, and the target lexical element set comprises all historical lexical elements from a sequence starting position to a current position and the current lexical element; performing aggregation processing on the plurality of attention scores historically accumulated by each target lexical element to generate an aggregation attention value corresponding to each target lexical element; dynamically determining a cache merging threshold according to the sorting result of the aggregation attention values of the target lexical units; recognizing the lexical element fragments with the aggregation attention values continuously lower than the cache merging threshold value and adjacent positions as target cache fragments; and executing fusion compression on the key value cache corresponding to the target cache fragment and updating the original cache. According to the method and the device, occupation of key value cache on video memory resources can be effectively reduced, and meanwhile, the semantic retention capability is remarkably improved.

The invention discloses a cascade speculation inference method and system based on hierarchical decline KV cache compression, and the method comprises the steps: firstly inputting a context prompt text into a target model for coding, generating a KV cache, and calculating an attention score between tokens; secondly, descending sorting is carried out based on the attention scores of the final input token, KV cache blocks corresponding to the first k attention scores are selected as cascading middle layer cache, a lightweight large language model is loaded to serve as a draft model, and a hierarchical decline KV cache compression strategy is adopted to maintain draft model cache. Then, based on all the caches, a double-layer cascade speculation reasoning framework is constructed, a target reasoning path is obtained, and the caches are updated; and finally, repeating the operation until target response data corresponding to the context prompt text is output according to the target reasoning path. According to the method, the KV cache proportion is reduced, meanwhile, the draft token acceptance rate of the target model of the full KV cache is improved, and reduction of precision is reduced.

The invention discloses a KV Cache compression and hierarchical management method and device for RAG acceleration and a medium, and belongs to the technical field of large model reasoning acceleration. In order to solve the problems that in an existing RAG KV Cache management technology, storage occupation is too large, and long-term importance and instant accessibility of data cannot be considered at the same time, the invention provides a hierarchical management strategy, and according to the method, a multi-dimensional popularity vector containing global popularity, session popularity and context popularity is defined for a KV Cache block. The method is characterized in that the compression precision of a cache block is independently determined by using global popularity so as to balance fidelity and performance; and meanwhile, the session heat and the context heat are used for independently determining the placement positions in heterogeneous hierarchies such as a GPU (Graphics Processing Unit), a CPU (Central Processing Unit) fixed memory and a CPU paging memory. According to the method, placement is guided through the real-time dimension, compression is guided through the importance dimension, context intelligent prefetching is achieved, and the reasoning performance of an RAG system, especially in a session and in a context pursuit scene is remarkably improved.

The invention relates to a KV cache compression method based on attention alignment, and belongs to the technical field of large language models. Comprising the following steps: a target model self-generates dialogue data reply, and constructs training data; adding a soft token into a word list of the target model, and carrying out soft token random initialization; finely adjusting specified parameters, splicing soft tokens at the tail of an original input sequence, and transmitting the soft tokens to the model; and respectively calculating the soft token and the attention distribution of the self-generated reply, calculating the mean square error of the soft token and the attention distribution of the self-generated reply as a loss function, and completing training. According to the method, a limited number of soft tokens playing an auxiliary role are introduced into original input, and importance discrimination of KV cache elements is realized and expelling is completed by aligning attention distribution of the soft tokens and real generated content; according to the method, better compromise is achieved between sequence length compression and performance loss, and the loss of model performance is better controlled while it is guaranteed that KV cache video memory space occupation is reduced.

Provided is an accelerator for accelerating batch large language model (LLM) inference via key-value cache compression. An accelerator, according to one embodiment, may comprise a plurality of compute cores. Here, each of the plurality of compute cores may comprise a plurality of processing units for processing an LLM inference operation on a per-token basis, and a direct memory access unit for managing operations of reading weights from a memory and writing key-value activation data back to the memory. In addition, the direct memory access unit may comprise a compression engine for managing online compression of the key-value activation data when the key-value activation data is written to the memory, a decompression engine for decompressing the compressed key-value activation data retrieved from the memory, and a memory management unit for managing reading and writing of the compressed key-value activation data in the memory.

The invention discloses a mixed precision parallel compression method for optimizing large model key value cache, which combines the advantages of mixed precision key value cache compression with an advanced system optimization technology. Based on the characteristic that key value pairs needing high-precision retention in mixed precision compression are the same as key value pairs used for attention calculation in a prefetching strategy, low-precision KV caches are stored in a GPU memory, and meanwhile predicted high-precision important KV caches are dynamically prefetched from a CPU memory according to needs. The technical problems that an existing multi-head attention mechanism-based method is incompatible with an existing pre-training model and cannot be directly applied to a closed source or a fine-tuned large model, and generalization of the method is reduced can be solved, and the situation that an existing pruning-based method is prone to deleting unimportant marks in the current stage can be solved. And contextual information loss is caused.

The invention provides a cache compression method and system, electronic equipment and a storage medium. The method comprises the following steps: generating an observation token sequence and a candidate set according to a first text in a buffer area and a second text in a cache; generating first attention mechanism information of the observation token and second attention mechanism information of the candidate token; determining an attention importance score according to the first attention mechanism information of the observation token and the second attention mechanism information of the candidate token; determining redundancy scores of the candidate tokens according to the second attention mechanism information of the candidate tokens; a target candidate token is screened out from all the candidate tokens according to a comprehensive score calculated by the attention importance score and the redundancy score of the candidate tokens, and a to-be-compressed token sequence is generated according to the target candidate token and the observation token; and performing compression caching according to the to-be-compressed token sequence to obtain a compressed text.

The invention relates to a large language model long context cache compression method, system and device and a medium, and the method comprises the steps: obtaining a long text data set as verification data, inputting a large language model, calculating key value cache, executing Fourier projection and expansion operation, and calculating data reconstruction loss of each feature dimension, identifying key value dimensions insensitive to long text context information; in a text data pre-filling reasoning stage, finite-order Fourier projection compression is performed on key value cache data in a GPU memory, and marked data with insensitive dimensions are compressed from an original storage mode of linearly increasing along with the text length to a fixed-length frequency spectrum for representation; in the decoding reasoning stage, inverse projection expansion is directly carried out on the compressed Fourier projection coefficient in a GPU memory to restore the insensitive dimension by customizing a kernel operator in the GPU, and the attention weight is calculated. Compared with the prior art, the method has the advantages of reducing the storage overhead of the large language model and the like.

The invention discloses a cache fusion-based large-scale language model reasoning acceleration method, electronic equipment and a storage medium, and aims at realizing minimization of information loss in a compression process through a mode of merging caches and effectively solving problems in a KV cache compression process. The method comprises the following steps of: 1, calculating a cache value of a given large language model based on a Token Merge technology, and performing pruning and merging operation according to the calculated cache value; 2, for a KV cache matrix of an original large language model, generating a binary mask through Bernoulli distribution for Token needing to be abandoned; 3, for the reserved KV cache matrix of the large language model, firstly calculating the merged probability value through the attention weight, and then averagely merging the abandoned Token into the reserved Token by using the binary mask in the step 2; and 4, in the forward propagation process of the large language model, continuously executing the processing processes from the step 1 to the step 3 on the generated KV cache matrix.

The present disclosure relates to the technical field of artificial intelligence and natural language processing, in particular to a key-value cache compression and sparse attention calculation method and system for large language model inference. The key-value cache compression and sparse attention calculation method for large language model inference comprises: an offline calibration stage; an online inference stage, comprising: a pre-filling step; an autoregressive generation step, for each newly generated word element: projecting the current query vector Q and the key vector K in the key cache into a low-dimensional space to obtain Q' and K'; calculating the approximate attention score based on Q' and K', and selecting the top k most relevant word element indexes I in descending order; calculating the accurate attention score based on Q and K[I], and calculating the output of the current word element with the value vector V[I]. The above scheme solves the memory and calculation bottleneck of large model inference in the scenario of long text sequence input, and has the advantages of reducing the memory occupation and the calculation complexity at the same time.

The invention discloses a dynamic multi-domain power consumption regulation-based ultra-low power consumption design method for an AI voice chip. The method comprises the following steps of 1, designing a chip power supply scheme and a clock scheme according to an application scene of a system; 2, designing a multi-level layered clock gating architecture control system; step 3, dividing the power supply network into a dynamic power supply domain and a normally open power supply domain; step 4, cache compression design of voice data is carried out; step 5, minimum addressable unit grouping optimization is carried out on a memory, and minimum addressable units with different granularities are supported to implement selective turn-off; and step 6, completing the design after the simulation is passed and an expected target is reached. Through a hierarchical gating layer design method, bidirectional optimization of dynamic energy efficiency management and time sequence precision is realized, and the overall clock power consumption is reduced.

The invention provides a low-rank-quantization mixed key value cache compression method and system, and belongs to the technical field of large language models, and the method comprises the steps: initializing a reasoning process and a KV cache structure, and loading low-rank projection parameters and quantization configuration of each layer / header; if calibration is not completed, off-line calibration is triggered to determine a low-rank base Pr, and a calibration result is solidified for subsequent reasoning; the Key / Value of the new token is received; acquiring kt and vt of the current layer when a new token is generated each time; whether the token falls into a recent window L or not is judged, and if yes, kt and vt of the token are directly written into a recent area with high precision; and if not, carrying out asymmetric compression. According to the invention, the end-to-end throughput is improved; the two-area cache reduces the memory access overhead in the decoding period, has good adaptability to multi-layer multi-head and different models / hardware, does not need to change the model structure, and can be smoothly integrated with the existing reasoning parallelization / scheduling strategy.

The invention discloses an inference model key value cache compression method based on supervised training and reinforcement learning, and the method comprises the following steps: S1, building a dynamic key value cache elimination mechanism in long text generation: judging the importance of each key value pair in an inference process through a key value pair scoring device, and selecting an important key value pair in each step for retention; s2, supervised training: enabling the scoring model to have the capability of identifying key value pairs; and S3, reinforcement learning: eliminating and modeling the key value cache in the reasoning process into a Markov decision process, and optimizing an elimination strategy by adopting a reinforcement learning algorithm. According to the application, the currently reserved key value cache is regarded as a state, the discarding of the key value cache is regarded as an action, and the loss of the model on the sequence is regarded as an award. And optimization is carried out through a GRPO algorithm to maximize rewards, so that a large model and a key value cache elimination strategy are adapted to each other, and better performance is obtained on a long reasoning task.

Systems and methods for tensor cache compression and / or decompression are disclosed. An example method includes receiving a tensor or dynamically generated data. The example method includes compressing the tensor (or the dynamically generated data) by applying a compression scheme to values of the tensor (or dynamically generated data). The example method also includes storing a compressed tensor into a tensor cache (or compressed dynamically generated data into a respective cache). The example method includes reading the compressed tensor from the tensor cache (or the compressed dynamically generated data from the respective cache), and decompressing the compressed tensor (or compressed dynamically generated data) by applying a decompression scheme to values of the compressed tensor (or compressed dynamically generated data). The example method further includes forwarding a decompressed tensor (or decompressed dynamically generated data) to a compute unit.

An example method may include receiving, in a virtualized execution environment, a data access request from a storage system, identifying, in view of a virtualized execution image associated with the virtualized execution environment, an application running in the virtualized execution environment, generating a cache classification that specifies whether data accessed by the application is suitable for cache compression, including, in the data access request, a tag indicating whether cached data is to be accessed in a compressed-memory cache, wherein the tag is determined in view of the cache classification, and sending, to a server of the storage system, the data access request. The application can be identified in view of metadata included in the virtualized execution image, where the metadata comprises one or more of an application name or an application version.

Key-value (KV) caching accelerates inference in large language models (LLMs) by allowing the attention operation to scale linearly rather than quadratically with the overall sequence length. Due to large context lengths in modern LLMs, the KV cache size can exceed the model size, negatively impacting throughput. To address this issue, a multi-granular clustering-based solution for KV cache compression can be implemented. Key tensors and value tensors corresponding to unimportant tokens can be approximated using clusters generated at different clustering levels with varying accuracy. Accuracy loss can be mitigated by using proxies generated at a finer clustering level for a subset of attention heads that are more significant.More significant attention heads can have a greater impact on model accuracy than less significant attention heads. Latency is improved by retrieving proxies from a faster memory for a subset of less significant attention heads when the impact on accuracy is smaller.

This invention relates to the field of artificial intelligence and large language model inference optimization technology, and discloses an adaptive key-value cache compression method and system for large language models. The method includes: constructing a lexical importance measurement mechanism; analyzing the attention head distribution structure in large language model inference and constructing multiple pruning strategies; designing an adaptive key-value cache compression hybrid strategy set based on the lexical importance measurement mechanism and the attention head distribution structure, and based on the pruning strategies; constructing a static adaptive key-value cache compression method to automatically allocate key-value cache compression strategies during the pre-filling stage of large language model inference; and adaptively compressing the key-value cache based on the allocated key-value cache compression strategies during the decoding stage. This invention can achieve efficient compression of the key-value cache without relying on explicit attention score calculation, and is compatible with system-level inference optimization frameworks, maintaining generation performance while significantly reducing memory consumption and enhancing long-context inference capabilities.

Disclosed are a computing apparatus and method, and a device, a chip and a system, which relate to the field of computers. The computing apparatus provided in the present application executes the computation of compressed sparse-format matrices on the basis of compression identifiers corresponding to element values in the compressed sparse-format matrices (for example, a matrix inner product and a matrix outer product), and outputs a matrix inner product result or a matrix outer product result of the compressed sparse-format matrices, such that the compressed sparse-format matrices are efficiently computed by using the large computing power of a computing unit that supports matrix computation, and the computing unit is expanded to support the matrix computation of unstructured sparse matrices, thereby improving the computational efficiency of the sparse matrices. Moreover, a merging circuit is provided inside the computing apparatus to accumulate intermediate result matrices of a matrix outer product in the computing apparatus, and a register is provided in the computing apparatus to cache intermediate results of the compressed sparse-format matrices, thereby reducing the storage space and access bandwidth required for computing the sparse matrices and further improving the computational efficiency of the sparse matrices.

A method for operating a transformer model includes algorithmically allocating a fixed budget for a key-value cache between multiple decoding layers per an allocation scheme that ensures progressively higher decoding layers in the transformer model are allocated progressively smaller quantities of cache memory. The method further includes configuring each of the multiple decoding layers of the transformer model to retain no more than a maximum number of key-value vector pairs in the key-value cache during a token decoding operation, the maximum number of key-value vector pairs being independently determined for each decoding layer of the multiple decoding layers based on the cache memory that is allocated to the decoding layer.

The invention discloses a data caching device and method, and the device comprises a caching module which is used for caching compressed data; the compression processing module comprises a compression unit and a decompression unit, and the compression unit is used for performing compression processing on the original data to obtain compressed data when the original data passes the comparison verification and outputting the compressed data to the cache module for caching; the decompression unit is used for decompressing the compressed data acquired from the cache module; the decompression unit further comprises a verification subunit, the verification subunit is used for comparing and verifying original data to be written into the first position of the cache module with first decompressed data, and the first decompressed data is obtained by decompressing existing data obtained from the first position in the cache module through the decompression unit.