Patents
Literature
Patsnap Eureka AI that helps you search prior art, draft patents, and assess FTO risks, powered by patent and scientific literature data.

54 results about "Cache reuse" patented technology

Fuzzy semantic matching-based large language model key value cache multiplexing method and system

The invention relates to the technical field of big language model reasoning, and discloses a big language model key value cache multiplexing method and system based on fuzzy semantic matching, and the method comprises the steps: generating a key value cache according to lexical elements in a historical reasoning request of a user, gathering a plurality of lexical elements into lexical element blocks, generating embedded vectors of the lexical element blocks, and building a vector database; calculating a cosine similarity between an embedded vector of a lexical block of a new reasoning request and a historical embedded vector in a vector database, and if a historical lexical block of which the cosine similarity exceeds a set threshold exists, obtaining a corresponding key value cache through a Hash index and multiplexing the key value cache; calculating an attention score, and dividing the plurality of lexical elements in the current lexical element block into keyword elements and non-keyword elements based on the attention score; key value caches of the keyword elements are recalculated; and the re-calculated key value cache of the keyword elements and the reuse key value cache of the non-key sub-elements form a mixed key value cache. According to the method, on the premise that the model precision is almost not reduced, the key value cache multiplexing technology is expanded to fuzzy semantic matching from accurate matching, unnecessary calculation overhead is effectively reduced, and then the reasoning efficiency is improved.
Owner:UNIV OF SCI & TECH OF CHINA

Multi-round dialogue method, device and equipment based on large model and storage medium

The invention discloses a multi-round dialogue method and device based on a large model, equipment and a storage medium, and relates to the field of natural language processing, and the method comprises the steps: screening out a target round dialogue corresponding to a current round dialogue from historical dialogues; extracting triple information of the current round of dialogue and the target round of dialogue, and constructing a target cue word based on the domain data and the triple information corresponding to the current round of dialogue; the triple information comprises intention information, entity information and slot position information; inputting the target cue word into a preset dialogue large model, and determining a target key value cache of the target cue word and a historical key value cache corresponding to the target round of dialogue; and generating a dialogue reply of the current round of dialogue based on the target key value cache and the historical key value cache. By calculating the semantic similarity of the historical dialogue and the current round, combining entity recognition and only retaining part of content, the memory used by the dialogue is effectively reduced, logic breakage is avoided, and the use efficiency of the video memory is effectively improved through key value cache multiplexing.
Owner:SHANDONG LANGCHAO YUNTOU INFORMATION TECH CO LTD

Multi-modal KV cache retrieval method and system based on hybrid architecture

The invention discloses a multi-modal KV cache retrieval method and system based on a hybrid architecture, relates to the technical field of data processing, and aims to solve the technical problem that in the prior art, due to the adoption of a full-online computing architecture and lack of a unified cross-modal representation framework, the computing efficiency and the modal fusion effect are difficult to consider at the same time. By constructing a hybrid processing architecture of offline pre-calculation and online processing, multi-modal data is subjected to unified fragmentation coding and KV cache and semantic fingerprints of the multi-modal data are pre-calculated in an offline stage, and cache fragments are intelligently assembled and position codes are dynamically remapped based on a user query intention in an online stage; therefore, online repeated calculation of multi-modal content is fundamentally avoided, coherent alignment and efficient fusion of cross-modal semantics are achieved, the real-time response capability, the cache reuse rate and the multi-modal generation quality of the system are remarkably improved, and high-concurrency scenes can be dynamically adapted.
Owner:SHANGHAI COSUNET NETWORK TECH CO LTD

Software component analysis optimization method based on Hash cache

The invention belongs to the technical field of electrical digital data processing, and particularly relates to a Hash cache-based software component analysis optimization method, which comprises the following steps of: S1, scanning multi-dimensional features for the first time, and calculating storage component Hash by fusing dynamic behavior features, the method comprises the following steps: acquiring dynamic behavior characteristics of components during operation through a lightweight sandbox environment, and extracting basic characteristics, content characteristics and dependency characteristics of a software package and each component; hash values are introduced to serve as unique identifiers of software packages and components, and the Hash values of the components and corresponding detection results are calculated and stored during first-time scanning; according to the method, the component hash value is verified preferentially during subsequent scanning, if the hash value exists in the cache library, the historical detection result is directly reused, the full-amount analysis process does not need to be repeatedly executed, and on the premise that analysis accuracy is guaranteed, repeated calculation is reduced through a cache reuse mechanism, SCA analysis efficiency is improved, and resource consumption is reduced.
Owner:SHENZHEN HAIYUNAN NETWORK SECURITY TECH CO LTD

Hypergraph neural network reasoning method and system

The invention relates to a hypergraph neural network inference method and system. The method comprises the following steps: extracting topological information with a vertex with the highest degree from compressed hypergraph topological representation; identifying the common vertex with the highest degree and the hyperedge connected with the common vertex and sending the common vertex and the hyperedge to a task queue; triggering hyperedge aggregation operation on each hyperedge in parallel based on common vertexes from the common vertex table in the task queue; hyperedge aggregation operation is executed, generated hyperedge aggregation intermediate results are temporarily stored in an intermediate result buffer area of an on-chip cache unit, and in the process of executing hyperedge aggregation operation of hyperedges in sequence based on the hyperedge storage sequence in the hypergraph topology, the hyperedge aggregation intermediate results from common vertexes of a common vertex table are reused, and the hyperedge aggregation intermediate results from the common vertexes of the common vertex table are reused. And thus, a final hyperedge aggregation result of each hyperedge is obtained. According to the method, calculation tasks are executed more compactly, meanwhile, a large amount of unnecessary memory access and communication are avoided through cache reuse, and the calculation efficiency of hypergraph neural network reasoning is improved.
Owner:HUAZHONG UNIV OF SCI & TECH

Key value cache multiplexing method for retrieval enhancement generation system

The invention provides a key value cache multiplexing method for a retrieval enhancement generation system. The method comprises three stages of knowledge retrieval, prompt construction and reasoning generation. In the knowledge retrieval stage, the system carries out vectorization on user input and retrieves related documents, and retrieval results serve as enhanced information; in the prompt construction stage, user input and enhancement information is coded into a token sequence, and the hash value and length of each part of the token sequence are calculated; in the inference generation stage, whether the key value cache is hit or not is judged through hash comparison, the key value cache is directly reused if the key value cache is hit, differential video memory allocation is executed if the key value cache is not hit, a new key value cache is generated in combination with a partition position coding strategy, and finally a result is generated through large language model inference and returned to a user. According to the method, the key value cache reuse rate of the large language model in multi-round reasoning of the retrieval enhancement generation system can be improved, so that the calculation amount and the video memory overhead are reduced.
Owner:HUNAN UNIV

Key-value cache reuse method, and related apparatus

Disclosed in the embodiments of the present application are a key-value cache reuse method, and a related apparatus, which can be applied to scenarios such as cloud technology, artificial intelligence, intelligent transportation, assisted driving and the Internet of Things. The method comprises: after acquiring an ith-round prompt, on the basis of historical token groups respectively generated by a plurality of tokens included in the ith-round prompt with the previous (i-1)th-round prompt, and according to token arrangement orders, matching tokens having the same token arrangement order, so as to obtain a prefix token group and remaining token groups; acquiring from a graphics processing unit a key-value cache of the prefix token group, and if the acquisition fails, acquiring from a central processing unit a key-value cache corresponding to the prefix token group; and sending the key-value cache of the prefix token group and the remaining token groups to an inference engine, such that the inference engine performs inference on the basis of the key-value cache of the prefix token group and the remaining token groups to obtain response content for the ith-round prompt. Thus, an inference engine directly reuses a key-value cache of a prefix token group without requiring recalculation, thereby improving the speed of inference computation.
Owner:TENCENT TECHNOLOGY (SHENZHEN) CO LTD

Intelligent agent tool calling optimization method and system based on combination tool and reflection

The invention provides an intelligent agent tool calling optimization method and system based on a combined tool and reflection. The method comprises the steps that a tool set is obtained, preprocessing and candidate tools are input, unified format conversion is conducted on tool parameters, a parameter log tracking mechanism is established, and a standardized tool set is output. Inputting the standardized tool set, screening a tool combination candidate set with a combinable relationship, and outputting an enhanced candidate tool library containing dynamically generated combination tools; dynamic planning and task execution are achieved through a ReAct normal form, a reusable combination tool is generated, and failure tool elimination and tool parameter generation error correction are achieved through double-layer error analysis and reflection; and finally, forming a system self-evolution dual-drive mechanism by dynamic generation of the combination tool and continuous accumulation of reflection experience. Through cache reuse and reflection mechanism design and through the identification and pre-generation mechanism of the dynamic combination tool, the performance under a multi-tool calling scene is remarkably improved.
Owner:WUHAN UNIV

Automatic Token generation method and device, computer equipment and readable storage medium

The invention discloses an automatic Token generation method and device, computer equipment and a readable storage medium. The automatic Token generation method comprises the steps that firstly, a user input request is acquired and analyzed to obtain a request analysis result; secondly, whether a target Token matched with the analysis result exists or not is inquired through the cache, and if yes, the target Token is directly returned; and if not, generating an undetermined Token and storing the undetermined Token in the cache system, and after the validity of the undetermined Token is verified, taking the undetermined Token as the target Token. According to the method, the repeated generation overhead is reduced through a cache multiplexing mechanism, the adaptability of the Token and the service scene is ensured in combination with a verification process, high efficiency, security and resource optimization of Token generation are realized, the method is suitable for scenes such as high-concurrency interface testing and authority management, and the system response efficiency and the Token reliability are remarkably improved.
Owner:DARK MATTER ARTIFICIAL INTELLIGENT (BEIJING) TECHNOLOGY CO LTD

Large model cluster shunting system and method based on performance perception

The invention discloses a large model cluster shunting system and method based on performance perception. The system comprises a routing decision module, a resource pooling module, a KV cache perception module and a performance perception module. Through dynamic resource pool division, KV cache index matching, instance health state monitoring and load evaluation, and in combination with a hierarchical scheduling strategy and a hit rate load scoring mechanism, intelligent routing of requests and efficient allocation of resources are realized. The method effectively solves the problems of non-uniform resource allocation, KV cache reuse failure, single-point bottleneck, resource waste and the like in a large model reasoning task, remarkably improves the system throughput, reduces the response delay, and has good fault-tolerant and self-recovery capabilities.
Owner:SHANGHAI COMMITTEE CHINA TELECOM GRP LABOR UNION +1

A method and system for scheduling large-scale inference requests based on global state awareness

PendingCN122340185ABatch processingTimeout
This invention discloses a method and system for scheduling large-scale model inference requests based on global state awareness, relating to the technical field of large language model inference systems. This method constructs a network and protocol layer, a session scheduling layer, and an inference engine layer within a single process. Requests are received and session objects are created via a network listening thread. Enqueueing and timeout cleanup are completed using a double-buffered queue and a polling thread. Worker threads claim sessions based on a global state-aware scheduling algorithm, complete protocol parsing and token encoding within the same process, and submit them to the inference engine. The inference engine performs continuous batch processing scheduling and KV cache reuse. Finally, the token is decoded, written back to the client, and session resources are released. This invention, through a co-process fusion architecture and a global state-aware scheduling algorithm, enables the service layer to obtain the internal state of the inference engine in real time and optimize scheduling decisions accordingly, eliminating cross-layer boundary overhead and information asymmetry problems.
Owner:ALL THINGS SEARCH (GUANGZHOU) ARTIFICIAL INTELLIGENCE TECHNOLOGY CO LTD

Multi-modal data generation method and device, medium, equipment and program product

PendingCN121580318AModal dataThresholding
The embodiment of the invention discloses a multi-modal data generation method and device, a medium, equipment and a program product, and the method comprises the steps: determining an input difference between an input feature of a current sampling step and an input feature of a previous sampling step in a process of processing a multi-modal data generation task through a diffusion model; determining a predicted output difference corresponding to the input difference by using a polynomial parameter corresponding to the multi-modal data generation task; in response to the fact that the predicted output difference is smaller than a preset threshold value, determining the cached output feature of the previous sampling step as the output feature of the current sampling step; in response to the fact that the predicted output difference is not smaller than a preset threshold value, based on the input characteristics of the current sampling step, determining the output characteristics of the current sampling step; and in response to completion of denoising iteration, obtaining target data of the multi-modal data generation task. Based on the difference between the input feature of each sampling step and the input feature of the previous sampling step, the cache multiplexing scheme is dynamically adopted, and the data generation speed and the generation quality are improved.
Owner:BEIJING SHENGSHU TECH CO LTD

Cross scenario fall through (XSFT)

A method for optimizing cache reuse in a computational system involves assigning unique identifiers to various Tweaks that affect computational outcomes and injecting Nodes with fields that form a bitmask representing these Tweak dependencies. During execution, the system tracks and updates each Node's dependency information, adjusts the cache key based on the actual Tweak dependencies identified, and stores a proxy in the cache for each executed Node within a Scenario. This proxy enables efficient cache reuse by matching with other Scenarios that share overlapping Tweak dependencies.
Owner:MORGAN STANLEY SERVICES GROUP INC

Signaling processing method, network function network element and computer readable storage medium

The invention discloses a signaling processing method, a network function network element and a computer readable storage medium, and belongs to the technical field of communication. According to the technical scheme of the invention, after a service request carrying a user identification number is received, a target network function network element associated with the user identification number is queried from locally cached first service information, and under the condition that the target network function network element cannot be queried from the first service information, the service request of the user identification number is queried. Sending a service discovery request carrying a user identification number to a network storage function NRF entity, receiving second service information returned by the NRF entity, the second service information including information of a target network function network element and target number segment information associated with the target network function network element, and adding the second service information to the first service information, and updating the first service information. According to the invention, the cache reuse rate of the NF side in the service discovery process in the single number segment scene is improved.
Owner:ZTE CORP

A software component analysis optimization method based on hash cache

The application belongs to the technical field of electric digital data processing, and particularly relates to a software component analysis optimization method based on a hash cache, which comprises the following steps: S1, first scanning of multi-dimensional features and fusion of dynamic behavior features to calculate and store component hashes: when performing first SCA scanning on software, dynamic behavior features of components during running are collected through a lightweight sandbox environment, and basic features, content features and dependency features of software packages and components are extracted; a hash value is introduced as a unique identifier of the software package and the component, the hash value of the component and the corresponding detection result are calculated and stored during the first scanning; during subsequent scanning, the hash value of the component is preferentially verified, if the hash value already exists in the cache library, the historical detection result is directly reused, and it is not necessary to repeatedly perform a full analysis process, under the premise of ensuring analysis accuracy, repeated calculation is reduced through a cache reuse mechanism, and the improvement of SCA analysis efficiency and the reduction of resource consumption are realized.
Owner:SHENZHEN HAIYUNAN NETWORK SECURITY TECH CO LTD

An edge-computing-based advertisement printing collaboration method, device and medium

This invention discloses a collaborative advertising printing method, device, and medium based on edge computing, relating to the field of edge computing technology. The method includes: distributing print data streams to edge computing nodes according to an advertising printing collaborative execution queue; continuously monitoring the changing trends of color compensation amplitude and paper feed compensation amplitude during print data stream correction; generating a compensation instability migration control sheet; based on the compensation instability migration control sheet, marking the continuation positions of unfinished tasks in the advertising printing collaborative execution queue; transferring unfinished tasks to available edge computing nodes in the node collaborative allocation relationship for continued execution; synchronously updating the cache reuse status and carrying stability status; and generating a closed-loop collaborative advertising printing archive. This invention improves the collaborative processing efficiency of advertising printing tasks among multiple edge computing nodes.
Owner:MAIGAOHEYI CULTURE TECH GRP CO LTD

Task processing acceleration method and device, equipment and medium

The invention relates to the technical field of data processing, can be applied to business scenes such as financial science and technology and medical health, and discloses a task processing acceleration method, device and equipment and a medium. The task processing acceleration method comprises the steps that task data attributes are extracted from a to-be-processed task queue, and batch division and priority distribution are carried out according to the attributes; during the period of processing the first task batch, if existence of a second task batch is detected, interrupting current processing and scheduling the current processing to a processing unit for execution; and performing adaptive optimization on the second task batch and adjusting the sparsification degree to generate an intermediate result, storing the intermediate result into the priority cache pool, and generating a final result based on cache multiplexing when a new second task batch is received. According to the method, the high-priority task is preferentially processed during task execution, and the self-adaptive optimization and cache multiplexing mechanism is combined, so that the high-priority task is quickly responded and efficiently processed, the utilization rate of computing resources is increased, and response delay is reduced.
Owner:PING AN TECH (SHENZHEN) CO LTD

Diffusion model time dimension acceleration method for intermediate result caching and reusing

The invention provides a diffusion model time dimension acceleration method for intermediate result caching and reusing, which comprises the following steps of: 1, configuring a cache reusing system which comprises a path planner, a cache and a reusing time counter; 2, a denoising process is simulated in a segmented mode, a path planner is trained in combination with an empirical regular scheme, more than two error accumulation conditions possibly encountered when the system acts on a diffusion model are simulated, output of all calculation and selective calculation of all layers of a backbone network are compared, scheme errors are judged, and the scheme errors serve as a loss function trained by the path planner; according to an empirical regularization scheme, a loss function is improved, cache and reuse frequencies are controlled, and error accumulation caused by continuous reuse is reduced; and step 3, enabling the cache reuse system to act on the diffusion model, and accelerating the image denoising process. According to the method, the overall calculation amount in the denoising process can be reduced, so that acceleration is realized.
Owner:NANJING UNIV

Key value cache multiplexing method and device based on end-side large model

The invention provides a key value cache multiplexing method and device based on an end-side large model, and relates to the technical field of artificial intelligence. According to the method, cross-request multiplexing of key value vectors can be achieved by introducing a target cache table, it is avoided that the end-side large model re-executes calculation of a pre-filling stage for each new request, and the calculation efficiency is improved. The calculation burden and the calculation complexity of the end-side equipment are remarkably reduced, the first word delay is shortened, the overall reasoning efficiency of the end-side large model is improved, and the user experience is improved. In addition, the method can also reduce performance requirements of computation-intensive tasks on the end-side equipment, can be introduced to flexibly adapt to hardware configuration of different end-side equipment, and has high universality and expandability. According to the method, an end-side large model is adopted, the method is completely operated on end-side equipment and does not need to depend on a cloud server, privacy protection and offline operation can be considered, the efficiency of end-side reasoning is guaranteed, meanwhile, a completely localized reasoning scheme is provided, and the requirements of a user for privacy protection and offline operation are met.
Owner:HEFEI IFLY DIGITAL TECH CO LTD

An agent-oriented multi-level cache reuse method

The application relates to the technical field of intelligent agents, and particularly provides a multi-level cache reuse method for intelligent agents. The method comprises the following steps: analyzing each natural language question, generating a question signature, and performing cache retrieval and writing in a question-result level layer; matching the natural language question and historical cache, directly reusing the result if a hit is obtained, entering a strategy planning reuse level layer if a miss is obtained, executing a historical tool strategy according to the question category, reusing the historical tool strategy if a hit is obtained, re-planning by the intelligent agent if a miss is obtained, and generating a new strategy; obtaining a tool list and a parameter list by using the intelligent agent according to the historical tool strategy or the new strategy and the natural language question, reusing the historical tool calling result through consistency matching of tool parameters, and obtaining an execution result; and updating a three-stage cache according to the execution result, so that the method realizes redundancy elimination of tool calling, strategy migration and result short-circuit reuse.
Owner:QILU UNIVERSITY OF TECHNOLOGY (SHANDONG ACADEMY OF SCIENCES) +1

Cache-based data query analysis result optimization method and system

The invention relates to the technical field of computer data processing, and discloses a cache-based data query analysis result optimization method and system, and the method comprises the steps: receiving a client data query request, and extracting an original parameter set; the parameters are subjected to effective cleaning and ASCII code ascending order arrangement, and a unique cache key is generated through normalized splicing and Hash operation; retrieving a cache database according to the result, and returning an analysis result if the result is hit; if the target data is not hit, returning to the source to query the underlying database, and writing the obtained target data into the cache based on an adaptive compression strategy and random expiration time, and in addition, the system establishes a mapping index of a data table and a cache key and monitors a data change event in a binary log of the database, so that active and accurate clearing of the cache is realized. According to the method, the influence of the sequence difference of the request parameters on the cache hit is eliminated through parameter standardization processing, the cache reuse rate is improved by combining adaptive storage and an active failure mechanism, the memory occupation is reduced, and the high real-time performance of data is guaranteed.
Owner:NORTH CHINA MUNICIPAL ENG DESIGN & RES INST

Method for optimizing query throughput of a graph database based on client-side reuse objects

The application discloses a kind of based on client multiplexing object's graph database query throughput optimization method, belong to graph database query optimization technical field, including: receiving from graph database service serialization response data;From the column number of result set in metadata is parsed, and object list containing corresponding number of value object is created;Parsing first row data, including parsing data structure information from metadata and caching, and the first row data parsed is filled into the corresponding value object in object list;Second row and subsequent each row data in result set are parsed, and the data structure information cached and the object list created are reused, and the current row each column data parsed is directly filled into the corresponding value object in object list.The application significantly reduces the dynamic memory allocation times in deserialization process by object reuse, metadata cache reuse and lazy loading optimization, and improves the processing throughput of query response.
Owner:杭州悦数科技有限公司

Key value cache scheduling method and system for language model reasoning

The invention provides a key value cache scheduling method and system for language model inference, and is applied to the technical field of video memory management and key value cache multiplexing of a large-scale language model inference engine. The method comprises the following steps: acquiring an input sequence, extracting a candidate multiplexing fragment, calculating a fragment hash value of the candidate multiplexing fragment, querying a corresponding historical logic block hash list in a fragment index table, and generating a current logic block hash list for the fragment in a pre-filling process, mapping the Hash of the current logic block to the physical address of the historical physical key value cache block through a prefix Hash table to realize logic Hash alias, and scheduling key value cache data in an underlying video memory to participate in autoregressive decoding according to the logic Hash alias; by means of the scheme, prefix tree matching limitation can be broken through, zero-copy cross-context multiplexing of semantic fragments at any position is achieved, pre-filling calculation redundancy and first word delay are remarkably reduced, and GPU video memory utilization efficiency and reasoning throughput under a multi-agent concurrent scene are improved.
Owner:MEMORY TENSOR (SHANGHAI) TECHNOLOGY CO LTD

Content generation method and related equipment

The embodiment of the invention provides a content generation method and related equipment, and the method comprises the steps: indicating the execution of a calling reasoning model through business indication information related to a content generation business, when de-noising processing is carried out on corresponding input content coding characteristics of the noisy data content under a time step tj in the T time steps under the constraint of the first data content, obtaining a cache type for cache multiplexing according to content analysis parameters indicated by a cache multiplexing strategy related to the time step tj; and if the cache type is an inter-step cache type, multiplexing a predicted noise feature corresponding to a time step ti having an inter-step multiplexing relationship with the time step tj, performing denoising processing on an input content coding feature corresponding to the time step tj, and performing content generation based on an output content coding feature corresponding to the time step tj obtained by the denoising processing, the second data content is obtained. Therefore, the model reasoning speed can be effectively improved, and waste of computing resources is avoided.
Owner:TENCENT TECHNOLOGY (SHENZHEN) CO LTD

A large language model key-value cache reuse method and system based on fuzzy semantic matching

The application relates to the technical field of large language model reasoning, and discloses a large language model key-value cache reuse method and system based on fuzzy semantic matching, which comprises the following steps: generating a key-value cache according to word units in a historical reasoning request of a user, gathering multiple word units into a word unit block, generating an embedding vector of the word unit block, and establishing a vector database; calculating the cosine similarity of the embedding vector of a word unit block of a new reasoning request and a historical embedding vector in the vector database; if there is a historical word unit block with a cosine similarity exceeding a set threshold, corresponding key-value cache is obtained through a hash index and reused; calculating an attention score, and distinguishing multiple word units in the current word unit block into key word units and non-key word units based on the attention score; recalculating the key-value cache of the key word units; and composing a hybrid key-value cache by the recalculated key-value cache of the key word units and the reused key-value cache of the non-key word units. The method can expand the key-value cache reuse technology from accurate matching to fuzzy semantic matching without reducing the model accuracy, effectively reduce unnecessary calculation overhead, and further improve the reasoning efficiency.
Owner:UNIV OF SCI & TECH OF CHINA

Key value cache multiplexing method and related device

The embodiment of the invention discloses a key value cache multiplexing method and a related device, which can be applied to scenes such as cloud technology, artificial intelligence, intelligent traffic, auxiliary driving and Internet of Things. After the ith round of cue word is obtained, according to a plurality of segmented words included in the ith round of cue word and historical segmented word groups generated by the previous i-1 rounds of cue word, matching the segmented words with the same segmented word arrangement sequence according to the segmented word arrangement sequence to obtain a prefix segmented word group and residual segmented word groups; and obtaining the key value cache of the prefix segmented word group from the graphics processor, and if the obtaining fails, obtaining the key value cache corresponding to the prefix segmented word group from the central processing unit. And sending the key value cache of the prefix segmented word group and the residual segmented word group to an inference engine, so that the inference engine performs inference according to the key value cache of the prefix segmented word group and the residual segmented word group to obtain response content for the ith round of cue word. Therefore, the inference engine directly multiplexes the key value cache of the prefix segmented word group, recalculation is not needed, and the inference calculation speed is increased.
Owner:TENCENT TECHNOLOGY (SHENZHEN) CO LTD

Multi-level KV cache reuse acceleration system for long context reasoning in RAG scenarios

PendingCN122285720Aimprove hit rateImprove prefix reuse rateParallel computingCache management
A multi-level key-value cache reuse acceleration system for long-context reasoning in RAG scenarios includes: a document retrieval module, a pre-filling task queue module, a multi-level key-value cache module, and a cache management engine. This invention achieves a significantly accelerated system by reducing TTFT and pre-filling overhead without sacrificing model accuracy or modifying the model structure through prefix tree cache management, hierarchical transfer-computation overlap, and asynchronous prefetching. It significantly reduces the first-term delay (TTFT) in the pre-filling stage.
Owner:SHANGHAI JIAOTONG UNIV

Mirror image construction method and device, equipment and storage medium

The embodiment of the invention relates to a mirror image construction method and device, equipment and a storage medium, and the method comprises the steps: obtaining a file change list corresponding to a mirror image layer construction instruction in response to the mirror image layer construction instruction; if the file change list comprises a deleted file list and a newly added file list, respectively constructing mirror image layers according to the deleted file list and the newly added file list to obtain a first mirror image layer corresponding to the deleted file list and a second mirror image layer corresponding to the newly added file list, the first mirror image layer is located below the second mirror image layer; caching the first mirror image layer if the first mirror image layer is cached, and caching the second mirror image layer if the second mirror image layer is not cached. According to the embodiment of the invention, cache multiplexing of cross-mirror-image construction can be realized, and the cache space is effectively saved.
Owner:BEIJING QIYI CENTURY SCI & TECH CO LTD

Analytical database dynamic cache management system based on single plug-in integrated architecture

The invention discloses an analytical database dynamic cache management system based on a single plug-in integrated architecture, which is characterized by comprising a dynamic cache plug-in, a bloom filter module, a pre-judgment type memory control module, a CTE / SQL cache module, a dynamic updating module and a database base table, and the dynamic cache management system is constructed through a five-level collaboration module. Cache optimization and memory security control of SQL query are achieved, the pre-judgment type memory control module actively loads high-frequency data to a memory, disk access is reduced, an intermediate result is reused through CTE or SQL cache, and repeated calculation and hard disk reading are avoided. Compared with the prior art, the method has the advantages of zero memory overload, high memory security, high calculation efficiency and the like, the execution efficiency of repeated query and sub-query in the analytical database is remarkably improved, the calculation overhead and time consumption are reduced, and the problems of resource overload and low calculation efficiency are effectively solved. The method is suitable for high-concurrency analysis scenes such as financial risk control and real-time large screens.
Owner:EAST CHINA NORMAL UNIV

Cost controlled phased resolution generation and cache reuse system and method

The application discloses a cost-controlled phased resolution generation and cache reuse system and method applied to the field of computer technology, which firstly performs rapid preliminary screening at a low resolution level, and removes candidate configurations which are obviously unreasonable in physical structure or whose energy significantly deviates from a threshold; then gradually improves the resolution of the configurations, introduces a physical constraint and an energy calculation model, and gradually approaches a high-fidelity simulation result; meanwhile, a structure feature cache mechanism is arranged at each resolution level, intermediate states and energy gradient information of the calculated configurations are standardized stored and indexed, and are called at a higher resolution level for subsequent same or similar configurations, so that repeated calculation is avoided; finally, through a resolution transition determination module, whether to enter a next resolution level or terminate calculation is dynamically decided according to an energy convergence trend of the current configuration, a structure stability index and a calculation resource budget, so that an optimal balance between physical rationality and calculation efficiency is realized in a global range.
Owner:南通诺瞳奕目医疗科技有限公司 +1