165 results about "Cache miss" patented technology

Techniques for cache management for LLM processing are described. Example embodiments include a signal hashing model that generates a key for particular context data. An LLM output corresponding to the context data is stored in a cache along with the key. For a user input received by the system, a cache lookup is performed using a key for context data corresponding to the received user input. For a cache hit, the stored output is used to respond to the user input. For a cache miss, a LLM processes the context data and the user input to generate an output within a first timeout. If the LLM is unable to generate an output within the first timeout, then in some cases, the LLM is allowed to continue processing until a second timeout, and a final or partial output from the LLM is stored in the cache.

The invention relates to the technical field of message pushing, in particular to an SaaS (Software as a Service) intelligent concurrent pushing system which comprises a tenant flow monitoring module, a resource binding module, a lock-free object writing module, a handle layout construction module and a cache prefetching delivery module. According to the method, the high-frequency business object is identified by monitoring the flow characteristics of the tenants, and the complete Slab page is dynamically applied and bound to the thread exclusive local allocation buffer area, so that physical isolation of bottom layer resources is realized to eliminate global lock competition and context switching overhead in a multi-thread environment; lock-free memory writing is executed based on a privatized pointer, the object construction efficiency is greatly improved, a compact storage structure in which a bitmap is adjacent to a connection handle is constructed, a hardware cache prefetching mechanism is triggered by utilizing bitmap reading, connection data is loaded to a CPU cache line in advance, and the connection data is stored in the cache line. And the cache miss rate is reduced and the concurrent delivery performance of massive messages is remarkably improved by utilizing a memory locality principle.

A cache writeback circuit is disclosed for writing back, without invalidating, dirty cache lines. The cache writeback circuit is configured to enter an active state based on detecting a trigger condition indicative of cache misses to a memory cache circuit within a memory hierarchy of a computer memory subsystem causing cache line eviction activity. During the active state, the cache writeback circuit is configured to identify a set of dirty cache lines in the memory cache circuit, and write back, without invalidating, cache lines of the identified set of dirty cache lines from the memory cache circuit to a memory circuit within a lower level of the memory hierarchy, such as DRAM. The cache writeback circuit may further be configured to identify the dirty cache lines via a cache walk operation, which can be suspended, for example, when a higher-priority cache operation occurs.

Caching techniques can include: receiving, from a host, a read I / O operation requesting to read current content of a logical address; determining whether a data cache includes a data cache entry corresponding to the logical address; responsive to determining the data cache includes the data cache entry corresponding to the logical address, performing data cache hit processing to service the read I / O operation using the data cache entry; responsive to determining the data cache does not include the data cache entry corresponding to the logical address, performing data cache miss processing including: determining whether a mapping cache includes a descriptor corresponding to the logical address; and responsive to determining the mapping cache includes the descriptor corresponding to the logical address, performing mapping cache hit processing to service the read I / O operation using the descriptor of the mapping cache.

The invention discloses a method for accelerating secure metadata access in a secure memory system, a memory controller and a system, and belongs to the field of secure memory systems, and the method comprises the following steps: when a page table item corresponding to a logic page where data accessed by a processor is located does not hit a TLB, obtaining the page table item from a memory page table, extracting a physical page address from the page table item, and storing the physical page address in a memory; a counter corresponding to a physical page where the data to be accessed is located and a father node of the counter in the integrity tree are prefetched through the physical page address; adding a replacement dirty block address in a miss request sent by the last level of cache, after receiving the miss request containing a field of the replacement dirty block address, executing conventional memory reading and decryption, positioning a counter corresponding to the replacement dirty block address, and performing prefetching by using an idle memory bandwidth; in addition to the secure metadata cache, the prefetching queue is maintained to temporarily store the prefetched metadata. The cache hit rate of the security metadata in the security memory system can be improved, and the performance overhead caused by the cache miss can be reduced.

An apparatus and a method for performing an authenticated encryption with associated data (AEAD) operation of an encrypted instruction and a golden tag stored in a memory device in an event of a cache miss are provided. The apparatus includes a bus control circuit, a block buffer, a tag buffer and an AEAD circuit. The bus control circuit receives a read address from a cache for reading the encrypted instruction and the golden tag from the memory device. The block buffer receives and stores the encrypted instruction from the bus control circuit, wherein a size of the block buffer is preset to be N times a size of one cache line. The tag buffer receives and stores the golden tag from the bus control circuit. The AEAD circuit performs the AEAD operation upon the encrypted instruction and the golden tag to check whether the encrypted instruction is tampered or not.

The invention discloses a distributed data preloading and querying method and device and a storage medium. The method comprises the steps of searching data corresponding to a query key in a cache region integrated by an application server; if the query result of the query key is hit in the cache region, returning the query result; if the query result of the query key is not hit in the cache region, determining whether a target query key which is the same as the query key exists in the executed database query; and if the target query key does not exist, initiating a query request based on the query key to the database, and storing a query result returned by the database into a cache region integrated by the application server to respond to the query of the same query key. According to the method and the device, the cache is integrated in the application server, the resource overhead of an independent cache server is eliminated, the concurrent query when the cache is missed is merged and processed, and the concurrency is changed into single query, so that the repeated request pressure of the database is reduced fundamentally, and the risk of cache breakdown is effectively prevented.

The invention discloses an optimization method and device for an embedded linked list container, electronic equipment and a storage medium, and relates to the technical field of computers, the method comprises the steps that a linked list node is embedded into a host data structure to serve as a member variable, extra memory occupation of a pointer domain in a traditional linked list is omitted, and the optimization efficiency is improved. Physical storage of the nodes and the host data structure is continuous, so that cache missing can be reduced; meanwhile, the stability of the linked list under high concurrency is guaranteed through atomized insertion and deletion operations, and a traditional independent node structure and a non-atomized operation are not adopted; the technical problems that in the prior art, a traditional linked list node pointer domain occupies an extra memory, so that expenditure is increased, discontinuous node physical distribution causes cache missing, access delay is increased, and system abnormal safety and operation reliability are affected by iterator failure under high concurrency can be solved. And the technical effects of reducing the memory overhead, reducing the access delay and improving the abnormal safety and the operation reliability of the system are achieved.

Methods and systems for cache miss monitoring and fulfillment are disclosed. A disclosed method comprises receiving, from a processing unit, a cache access request, determining, based on a cache failing to fulfill the cache access request, that a cache miss has occurred, populating a request buffer based on the cache miss and the cache access request, populating a transaction buffer with a transaction entry for the request buffer based on the cache miss request and the cache access request, determining by the request buffer, that information requested in the cache access request should be retrieved from main memory, determining, by the transaction buffer if information was not retrieved, that the cache access request satisfies criteria for creating a cache miss tag, creating the cache miss tag based on the cache access request, and storing the cache miss tag in a portion of the cache.

The invention provides a key value cache multiplexing method for a retrieval enhancement generation system. The method comprises three stages of knowledge retrieval, prompt construction and reasoning generation. In the knowledge retrieval stage, the system carries out vectorization on user input and retrieves related documents, and retrieval results serve as enhanced information; in the prompt construction stage, user input and enhancement information is coded into a token sequence, and the hash value and length of each part of the token sequence are calculated; in the inference generation stage, whether the key value cache is hit or not is judged through hash comparison, the key value cache is directly reused if the key value cache is hit, differential video memory allocation is executed if the key value cache is not hit, a new key value cache is generated in combination with a partition position coding strategy, and finally a result is generated through large language model inference and returned to a user. According to the method, the key value cache reuse rate of the large language model in multi-round reasoning of the retrieval enhancement generation system can be improved, so that the calculation amount and the video memory overhead are reduced.

The invention discloses an Ethernet-based distributed cache coherence protocol, an implementation method, an implementation device and an implementation system. A plurality of computing nodes are connected through a packet switching network. Each computing node comprises a CPU / GPU (Central Processing Unit / Graphics Processing Unit) and a local cache thereof, and is provided with a cache agent. The far-end memory is organized in home nodes, and each home node manages a part of physical address space and is equipped with a directory controller. When the CPU of the computing node accesses a far-end memory address and does not hit in the local cache, the CA of the computing node replaces the far-end memory address and communicates with the DC managing the address through the network so as to maintain the cache consistency of the data among all the nodes. Based on a cache consistency protocol of a directory, the CXL.cache consistency of a plurality of independent computing nodes can be maintained in a low-overhead and high-reliability mode on a high-delay and lossy packet switching network, and broadcast storm caused by a monitoring protocol is avoided.

The invention discloses an optimization method and system for direct I / O read performance under a Linux system. The method comprises the steps that a cache control mark used for controlling cache access is introduced; in the file opening stage, whether a cache mechanism is started or not is judged, if yes, whether a cache is hit or not is detected when direct I / O reading operation is executed, data are directly read from the cache and returned when the cache is hit, standard direct I / O reading and data returning are called when the cache is not hit, meanwhile, asynchronous caching is conducted on the data, and a cache mark is set; when the direct I / O write operation is executed, whether a cache mechanism is started or not is judged, if yes, whether the cache is hit or not is detected firstly, when the cache is hit, original cache data is set to be invalid, asynchronous caching is conducted on the data, meanwhile, standard direct I / O write-in data is called, and after asynchronous caching and data write-in operation are completed, the cache data is set to be valid. According to the invention, the reading performance of direct I / O can be improved.

implemented method, system, and computer program product for minimizing the effects of cache thrashing involving non-temporal workloads. The cache activities of a workload, including the cache activities (e.g., number of cache hits) involving local and peer caches, are monitored. Based on analyzing the metrics of such monitored cache activities, a determination is made as to whether a non-temporal workload is identified. For example, such a determination may be based on comparing the metrics of the monitored cache activities of the workload to a threshold value. Upon identifying a non-temporal workload, the cache line(s) associated with the non-temporal workload are identified. The persistence policy for the identified cache line(s) is then altered. For example, the persistence policy for the identified cache line(s) may be altered by reducing the tenure of such a cache line(s) thereby reducing the number of cache misses or evictions and minimizing the effects of cache thrashing.

The invention relates to the technical field of large model recommendation, and discloses a large model recommendation algorithm based on a KV cache lightweight optimization technology. The algorithm comprises the steps of obtaining a user historical behavior data sequence, and performing cleaning and standardization processing to obtain standardized user behavior data; the method comprises the steps that a user behavior vector is mapped to an embedding space to generate a user behavior vector, a lightweight key value cache model is trained in combination with an item feature vector in an item library, keys are embedded according to user requests, and values are embedded according to recommended items. When user recommendation requests are received, request features are extracted and mapped into request vectors, and the requests are classified according to the similarity of the vectors and keys in the key value cache; if the cache is hit, directly reading a corresponding value as a recommendation result; and if the cache is not hit, processing the request vector by adopting a pre-training large model to generate a recommendation result, and storing the result and a corresponding key into a key value cache. According to the algorithm, through KV cache lightweight design, the recommendation effect is guaranteed, meanwhile, the large model calling frequency is reduced, and the recommendation response speed is increased.

The invention is suitable for the technical field of cache management, and provides a cache replacement method and system combining pseudo-random and recent minimum access prediction.The method comprises the following steps that a pseudo-random replacement algorithm based on lfsr is defaulted to be used for cache line replacement; monitoring a cache miss event, and recording miss address information in the CMHQ module; when the miss frequency of the data stored in the FIFO in the CMHQ module exceeds a preset threshold value, a replacement algorithm of the cache group is switched into a rrip algorithm; the rrip resources are dynamically managed, and when the cache group is not frequently replaced any more, the rrip resources are recycled, and the pseudo-random algorithm is reused; according to the composite algorithm based on the combination of the pseudo-random algorithm of lfsr and the rrip algorithm, the advantages of the pseudo-random algorithm and the rrip algorithm are taken into consideration, the advantage that few resources are used as the pseudo-random algorithm is achieved, and the advantage that the rrip algorithm has the advantage that the effect on various loads is good is also achieved.

The invention relates to a data interface high-concurrency processing method based on multi-level cache, and belongs to the technical field of computer software. According to the method, an SDK interface receives a user data query request; the request firstly arrives at the local cache, the system queries according to the Key, if the request is hit, the data is directly returned, and the process is ended; if the local cache is not hit, querying a distributed cache Redis; if the distributed cache is hit, returning the data to the user, and asynchronously writing the data back to the local cache for subsequent request use; if the Redis still does not hit, query is executed through the unified data access service; the database query result is backfilled to the distributed cache and the local cache, and a subsequent request can obtain data from the distributed cache and the local cache. Through an automatic adaptation mechanism, data consistency and access correctness among the databases are guaranteed, and the problem of cross-database compatibility is effectively solved.

Technology is described for an object cache layer for a rules engine. The object cache layer may store derived objects. The object cache layer may take advantage of machine learning for incoming objects that have variable attributes. A trainable hash function may use a machine learning model to predict the incoming event schema and signature of derived objects from the incoming objects or queries. The trainable hash function may determine an incoming event schema and signature of a derived object using the machine learning model and a set of attributes of an incoming object. A cache manager of the object cache layer may use a hash value determined by the trainable hash function using the signature of the incoming object to determine whether to access the derived object in the cache. The trainable hash function may be trained at runtime using training signatures from the rules engine on cache misses.

The embodiment of the invention relates to a data processing method, device and system for cache mode management and a storage medium. Cache modes comprise a fixed cache mode and an adjustable cache mode, and the data processing method for cache mode management comprises the following steps: counting the number of times of continuous cache miss, and comparing with the storage capacity of a unit cache line; if the number of times of cache miss is equal to the storage capacity of the unit cache line, setting the cache mode to be an adjustable cache mode; if the number of times of cache miss is not equal to the storage capacity of the unit cache line, setting the cache mode to be a fixed cache mode; in the fixed cache mode, the storage capacity of the cache unit address is fixed; in the adjustable cache mode, the storage capacity of the cache unit address changes along with the change of the length of the data request. According to the data processing method, through a logic self-adaptive dynamic adjustment mode, the performance is ensured, and the waste of bandwidth is avoided.

The invention provides an artificial intelligence acceleration method and system based on heterogeneous hardware, and relates to the technical field of computers. According to the method, the cache miss rate, the instruction throughput and the chip power consumption waveform when the heterogeneous hardware executes an artificial intelligence reasoning task are collected; performing mining processing to obtain a performance jitter value, correlating with a calculation link corresponding to the artificial intelligence reasoning task to generate cache sensitivity, extracting waveform features to obtain an access index, taking the two as input, performing reinforcement learning to obtain a mapping strategy, and monitoring a task execution state to trigger a predetermined condition; and the priority adjustment amount is dynamically calculated and preemptive scheduling is executed, so that artificial intelligence reasoning task acceleration can be realized, and the response efficiency and the energy efficiency ratio in a high-concurrency scene are improved.

This invention relates to the field of computer technology, providing a request management method, apparatus, electronic device, and storage medium. The method includes: upon detecting a cache miss in a data access request, checking whether an incomplete request entry corresponding to a first address exists in a preset request management structure based on a first address carried by the data access request. The preset request management structure is a pre-constructed multi-way set-associative cache structure for grouping and managing cache miss requests. If the incomplete request entry exists in the preset request management structure, adding the data access request to the waiting queue of the incomplete request entry. This invention, by employing a multi-way set-associative cache structure for grouping and managing cache miss requests, enables parallel lookups of cache miss requests, significantly improving the processing efficiency of cache miss requests.

This invention relates to a processor pipeline system, a long-distance jump processing method, and related equipment. The system is optimized for long-distance jump scenarios. During the instruction fetch stage, long-distance jump detection is designed for the jump address, and an L0 instruction cache is added to the cache system. When a long-distance jump occurs, the instruction fetch timing is optimized by simultaneously querying multiple levels of cache, thereby saving additional cache read time caused by potential cache misses. Furthermore, based on the fast read / write performance of the L0 instruction cache, this invention designs a pre-fetch instruction method, which can further reduce instruction read latency and optimize processor performance.

The invention relates to a low-overhead processor cache micro-architecture defense method and device and computer device.The low-overhead processor cache micro-architecture defense method comprises the steps that when a missing state keeping register module takes out a loading instruction waiting for data from a replay queue, a target branch mask of the loading instruction is obtained; the replay queue is used for storing an instruction which is stagnated due to miss of the cache; judging whether the loading instruction is in a speculative execution state or not according to the target branch mask; and when the loading instruction is in the speculative execution state, stopping a cache write-in operation corresponding to the loading instruction. Through the method and the device, the problem of sensitive information leakage caused by incapability of defending against the cache side channel attack is solved, defending against the cache side channel attack is realized, and sensitive information leakage is prevented.

The invention discloses a memory access management method for a heterogeneous multi-core system oriented to a deep neural network, belonging to the storage system structure field of a computer system. The invention utilizes CPU-GPU heterogeneous multi-core system to accelerate DNN model training, and designs a memory access controller according to its memory access characteristics. In the DNN training process, the final cache shared by multiple cores is unloaded, preretrieved and released, and the fine-grained data transmission process significantly improves the utilization rate of the final cache. In addition, the memory access controller also designed a delay hiding mechanism, by overlapping the access process of a large number of intermediate data in the feature extraction layer and the calculation process, reducing the calculation performance loss caused by the miss of the final level cache and the need to wait for the memory access response of DRAM during the model calculation process, and optimizing the training efficiency.

The invention discloses a cache implementation method and device, and the method comprises the steps: setting a plurality of replacement algorithms for a cache, one of the replacement algorithms being a main replacement algorithm, and the rest being auxiliary replacement algorithms; and circularly executing the steps according to a preset frequency: counting an accumulated value of a cache miss number of each replacement algorithm; determining the cache hit rate of each replacement algorithm according to the accumulated value of the cache miss number; when the replacement algorithm corresponding to the highest cache hit rate is an auxiliary replacement algorithm, setting the auxiliary replacement algorithm corresponding to the highest cache hit rate as a new main replacement algorithm; wherein when the cache receives the access request, the cache miss number of each replacement algorithm is updated. The cache hit rate can be improved.

The application provides an RISC-V multi-core heterogeneous platform intelligent load balancing method and system, and relates to the technical field of resource allocation and scheduling. Micro-architecture performance data of each processing core in the RISC-V multi-core heterogeneous platform is acquired to construct a state vector; the micro-architecture performance data comprises instruction cycle number, cache miss rate at each level and memory pause proportion; the state vector is input into a pre-trained deep Q network model to generate optimal action instructions, so that the load balancing of thread resources and core capacity is realized; the optimal action instructions comprise thread migration, thread exchange and core frequency adjustment; the pre-training of the deep Q network model is performed on a parallel computing program running on the RISC-V platform; the environment is randomly disturbed before the program runs; the action is executed through a random strategy, and state, action and reward data are collected; an offline experience dataset is constructed to perform pre-training. Dynamic, cooperative and adaptive optimization of parallel computing of the RISC-V multi-core heterogeneous platform is realized.

Techniques are disclosed relating to generating user-specific time-to-live (TTL) values using machine learning. In various embodiments, a server system maintains a cache data store that is operable to store data for a plurality of users of a web service. In response to a cache miss for a request from a first one of the plurality of users, the server system may generate a user-specific TTL value for the first user. In various embodiments, generating the user-specific TTL value may include using a machine learning model to generate a predicted future access pattern for the first user that indicates a distribution of time periods during which the first user is expected to access the web service and, based on the predicted future access pattern, determining the user-specific TTL value for the first user.

Systems, apparatuses, and methods are disclosed for arbitrating threads in a computing system. The computing system includes a processor having a plurality of cores that are each capable of concurrently processing instructions of a plurality of threads. When a thread throttling unit receives an indication that a shared cache has a resource contention, the throttling unit sets a cache miss threshold for the cache. If the number of cache misses exceeds the threshold, the throttling unit notifies a particular upstream computing unit to throttle the processing of instructions of the thread. After a time period elapses, if the cache continues to exceed the threshold, the throttling unit notifies the upstream computing unit to more restrictively throttle the thread by performing one or more of decreasing a selection rate and increasing the time period. Otherwise, the unit notifies the upstream computing unit to less restrictively throttle the thread.

A processor includes a time counter and issuing instructions and executing instructions at a future time, which is based on the time counter. The execution times are based on fixed latency times of instructions, with the exception of the load instruction, which is based on the data cache hit latency time. A data cache miss causes the load instruction to fetch data from the level 2 cache, wherein a time tracker unit adjusts the level 2 cache latency time based on a counter.

The application provides a cache allocation method, device, equipment, medium and program product, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring a multi-feature data stream, and extracting a target feature vector of the multi-feature data stream; inputting an expected cache miss rate set by a user in advance and the target feature vector into a cache strategy decision model, outputting an optimal cache eviction strategy, and inputting the optimal cache eviction strategy, the expected cache miss rate set by the user in advance and the target feature vector into a cache capacity prediction model, and outputting a target cache capacity allocation result. In the application, real-time data streams are taken as inputs to realize optimal strategy selection of a cache eviction strategy pool and optimal resource allocation of cache capacity respectively. Through mutual promotion of working capabilities of the cache strategy decision model and the cache capacity prediction model, the two cache management modes are efficiently realized, the management cost is greatly reduced, and the resource efficiency is improved.

The invention relates to a post-link optimization method and device based on software and hardware collaboration and a storage medium. The method comprises the steps that a branch record storage method is designed in a CPU core in advance; after the program runs, the running process of the program is rebuilt, the positions where cache miss and branch prediction errors occur are located, and meanwhile, a profile file is generated according to the format of the profile file of a compiler and used for post-link optimization of the program. According to the method, the specific positions of events such as cache miss and branch prediction errors can be conveniently positioned. Meanwhile, the branch record of program operation can be used for generating a profile file compatible with a compiler, and data support is provided for post-link optimization of the program.