Approximate LRU Cache Eviction via Distributed Data Structure
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In multi-threaded environments, conventional LRU cache eviction mechanisms suffer from contention issues due to multiple threads competing for access to a single linked list, leading to impaired system performance as threads stall waiting for access to the head of the list.
Innovation Solution
Implementing a distributed data structure with multiple linked lists, each associated with a thread, and using atomic operations to update timestamps and evict cache entries, allowing concurrent access and reducing contention by maintaining separate linked lists for each thread.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a single linked list is used for LRU cache eviction, then the LRU policy can be implemented, but multiple threads experience contention and stall when accessing the head of the list
Solution Approach 1:
The patent divides the single shared linked list into multiple separate linked lists, with each thread having its own dedicated linked list for tracking LRU order. This segmentation eliminates contention at the head of the list, as each thread independently maintains its own list without needing to lock shared structures, thereby resolving the contradiction between maintaining correct LRU eviction and achieving high multi-threaded performance
Solution Approach 2:
The patent merges multiple per-thread linked lists into a single global linked list only when eviction is needed. At eviction time, the tail elements from all per-thread lists are combined and compared to identify the overall LRU entry. This merging approach allows concurrent access during normal operations while still enabling correct global LRU eviction when necessary
2Measurement precision
If threads simultaneously update the linked list head, then recent accesses can be tracked, but locking the head prevents other threads from updating the list
Solution Approach 1:
By giving each thread its own separate linked list, the patent eliminates the need for threads to lock a shared head structure when updating access times. Each thread can independently and concurrently update its own list's head without interfering with other threads, thus maintaining precise access time tracking while eliminating thread waiting time
Solution Approach 2:
The per-thread linked lists act as intermediaries that allow threads to update their access patterns without direct contention. Instead of threads competing for access to a single shared list head, each thread uses its own list as an intermediary structure, which is then merged with others only when eviction decisions need to be made
Data Source
AI summary
Exemplary methods for managing cache based on approximate least recently used (LRU) cache entries include maintaining a distributed data structure (DDS) of data elements, each corresponding to a cache entry of a plurality of cache entries, wherein each data element can be atomically accessed by multiple threads. In response to a cache eviction request from a first thread, determining an approximately LRU cache entry among the cache entries based on values atomically accessed from a first subset of the DDS of data elements, wherein the first subset of the DDS is atomically accessed using an atomic instruction without acquiring a lock to prevent another thread from accessing the first subset of the DDS to determine other approximately LRU cache entries among the cache entries, while allowing a second thread accessing a second subset of the DDS substantially concurrently. Evicting the determined approximately LRU cache entry.