Multi-Tier Cache Allocation for Fair Multi-Tenant Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional in-memory caching systems for multiple tenants face issues of unfair cache utilization, leading to increased latency and inefficiency due to either overutilization by some tenants in shared caching or underutilization of dedicated caching.
Innovation Solution
Implementing a multi-tiered caching system with dedicated and shared allocations, where data is migrated between tiers based on access time and hit ratios, using eviction rules to optimize cache usage and ensure fair access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If tenants are provided dedicated shares of the cache for fairness, then cache access latency for individual tenants is reduced, but cache utilization efficiency deteriorates due to idle memory
Solution Approach 1:
The cache is segmented into two distinct tiers: a dedicated allocation tier that guarantees minimum cache space for each tenant to ensure fairness and acceptable latency, and a shared allocation tier that allows efficient utilization of remaining cache space. This segmentation resolves the contradiction by providing both dedicated protection and shared efficiency.
Solution Approach 2:
The system dynamically manages data placement between dedicated and shared cache allocations based on access patterns, hit ratios, and cache fullness. Data can be migrated from dedicated to shared allocation when appropriate, and the system adapts to changing conditions in real-time, optimizing both fairness and utilization efficiency dynamically.
2Loss of energy
If tenants are allowed to store data anywhere in a shared cache, then cache utilization efficiency is improved, but cache access latency increases due to unfair resource consumption
Solution Approach 1:
The cache is divided into dedicated and shared allocations, where the dedicated portion protects each tenant from being displaced by others, ensuring fair access and acceptable latency. The shared portion allows efficient utilization by any tenant, resolving the contradiction between fairness and efficiency.
Solution Approach 2:
The system introduces an intermediary management layer that controls data placement and migration between dedicated and shared allocations. This intermediary ensures that shared cache usage does not negatively impact dedicated tenant performance, mediating between the competing requirements of efficiency and fairness.
3Loss of energy
If data is migrated from dedicated allocation to shared allocation, then cache utilization efficiency is improved, but reliability of dedicated cache access deteriorates
Solution Approach 1:
The dedicated allocation tier maintains guaranteed cache space for each tenant, ensuring reliable access. Migration to shared allocation only occurs for data that can tolerate being in the shared tier, and the segmented structure ensures that dedicated reliability is preserved while enabling shared efficiency.
Solution Approach 2:
The system changes parameters such as access frequency thresholds and hit ratios to determine when data should be migrated from dedicated to shared allocation. By carefully selecting which data to migrate based on these parameter changes, the system improves utilization while preserving the reliability needed for critical data in the dedicated tier.
Data Source
AI summary
Embodiments of the invention are directed to systems and methods for utilizing a multi-tiered caching architecture in a multi-tenant caching system. A portion of the in-memory cache may be allocated as dedicated shares (e.g., dedicated allocations) that are each dedicated to a particular tenant, while another portion of the in-memory cache (e.g., a shared allocation) can be shared by all tenants in the system. When a threshold period of time has elapsed since data stored in a dedicated allocation has last been accessed, the data may be migrated to the shared allocation. If data is accessed from the shared allocation, it may be migrated back to the dedicated allocation. Utilizing the techniques for providing a multi-tiered approach to a multi-tenant caching system can increase performance and decrease latency with respect to conventional caching systems.