Tiered Memory Allocation for Multi-Core Cache Thrashing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data management methods for multi-core, multi-thread environments are sub-optimal due to issues like data cache thrashing, shared memory updates by multiple cores, and inefficient load distribution, leading to poor resource utilization and increased latency.
Innovation Solution
A tiered memory allocation system with 'big blocks', 'intermediate blocks', and 'small blocks' is introduced, where each block size is optimized for cache line efficiency, and an intermediate block manager handles allocation and deallocation using push-down lists, allowing lock-free operations and minimizing memory fragmentation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If a general purpose operating system uses generic algorithms for memory management and scheduling, then the system is easy to operate and maintain, but the resource utilization is sub-optimal and latency increases
Solution Approach 1:
The application directly manages its own memory allocation and scheduling needs without relying on generic OS algorithms. The application implements custom memory management that segments data according to its specific access patterns and lifetime requirements, enabling optimal resource utilization while maintaining ease of operation through automated management.
Solution Approach 2:
The system changes memory management parameters by introducing tiered memory structures with different allocation strategies. Instead of uniform memory management, the system creates multiple memory tiers with different characteristics (fast access, large capacity, etc.) and dynamically assigns data to appropriate tiers based on access patterns, improving both resource utilization and ease of operation.
2Productivity
If the application is hard coded to directly control its own scheduling with minimal executive, then resource utilization is efficient, but the application becomes difficult to program and has little flexibility
Solution Approach 1:
The system introduces an intermediary memory management layer that bridges the application and the hardware. This intermediary provides high-level abstractions for memory allocation and management while automatically handling the low-level details of scheduling and resource allocation, enabling efficient resource utilization without requiring direct hardware control in the application code.
Solution Approach 2:
The application is segmented into independent data structures with explicit lifetime management. Each data structure can be independently allocated and deallocated based on its specific requirements, allowing the system to optimize resource utilization while maintaining programming flexibility through modular design.
3Productivity
If data is divided horizontally into different memory pools with different processors, then load distribution is improved, but additional processing is required to route requests and latency increases
Solution Approach 1:
The system adds a temporal dimension to data management by introducing lifetime-aware allocation. Instead of only spatial distribution across memory pools, the system considers the lifetime of data structures and allocates them accordingly. This enables better load distribution while reducing latency by keeping frequently accessed data in optimally positioned memory regions.
Solution Approach 2:
The system performs preliminary allocation and positioning of data structures based on their lifetime characteristics and access patterns. By pre-organizing data in memory tiers and pools before it is actually accessed, the system eliminates the need for additional routing processing and reduces latency while maintaining effective load distribution.
4Stability of the object's composition
If slab allocators are used for memory management, then memory fragmentation is reduced, but cache thrashing increases as active data is spread out over large memory regions
Solution Approach 1:
The system applies different memory management qualities to different data structures based on their specific characteristics. Instead of uniform slab allocation, the system creates localized memory regions with optimized characteristics for different data types and access patterns. This enables effective memory fragmentation management while improving cache performance by keeping related data together in appropriately sized memory regions.
Solution Approach 2:
The memory allocation system becomes dynamic and adaptive, adjusting allocation strategies based on runtime observations of data access patterns and lifetime characteristics. The system dynamically reorganizes memory regions to optimize both fragmentation management and cache performance, rather than using static slab allocations that compromise one aspect for the other.
Data Source
Figure 1
Figure 2
Figure 3A~3
AI summary
A method for managing memory in a system for an application, comprising: assigning a first block (i.e., a big block) of the memory to the application when the application is initiated, the first block having a first size, the first block being assigned to the application until the application is terminated; dividing the first block into second blocks (i.e., intermediate blocks), each second block having a same second size, a second block of the second blocks for containing data for one or more components of a single data structure to be accessed by one thread of the application at a time; and, dividing the second block into third blocks (i.e., small blocks), each third block having a same third size, a third block of the third blocks for containing data for a single component of the single data structure.