Work Scheduler Offload for Low-Latency Memory Allocation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Memory management in cloud computing systems, particularly in data centers, is resource-intensive and contributes to latency and jitter due to the high CPU cycles required for memory allocation and de-allocation, especially when using software implementations like TCMalloc, glibc's malloc, and jemalloc.
Innovation Solution
A work scheduler is employed to manage memory allocation and deallocation tasks, offloading these operations from the CPU to a queue management device (QMD) that uses a work scheduler to streamline memory operations, reducing the need for software locking and enabling efficient management of central caches and page heaps.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If software-based memory management (TCMalloc, glibc's malloc, jemalloc) is used, then memory allocation can be implemented, but CPU cycles are consumed and latency increases
Solution Approach 1:
The patent replaces software-based memory management mechanisms with a hardware-based queue management device. The QMD uses dedicated hardware circuits including a work scheduler, allocation circuitry, and garbage collection circuitry to manage memory allocation and deallocation, eliminating the need for CPU-intensive software implementations like TCMalloc, glibc's malloc, or jemalloc. This substitution of mechanical (software) system with hardware-based system directly addresses the contradiction by reducing CPU cycle consumption while maintaining memory allocation functionality.
Solution Approach 2:
The patent introduces a queue management device (QMD) as an intermediary between the CPU and memory management operations. The QMD receives memory allocation requests from the CPU, processes them through dedicated hardware circuits, and returns allocated memory without requiring the CPU to directly manage memory operations. This intermediary approach offloads the memory management burden from the CPU, reducing latency and CPU cycle consumption while maintaining efficient memory allocation.
2Productivity
If CPU performs memory allocation operations, then memory can be allocated, but CPU utilization increases
Solution Approach 1:
The patent extracts memory management operations from the CPU and implements them in a dedicated queue management device. The QMD contains specialized hardware circuits including allocation circuitry that handles memory allocation requests independently of the CPU. By taking out memory management functions from the CPU, the system maintains high memory management throughput while significantly reducing CPU utilization, as the QMD processes allocation requests using its own dedicated hardware resources rather than consuming CPU cycles.
3Reliability
If software locking is used for memory management, then memory allocation can be coordinated, but system complexity increases
Solution Approach 1:
The patent replaces software locking mechanisms with hardware-based synchronization and coordination within the queue management device. The QMD uses dedicated hardware circuits including a work scheduler and allocation circuitry that coordinate memory allocation operations without requiring software locks. The hardware-based approach provides reliable coordination of memory allocation across multiple CPU cores while eliminating the complexity of software locking mechanisms, as the QMD's internal hardware architecture inherently manages concurrency and resource access.
Data Source
AI summary
Examples described herein relate to a work scheduler that includes at least one processor and at least one queue. In some examples, the work scheduler receives a request to allocate a region of memory and based on availability of a memory segment associated with a central cache to satisfy the request to allocate a region of memory, provide a memory allocation using an available memory segment entry associated with the central cache from the at least one queue. In some examples, the work scheduler assigns a workload to a processor and controls when to pre-fetch content relevant to the workload to store in a cache or memory accessible to the processor based on a position of the workload in a work queue associated with the processor.