Accelerator Latency Reduction via Translation Hints
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional accelerator designs face significant latency overhead due to the need for virtual address to physical address translation, which hinders low-latency processing in data analytics and Big Data workloads.
Innovation Solution
The implementation of techniques that provide translation hints from CPU cores to accelerators, using mechanisms such as the Enq_with_translations instruction and IOMMU snooping, allows accelerators to start processing jobs early by utilizing cached TLB translations for input data and buffering output data until physical translations are verified.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional address translation mechanisms are used in SVM-capable accelerators, then the accelerator can access virtual memory address space, but latency overhead increases due to address translation requirements
Solution Approach 1:
The patent applies preliminary action by having the CPU core provide address translation hints to the accelerator before the accelerator needs to access the memory. The hint includes a virtual address and an associated physical address that the CPU has previously determined, allowing the accelerator to bypass the normal address translation process and directly access the correct memory location, thereby reducing latency while maintaining virtual memory access capability
2Reliability
If address translation is performed for each buffer access, then correct memory access is ensured, but processing speed decreases due to translation overhead
Solution Approach 1:
The CPU core performs the address translation in advance and provides the physical address as a hint to the accelerator. This preliminary action ensures that the translation is performed by the more capable CPU translation mechanism, maintaining accuracy, while the accelerator can use the pre-translated physical address to access memory directly, improving processing speed by eliminating repeated translation operations
Solution Approach 2:
The system implements feedback by having the accelerator verify the address translation hint against the actual memory access requirements. If the hint is correct, the accelerator uses it directly; if verification fails, the accelerator can request proper address translation, ensuring reliability while optimizing for speed in the common case where hints are accurate
Data Source
AI summary
Methods, apparatus and associated techniques and mechanisms for reducing latency in accelerators. The techniques and mechanisms are implemented in platform architectures supporting shared virtual memory (SVM) and includes use of SVM-enabled accelerators, along with translation look-aside buffers (TLBs). A request descriptor defining a job to be performed by an accelerator and referencing virtual addresses (VAs) and sizes of one or more buffers is enqueued via execution of a thread on a processor core. Under one approach, the descriptor includes hints comprising physical addresses or virtual address to physical address (VA-PA) translations that are obtained from one or more TLBs associated with the core using the buffer VAs. Under another approach employing TLB snooping, the buffer VAs are used as lookups and matching TLB entries ((VA-PA) translations) are used as hints. The hints are used to speculatively pre-fetch buffer data and speculatively start processing the pre-fetched buffer data on the accelerator.