Neural Processing Device Dual-Mode TLB Updates for Faster Translation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural processing units (NPUs) face limitations in efficiently translating virtual addresses to physical addresses due to the reliance on a single update mode for the translation lookaside buffer (TLB), which hinders high-performance deep-learning tasks.
Innovation Solution
The neural processing device employs a dual-mode TLB update mechanism, combining pull and push modes to optimize address translation, with criteria-based update signals for efficient address translation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If TLB update is performed only in pull mode (when TLB miss occurs), then the system complexity is reduced, but the address translation speed and deep-learning task performance deteriorate
Solution Approach 1:
The patent implements a dynamic TLB update mechanism that can switch between pull mode and push mode based on workload characteristics. The command processor determines whether to generate push mode update signals based on criteria such as whether the workload is for the next layer of a neural network model, front layer, or single stream inference. This dynamic adaptation allows the system to optimize address translation speed for deep-learning tasks while maintaining manageable complexity through conditional logic rather than always using the more complex push mode.
Solution Approach 2:
The patent changes the operational parameters of the TLB update mechanism by introducing different update modes (pull mode and push mode) with different triggering conditions. In pull mode, updates occur on TLB miss; in push mode, the command processor proactively generates update signals based on workload type and memory access patterns. This parameter change enables the system to adapt the update frequency and timing to match the specific requirements of deep-learning workloads, thereby improving address translation speed without unnecessarily increasing complexity.
2Productivity
If TLB is updated proactively in push mode based on workload criteria, then address translation efficiency is improved, but the control logic complexity increases
Solution Approach 1:
The patent applies preliminary action by having the command processor proactively generate push mode TLB update signals before TLB misses occur. The command processor evaluates workload characteristics (such as whether the task is for the next layer of a neural network model, front layer, or single stream inference) and memory access patterns in advance, and issues update signals accordingly. This preliminary action prevents future TLB misses and improves address translation efficiency for predictable deep-learning workloads, while the complexity is managed by limiting push mode to specific workload types rather than all operations.
3Productivity
If dual-mode TLB update mechanism is implemented, then deep-learning task performance is enhanced, but the device complexity increases
Solution Approach 1:
The patent implements a dynamic TLB update mechanism that can switch between pull mode and push mode based on workload characteristics. The command processor determines whether to generate push mode update signals based on criteria such as whether the workload is for the next layer of a neural network model, front layer, or single stream inference. This dynamic adaptation allows the system to optimize address translation speed for deep-learning tasks while maintaining manageable complexity through conditional logic rather than always using the more complex push mode.
Solution Approach 2:
The patent applies local quality by implementing push mode TLB updates selectively for specific types of deep-learning workloads rather than uniformly for all operations. The command processor evaluates workload characteristics and applies push mode updates only when criteria are met (such as next layer processing, front layer processing, or single stream inference with on-chip memory access). This localized application of the more complex push mode mechanism enhances performance where needed while avoiding unnecessary complexity for other workload types.
Data Source
AI summary
A neural processing device and a method of updating translation lookaside buffer thereof are provided. The neural processing device includes at least one processor module each of which includes at least one micro translation lookaside buffer (TLB), a hierarchical memory that is accessed by the at least one micro TLB, and a command processor configured to update the at least one micro TLB in a push mode by generating a first update signal which indicates update of the at least one micro TLB and transmitting the first update signal to the at least one micro TLB.