Dynamic Code Loading for Sequential Neural Network Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In computing systems, especially those utilizing neural networks, the instruction buffer often needs to be refilled frequently due to hardware limitations, requiring continuous reloading of instruction code during inference processes, which can be inefficient and resource-intensive, especially for sequential processors lacking flow control capabilities.
Innovation Solution
The solution involves dynamically creating two instruction blocks at runtime: a no-operation (NOP) block and a reset block, which are stored in memory alongside a regular code block. The NOP block is used for initial inferences, while the reset block is used for the Nth inference to reset the DMA queues, allowing for continuous execution without external processor intervention and reducing storage requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Duration of action of stationary object
If the instruction buffer is refilled frequently during inference, then the neural network can continue executing, but the system efficiency decreases and resource consumption increases
Solution Approach 1:
The patent applies preliminary action by pre-creating and storing multiple instruction blocks (including NOP blocks and reset blocks) in memory before the inference process starts. This allows the instruction buffer to be refilled in advance without interrupting the execution of current instructions, thereby maintaining continuous operation while improving efficiency through reduced runtime interruptions.
Solution Approach 2:
The patent segments the instruction code into multiple discrete blocks (regular code blocks, NOP blocks, and reset blocks) that can be independently managed and loaded into the instruction buffer. This segmentation enables selective loading of appropriate blocks based on execution needs, allowing continuous inference while optimizing resource usage by loading only necessary instruction segments.
2Ease of operation
If external processor intervention is used to manage instruction buffer refilling, then flow control is achieved, but the computational burden on external processors increases
Solution Approach 1:
The patent implements self-service by enabling the neural network processor to autonomously manage its own instruction buffer refilling process. The processor internally generates and executes NOP blocks and reset blocks without requiring external processor intervention, thereby maintaining flow control capabilities while eliminating the computational burden on external processors.
Solution Approach 2:
The patent introduces intermediary instruction blocks (NOP blocks and reset blocks) that mediate between the execution engine and the external environment. These intermediary blocks handle the complex tasks of buffer management and flow control internally, allowing the system to maintain operational ease while avoiding external processor involvement.
3Duration of action of stationary object
If multiple instruction blocks are stored in memory, then continuous execution is enabled, but storage requirements increase
Solution Approach 1:
The patent applies local quality by storing different types of instruction blocks (NOP blocks, reset blocks, regular code blocks) in specific locations within memory, each optimized for its particular function. This localized organization allows the system to enable continuous execution through selective loading of appropriate blocks while minimizing overall storage requirements by only loading necessary blocks into the instruction buffer at any given time.
Solution Approach 2:
The patent implements dynamics by making the instruction buffer loading process adaptive and flexible. The system dynamically selects which instruction blocks to load based on the current execution state and requirements, rather than statically storing all possible instructions. This dynamic approach enables continuous execution capability while reducing storage requirements by loading only necessary blocks on demand.
Data Source
AI summary
Embodiments include techniques for enabling execution of N inferences on an execution engine of a neural network device. Instruction code for a single inference is stored in a memory that is accessible by a DMA engine, the instruction code forming a regular code block. A NOP code block and a reset code block for resetting an instruction DMA queue are stored in the memory. The instruction DMA queue is generated such that, when it is executed by the DMA engine, it causes the DMA engine to copy, for each of N inferences, both the regular code block and an additional code block to an instruction buffer. The additional code block is the NOP code block for the first N−1 inferences and is the reset code block for the Nth inference. When the reset code block is executed by the execution engine, the instruction DMA queue is reset.