Programmable Vector Engine for Beam Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Beam search operations in machine learning, particularly in natural language processing, are computationally expensive and memory-intensive due to the need for extensive software sorting algorithms, leading to poor throughput and long latency.
Innovation Solution
A computational pipeline with native hardware support for beam search operations is designed, utilizing a series of arithmetic logic unit (ALU) stages with feedback paths to efficiently find the largest values and their indices in tensors, and new machine instructions (Max-N, Find-Index-N, and Match-Replace-N) are introduced to streamline these operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If software sorting algorithms are used for beam search operations, then the operations can be performed with general-purpose hardware, but the computational overhead and memory usage increase significantly
Solution Approach 1:
The patent replaces software-based sorting algorithms with dedicated hardware circuitry including comparison units, selection units, and feedback registers. This substitution transforms the mechanical/software sorting process into an optimized hardware system that performs beam search operations with reduced computational overhead and improved efficiency.
2Adaptability or versatility
If software sorting algorithms are used for beam search operations, then the operations can be performed with general-purpose hardware, but the memory usage increases significantly
Solution Approach 1:
The patent extracts the beam search functionality from general-purpose software and implements it in dedicated hardware components. By taking out the sorting and selection operations and implementing them in hardware with feedback registers and selection units, the system reduces memory usage while maintaining operational capability.
3Adaptability or versatility
If software sorting algorithms are used for beam search operations, then flexibility is maintained, but the throughput decreases and latency increases
Solution Approach 1:
The patent segments the beam search operation into distinct hardware components: comparison units for evaluating candidates, selection units for choosing top-k results, and feedback registers for maintaining state. This segmentation allows parallel processing and improves throughput while maintaining operational flexibility through configurable parameters.
Solution Approach 2:
The patent replaces software-based beam search operations with dedicated hardware circuitry including comparison units, selection units, and feedback registers. This substitution transforms the mechanical/software sorting process into an optimized hardware system that performs beam search operations with reduced computational overhead and improved efficiency.
Data Source
AI summary
To perform a beam search operation on an input tensor using a data processor with native hardware support, the data processor can be programmed with a set of instructions. The set of instructions can include a first machine instruction that operates on the input tensor to obtain N largest values in the input tensor, a second machine instruction that operates on the input tensor to obtain indices corresponding to the N largest values in the input tensor, and a third machine instruction that operates on the input tensor to replace the N largest values in the input tensor with a minimum value.