Hardware Accelerator Top-K Probability Transmission
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Mobile devices experience delays and high latency when performing computations for applications like optical character recognition (OCR) due to the large number of computations typically handled by the host processor, leading to inefficient power consumption.
Innovation Solution
A hardware accelerator is used to process slices of input data using a machine learning model, determining top K probabilities and transmitting only these results to the host processor, reducing the computational load and power consumption by distributing computations across multiple compute units and minimizing data transmission over buses like PCIe.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the host processor performs all computations for machine learning inferences, then the application functionality is complete, but the latency is high and power consumption is excessive
Solution Approach 1:
The patent divides the computational workload into segments by introducing a hardware accelerator that handles specific machine learning inference computations separately from the host processor. The host processor prepares input data and receives results, while the hardware accelerator performs the computationally intensive neural network computations, effectively segmenting the processing tasks to reduce latency and improve overall productivity.
2Use of energy by moving object
If the host processor handles all computations, then the system architecture is simple, but the power consumption is high due to the large number of computations
Solution Approach 1:
The patent introduces a hardware accelerator as an intermediary component between the host processor and the machine learning model. This intermediary is specifically designed to perform neural network computations efficiently, reducing the power consumption burden on the host processor while maintaining high computation throughput. The hardware accelerator acts as a specialized mediator that handles computationally intensive tasks with lower energy consumption.
3Loss of information
If all probability values are transmitted from the hardware accelerator to the host processor, then the host processor has complete information, but the data transmission overhead increases
Solution Approach 1:
The patent extracts only the essential information from the hardware accelerator's computations and transmits it to the host processor. Instead of transmitting all probability values, the system identifies and transmits only the top-K highest probability values, which contain the most important information for making inference decisions. This extraction principle reduces data transmission volume while preserving the necessary information completeness.
Data Source
AI summary
A hardware accelerator can receive, from a host processor, a slice of input data at a time-step. The hardware accelerator can process the input data using a machine learning model deployed on the hardware accelerator to compute a respective probability among multiple probabilities for each of multiple classes. The respective probability for each class being a likelihood that content in the slice belongs to the class. The hardware accelerator can determine, from the multiple probabilities, a preset number of highest probabilities for the slice of input data. The hardware accelerator can transmit the preset number of highest probabilities for the slice to the host processor. Related apparatus, systems, techniques and articles are also described.