FPGA AI Inference Acceleration via Layer Profiling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing FPGA-based AI solutions lack flexibility, leading to inefficiencies in AI inference speed performance, particularly at the edge, where power consumption and cost are concerns, and new AI model topologies or layers with better accuracy and efficiency are rapidly developed, causing delays in deployment due to the need for updates in custom AI accelerator IP cores.
Innovation Solution
A method utilizing FPGAs for AI inference software stack acceleration, involving quantization of neural network models, layer-by-layer profiling, identification of compute-intensive layers, and implementation of acceleration using layer accelerators, which can be either library-provided or custom, to enhance inference speed without increasing cost or power usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If custom AI accelerator IP cores are used on FPGA, then hardware acceleration capability is improved, but flexibility and adaptability to new AI models deteriorate
Solution Approach 1:
The patent implements a dynamic hybrid architecture where the FPGA fabric can be reconfigured at runtime to support different AI layers. The system includes a layer accelerator for common operations and a programmable computing unit that can dynamically adapt to handle new or unsupported layer types, allowing the hardware to evolve with new AI models without physical redesign
Solution Approach 2:
The patent creates a universal AI inference engine that combines multiple functions: a layer accelerator for standardized operations, a programmable computing unit for custom operations, and a memory subsystem. This multi-functional design allows the same hardware platform to support both traditional accelerated layers and new emerging AI model topologies through software configuration
2Productivity
If more powerful mobile GPU or MCU is used, then inference speed performance is improved, but cost and power consumption increase
Solution Approach 1:
The patent segments the AI inference workload into two parts: compute-intensive layers that are accelerated by dedicated FPGA layer accelerators, and other layers that run on the programmable fabric. This segmentation allows efficient hardware acceleration of bottleneck operations while maintaining overall system power efficiency, avoiding the need for a more powerful GPU/MCU
Solution Approach 2:
The patent replaces general-purpose mobile GPU/MCU mechanics with specialized FPGA hardware acceleration. By substituting the mechanical processing units with FPGA-based layer accelerators for compute-intensive operations, the system achieves higher inference speed without the power consumption penalty of upgrading to more powerful mobile processors
3Adaptability or versatility
If IP core is updated to support new AI layers, then support for new models is improved, but design cycle time increases
Solution Approach 1:
The patent performs preliminary action by pre-implementing a comprehensive layer accelerator that covers most common AI layers, and designing a programmable computing unit that can handle new layer types through configuration rather than hardware redesign. This preliminary setup enables rapid adaptation to new models without lengthy design cycles
Solution Approach 2:
The system implements dynamic adaptability where the programmable computing unit can be reconfigured at runtime or through firmware updates to support new AI layers. This dynamic approach allows the hardware to evolve with new models without requiring physical redesign or long requalification cycles
Data Source
AI summary
The present invention relates to a method of using field-programmable gate array (FPGA) for artificial intelligence (AI) inference software stack acceleration which combines the advantages of flexibility from the AI inference software stack and the programmable hardware acceleration capability of the FPGA, wherein said method comprises the steps of performing quantization on neural network (NN) model, performing layer-by-layer profiling of said NN model using AI inference software stack, identifying compute-intensive layer type of said NN model and implementing acceleration using layer accelerator on said compute-intensive layer type.