Computer Vision Inference Optimization for FPGA vs ASIC Deployment

Overview of Technical Issues:

The inference processing unit on FPGA provides insufficient computational throughput for computer vision algorithms, resulting in slower-than-required inference speed that fails to meet real-time performance targets; the goal is to optimize inference execution to achieve target throughput while balancing power efficiency, development flexibility, and deployment constraints between FPGA and ASIC platforms.

Solution directions generated for this problem

Problem Direction 1 :

ImproveComputational throughput
VS
ConstraintPower consumption

Inspiration 1 : Cross-domain reference

Application Principle: #35 Parameter changes
Cross-domain applicability Assess applicability
Enhancing processing performance of a DNN module by bandwidth control of fabric interface
Innovative Solution Refine solution

Adaptive precision scaling inference engine for dynamic power-throughput optimization

Adaptive precision engine balances throughput and power via dynamic bit-width adjustment
How to solve :
  • Implement runtime precision controller that monitors frame complexity metrics (edge density, texture variance) and adjusts computation bit-width between 4-bit, 8-bit, and 16-bit integer modes per layer — simple frames use 4-bit (0.8W), complex frames scale to 8-bit (1.5W), critical layers to 16-bit (2.2W)
  • Deploy dual-mode MAC arrays on FPGA with configurable datapaths: 4-bit mode processes 4× operations per cycle at 150MHz (600 GOPS), 8-bit mode 2× operations (300 GOPS), 16-bit mode 1× (150 GOPS) — precision selection occurs every 16 clock cycles based on confidence threshold feedback from previous layer output
  • Integrate look-ahead frame analyzer using lightweight 2-bit feature extractor (0.1W overhead) that pre-classifies incoming frames into low/medium/high complexity bins during vertical blanking interval, pre-configuring precision parameters before frame processing starts to eliminate transition latency
Expected Effect : Average throughput 420 GOPS at 1.2W mean power, 35% power reduction vs fixed 8-bit at equal accuracy
Risk Control :
  • precision transition glitches causing inference artifacts
  • confidence threshold miscalibration reducing accuracy below 95%
  • frame analyzer adding 200μs latency overhead

Problem Direction 2 :

ImproveInference execution speed
VS
ConstraintDesign complexity

Inspiration 1 : Cross-domain reference

Application Principle: #1 Segmentation
Cross-domain applicability Assess applicability
Air intake duct of saddle-ridden vehicle
Innovative Solution Refine solution

Modular tile-based inference pipeline with pre-optimized memory layout

Divide inference into independent tiles with local control to boost speed without global coordination
How to solve :
  • Partition the computer vision algorithm into independent processing tiles (4×4 or 8×8 grid), each handling one spatial image region with dedicated local BRAM — eliminates complex global scheduling logic while enabling parallel execution across FPGA fabric
  • Pre-arrange neural network weights in tile-local block RAM during FPGA configuration in sequential access pattern matching MAC operation order — removes runtime address calculation and data shuffling overhead, each tile uses simple increment-only address counters
  • Implement simple handshake-based dataflow between tiles using 2-signal ready/valid protocol — downstream tile pulls data when buffer available, upstream tile stalls when backpressure detected, achieving pipeline flow control with <5 LUTs per interface versus complex credit-based or arbitration schemes
  • Quality control: verify tile output matches golden reference within ±2% accuracy, measure inter-tile latency ≤3 clock cycles, confirm each tile consumes ≤500 FPGA slices, validate throughput meets 30fps minimum at 100MHz clock
Expected Effect : Inference speed +60-80% vs monolithic design; design complexity -40% (no global arbiter); tile replication scales linearly
Risk Control :
  • tile boundary artifacts in output
  • memory bandwidth contention at tile edges
  • timing closure difficulty across distributed tiles

Problem Direction 3 :

ImproveProcessing reliability
VS
ConstraintPower consumption

Inspiration 1 : Cross-domain reference

Application Principle: #11 Beforehand cushioning
Cross-domain applicability Assess applicability
small data communication
Innovative Solution Refine solution

Adaptive reserve capacity activation for reliable FPGA inference under variable workload

Deploy reserve capacity activated only during workload peaks
How to solve :
  • Provision FPGA with 20-30% reserve processing elements that remain clock-gated during normal operation — activate via workload monitor when frame complexity exceeds baseline threshold (e.g., edge density >15% or object count >8 per frame)
  • Implement dual-threshold activation logic: light reserve (10% capacity) triggers at 85% baseline utilization, full reserve (30% capacity) triggers at 95% utilization, with 50ms hysteresis to prevent oscillation
  • Integrate frame-level power budget controller that tracks rolling average power over 100ms window — reserve units auto-deactivate when average power approaches thermal limit (e.g., 90% of 5W budget) while maintaining minimum throughput guarantee of 25 fps
Expected Effect : Reliability +40% under peak load, average power +8% only
Risk Control :
  • threshold calibration for diverse scenes
  • thermal transient during rapid activation
  • reserve unit verification coverage

Problem Direction 4 :

ImproveComputational throughput
VS
ConstraintDesign complexity

Inspiration 1 : Cross-domain reference

Application Principle: #1 Segmentation
Cross-domain applicability Assess applicability
Portable device comprising a touch-screen display, and method for controlling same
Innovative Solution Refine solution

Modular tile-based inference architecture with independent processing regions

Partition FPGA into independent processing tiles for parallel execution
How to solve :
  • Divide the FPGA fabric into 4–16 independent processing tiles, each handling a spatial image region (e.g., 128×128 pixels) or feature channel subset with local control logic only
  • Each tile implements a simple identical MAC array (e.g., 8×8 multiply-accumulate units at 16-bit precision) with dedicated local BRAM (36 Kb per tile) — no global coordination bus or complex arbitration required
  • Deploy lightweight asynchronous handshake protocol (ready/valid signals) between adjacent tiles for boundary data exchange, eliminating centralized scheduling and clock domain crossing logic
Expected Effect : Throughput +300–400% via spatial parallelism; design complexity per tile remains constant; verification effort scales linearly
Risk Control :
  • tile boundary synchronization failure
  • BRAM resource exhaustion per tile
  • routing congestion between tiles
Patsnap Eureka Solution