Sparse Kernel Convolution via Parallel MAC and Load Instructions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional image processing systems face inefficiencies in convolving images with sparse kernels due to the inability to execute vector and scalar load operations in a single clock cycle, leading to wasted clock cycles and high power consumption, as well as significant memory bandwidth requirements.
Innovation Solution
An image processing system that retrieves a merged kernel with non-zero coefficients and corresponding skip values, allowing for parallel execution of multiply-accumulate and load instructions in one clock cycle, eliminating the need to load rows associated with zero coefficients and optimizing memory bandwidth.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If conventional vector and scalar load operations are executed sequentially, then the operations can be performed with standard architecture, but the convolution cannot be completed in a single clock cycle leading to low processing speed
Solution Approach 1:
The patent merges the vector load operation and scalar load operation into a single unified load operation that can be executed in parallel with the MAC operation. This is achieved by organizing kernel coefficients into contiguous memory locations and using a single load instruction to fetch both the row of image data and the required kernel coefficient simultaneously, thereby enabling completion of convolution in one clock cycle without increasing architectural complexity
Solution Approach 2:
The patent reorganizes the storage layout of kernel coefficients from a traditional scattered arrangement to a contiguous linear arrangement in memory. By mapping multi-dimensional kernel access patterns onto a one-dimensional contiguous memory structure, the system enables simultaneous loading of multiple data elements through a single memory operation, achieving parallel execution without requiring complex multi-dimensional memory addressing
2Productivity
If multiplication and accumulation operations are performed with zero weight coefficients, then the complete convolution process is followed, but clock cycles are wasted and power consumption increases
Solution Approach 1:
The patent extracts and eliminates operations involving zero weight coefficients from the convolution process. By identifying and removing redundant multiplications and accumulations with zero values, the system performs only necessary computational operations. This is achieved through sparse kernel representation where zero coefficients are not stored or processed, directly increasing effective throughput while reducing power consumption proportional to the number of eliminated operations
Solution Approach 2:
The patent implements skipping of unnecessary load operations for rows associated with zero coefficients. The system detects when a kernel coefficient is zero and skips the corresponding vector load and MAC operation entirely, rushing through the convolution process by only performing computations on non-zero elements. This reduces both the number of executed instructions and the associated power consumption while maintaining correct convolution results
3Adaptability or versatility
If multiple kernels are used for various image processing functions, then diverse processing capabilities are achieved, but memory bandwidth requirements increase significantly
Solution Approach 1:
The patent implements a universal kernel storage and access mechanism that serves multiple image processing functions through a single memory interface. By designing the kernel data structure and access pattern to accommodate different kernel types and convolution operations, the system achieves multi-functionality without requiring separate dedicated memory paths for each operation, thereby reducing overall memory bandwidth requirements while maintaining versatility
Solution Approach 2:
The patent performs preliminary organization of kernel coefficients into an optimized contiguous layout before convolution operations begin. By pre-processing the kernel data to arrange coefficients in an access-efficient manner, the system reduces the memory bandwidth required during actual convolution operations. This preliminary arrangement enables batch loading of multiple kernel coefficients and minimizes repeated memory accesses across different processing functions
Data Source
AI summary
An image processing system for convolving an image includes processing circuitry that is configured to retrieve the image including a set of rows, a merged kernel, multiple skip values and a pixel base address. The merged kernel includes all non-zero coefficients of a set of kernels. Each skip value corresponds to a location offset of each non-zero coefficient with respect to a previous non-zero coefficient. Further, the processing circuitry is configured to execute a multiply-accumulate (MAC) instruction and a load instruction parallelly in one clock cycle for multiple times, on the set of rows and the merged kernel to convolve the image with the merged kernel. Each row on which the MAC and load instructions are executed is associated with a corresponding non-zero coefficient and a corresponding skip value. The load instruction is executed based on the pixel base address, the corresponding skip value, and a width of each row.