Hardware Accelerator Direct Fixed Point Conversion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current machine learning (ML) models face inefficiencies in inference latency and power consumption due to the need for sequential conversions between different data formats, particularly when processing data from cameras that generate unsigned integer (uint) data, which requires conversion to floating point and then to int8 format for processing by neural processing units (NPUs).
Innovation Solution
Implementing a hardware accelerator that performs direct fixed point to fixed point conversions, mimicking floating point precision through high precision parameters, thereby avoiding intermediate floating point conversions and simplifying preprocessing and quantization operations into a single-step operation, reducing latency and power consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If sequential conversions between uint, float, and int8 formats are performed, then data precision is maintained, but inference latency and power consumption increase
Solution Approach 1:
The patent combines multiple sequential conversion operations (uint to float, then float to int8) into a single direct fixed-point conversion operation. This merging eliminates the intermediate floating-point representation step, reducing the number of operations while maintaining precision through carefully designed scaling factors and offset values that directly map uint to int8 format.
Solution Approach 2:
The invention extracts and eliminates the unnecessary intermediate floating-point conversion step from the data processing pipeline. By removing this intermediate representation, the system achieves direct conversion from uint to int8 format, reducing latency and power consumption while preserving the essential precision requirements through optimized fixed-point arithmetic.
2Measurement precision
If sequential conversions between uint, float, and int8 formats are performed, then data precision is maintained, but power consumption increases
Solution Approach 1:
The patent combines multiple sequential conversion operations (uint to float, then float to int8) into a single direct fixed-point conversion operation. This merging eliminates the intermediate floating-point representation step, reducing the number of operations while maintaining precision through carefully designed scaling factors and offset values that directly map uint to int8 format.
Solution Approach 2:
The invention replaces the computationally expensive floating-point conversion pathway with a more efficient fixed-point conversion approach. By using simpler fixed-point arithmetic operations with pre-computed scaling factors, the system achieves the same precision goals with significantly lower power consumption, effectively substituting expensive operations with cheaper alternatives.
3Loss of time
If direct fixed point to fixed point conversion is performed, then inference latency and power consumption are reduced, but conversion precision may be compromised
Solution Approach 1:
The patent employs parameter changes by introducing optimized scaling factors and offset values that are specifically tuned for the direct uint to int8 conversion. These parameters are designed to compensate for the lack of intermediate floating-point representation, ensuring that the direct fixed-point conversion maintains precision equivalent to or better than the sequential conversion pathway.
Solution Approach 2:
The invention applies preliminary action by pre-computing scaling factors and conversion parameters during system initialization or model training. These pre-computed parameters are then used during inference to enable accurate direct conversion without requiring runtime floating-point calculations, thus achieving both speed and precision.
4Measurement precision
If sequential CPU operations are used for data conversion, then conversion accuracy is maintained, but CPU resources are consumed and other tasks are delayed
Solution Approach 1:
The patent substitutes the general-purpose CPU-based sequential conversion mechanism with a specialized hardware accelerator designed for direct fixed-point conversion. This hardware substitution performs the conversion in parallel with other processing tasks, maintaining accuracy through dedicated conversion logic while freeing CPU resources for higher-level task management and other computational work.
Solution Approach 2:
The invention segments the data processing pipeline by separating the conversion operation from the main CPU workflow. The direct fixed-point conversion is handled by a dedicated hardware module that operates independently, allowing the CPU to focus on task coordination and other computationally intensive operations without being burdened by sequential conversion overhead.
Data Source
AI summary
Artificial intelligence (AI) operation is improved by combining pre-processing with quantization and post-processing with dequantization. Floating point conversion may be implemented as fixed point to fixed point conversion. Floating point conversion and precision may be mimicked, for example, using high precision parameters in a fixed point to fixed point conversion. Mimicking floating point using hardware acceleration reduces sequential operations, such as machine learning model preprocessing and quantization by a CPU, to one or two clock cycles in a single step operation. Accordingly, computing resources, such as computing device cameras, may provide raw data to a hardware accelerator configured to quickly render the input in the correct format to an inference model by simultaneously performing preprocessing and quantization, substantially reducing inference latency and device power consumption while freeing up a CPU for other tasks.