Stochastic K-Means Inference Circuit for Low-Power Online Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Convolutional neural networks (CNNs) face challenges in operating with reduced bit resolution during both feed-forward and learning modes, particularly due to noisy scaling operations and limited memory, which limits their efficiency in power-constrained hardware implementations.
Innovation Solution
A stochastic K-means clustering method is introduced for CNNs, where template updates are performed probabilistically rather than deterministically, allowing for incremental or decremental changes in memory elements based on the difference between input data and template values, thereby avoiding the need for floating-point operations and accommodating limited bit-resolution memory.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If deterministic K-means clustering with scaling operations is used, then learning accuracy is improved, but power consumption increases and hardware implementation becomes complex
Solution Approach 1:
The patent replaces expensive floating-point arithmetic operations with cheap integer arithmetic operations. The stochastic update rule uses only addition, subtraction, and comparison operations on integer values, eliminating the need for complex floating-point units in hardware. This allows accurate learning to be achieved through simple, low-power integer computations that can be implemented efficiently in hardware.
Solution Approach 2:
The patent changes the mathematical parameters of the K-means algorithm from continuous floating-point values to discrete integer values. The stochastic update rule modifies template elements by integer increments or decrements based on probabilistic decisions, rather than using continuous scaling operations. This parameter transformation enables accurate learning while reducing power consumption through simpler arithmetic.
2Measurement precision
If floating-point operations are used for K-means clustering, then learning precision is improved, but device complexity increases
Solution Approach 1:
The patent substitutes complex floating-point arithmetic operations with simple integer arithmetic operations. The stochastic update rule requires only basic integer addition, subtraction, and comparison operations, which can be implemented with simple logic circuits rather than complex floating-point units. This substitution dramatically reduces hardware complexity while maintaining learning precision through the probabilistic update mechanism.
3Measurement precision
If high bit-resolution memory is used, then computation accuracy is improved, but power consumption and memory requirements increase
Solution Approach 1:
The patent uses low-bit-resolution integer representations for template elements and input data, replacing the need for high-bit-resolution floating-point memory. The stochastic update rule operates efficiently on these compact integer values, achieving accurate learning with minimal memory resources. This approach trades off some representation precision for dramatically reduced memory requirements and power consumption.
4Speed
If standard deterministic K-means is implemented, then convergence speed is improved, but adaptability to limited resolution hardware decreases
Solution Approach 1:
The patent introduces dynamic stochastic updates that adapt to the discrete nature of integer arithmetic. Rather than using fixed deterministic scaling operations that break down at low bit resolutions, the stochastic update rule dynamically adjusts template elements through probabilistic integer increments or decrements. This dynamic approach maintains convergence while adapting to the constraints of limited-resolution hardware.
Solution Approach 2:
The patent transforms the K-means algorithm parameters from continuous floating-point domain to discrete integer domain. The stochastic update rule uses probability distributions over integer changes rather than continuous scaling factors. This parameter transformation enables the algorithm to converge effectively on hardware with limited bit resolution, achieving both speed and hardware adaptability.
Data Source
AI summary
Described is an inference circuit for pattern recognition for use within convolutional neural nets for online learning using K-means clustering. The inference circuit includes a set of templates, each template having a template data memory. The inference circuit also include at least one match element, the match element being operable for receiving an input data pattern and determining a degree of match between the input data pattern and a template. A best match logic circuit is included for selecting a template in the set of templates that best matches the input data pattern, said template being a best match template. Finally, an updated is included for probabilistically modifying the template data memories based on the best match template.