Neural Network Binarization via Stochastic Beta Activation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural networks face challenges in being deployed on resource-constrained devices due to their large size and computational intensity, particularly when dealing with floating-point operations.
Innovation Solution
The technique involves binarizing node outputs by augmenting existing activation function outputs with a stochastic node sampled from a Beta distribution during training, allowing for a quasi-binary regime. At inference time, this stochastic activation is substituted with a thresholding operation for binary representation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If floating point variables are used to represent neural network outputs, then model accuracy is maintained, but memory usage and computational complexity increase significantly
Solution Approach 1:
The patent changes the parameter representation from floating point to binary by introducing a stochastic activation function that outputs binary values. This parameter transformation maintains model accuracy while dramatically reducing memory usage from 32-64 bits per value to just 1 bit per value.
Solution Approach 2:
The patent introduces a stochastic activation function as an intermediary component between the weighted sum calculation and the output. This intermediary uses Beta distribution sampling to generate binary outputs, serving as a bridge that preserves information while enabling binary representation.
2Measurement precision
If floating point operations are performed in neural networks, then computational accuracy is maintained, but processing power requirements increase
Solution Approach 1:
The patent transforms computational operations from floating point arithmetic to binary operations by changing the output parameter type. Binary operations require significantly less processing power while the stochastic activation function ensures computational accuracy is maintained through probabilistic information preservation.
3Quantity of substance
If quantization is applied to reduce model size, then memory usage decreases, but information loss occurs and model quality deteriorates
Solution Approach 1:
The patent introduces a stochastic activation function as an intermediary that prevents information loss during quantization. By using Beta distribution sampling, the intermediary preserves probabilistic information that would otherwise be lost in traditional deterministic quantization, maintaining model quality while achieving binary model size reduction.
Solution Approach 2:
The patent makes the activation function dynamic and stochastic rather than static and deterministic. This dynamic approach allows the binary outputs to adaptively represent the underlying continuous values through probability distributions, preventing information loss that would occur with fixed quantization thresholds.
4Quantity of substance
If traditional binarization is applied to neural network outputs, then model size is reduced, but training becomes infeasible due to non-differentiability
Solution Approach 1:
The patent introduces a stochastic activation function as a trainable intermediary during the training phase. This intermediary is differentiable and uses gradient descent to optimize parameters, making training feasible. At inference time, it produces binary outputs, achieving model size reduction without sacrificing trainability.
Solution Approach 2:
The patent creates a dynamic system where the activation function behaves differently during training versus inference. During training, it is stochastic and differentiable for gradient computation; during inference, it produces deterministic binary outputs. This dynamic behavior resolves the contradiction between trainability and binary representation.
Data Source
AI summary
Persistent storage contains a representation of a neural network including an input layer, and output layer, and a hidden layer, wherein nodes of the hidden layer incorporate serialized activation functions, wherein the serialized activation functions for each of the nodes include a sigmoid function and a Beta function, wherein the sigmoid function is applied to weighted outputs from nodes of a previous layer of the neural network, wherein the Beta function is applied to a conductance hyper-parameter and respective outputs of the sigmoid function, and wherein outputs of the Beta function are provided to a subsequent layer of the neural network. One or more processors are configured to train the neural network until the outputs of the sigmoid function for the nodes of the hidden layer are substantially binary.