Dynamic Boltzmann Machine Limited Connections
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional dynamic Boltzmann machines with hidden units face limitations in bidirectional training and variance of gradient estimation, leading to inefficiencies in learning processes.
Innovation Solution
Implementing limited connections in the dynamic Boltzmann machine model where current observations depend only on the latest hidden units and all previous observations, while hidden units are independent of older units, allowing for polynomial-time gradient computation and optimization using stochastic Gradient Descent.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If bidirectional training is used in conventional dynamic Boltzmann machines, then both forward and backward models can be trained, but all previous data must be stored during training period and computational complexity increases
Solution Approach 1:
The patent segments the training process into unidirectional forward-only training, eliminating the need for bidirectional training. By removing the backward model and its associated time-reversal requirements, the computational complexity and data storage requirements are significantly reduced while maintaining training effectiveness through the proposed limited connection constraints.
Solution Approach 2:
The patent extracts and removes the backward model component from the training architecture. By taking out the unnecessary backward training mechanism, the system achieves simpler computation and reduced memory requirements while still achieving reliable training through the forward model with limited connections.
2Productivity
If Monte Carlo estimator is used for gradient computation, then gradient can be approximated by sampling, but variance caused by sampling slows down learning
Solution Approach 1:
The patent changes the parameter constraints of the Boltzmann machine by imposing limited connections where current observations depend only on latest hidden units and all previous observations, while latest hidden units are independent of older hidden units. This parameter change enables exact gradient computation in polynomial time, eliminating sampling variance and accelerating learning while maintaining precision.
3Adaptability or versatility
If unconstrained connectivity is used in Boltzmann machines, then model flexibility is high, but learning efficiency is insufficient for practical problems
Solution Approach 1:
The patent applies local quality by imposing specific connection constraints on different parts of the network. Current observations are allowed to connect to latest hidden units and all previous observations, while latest hidden units are constrained to be independent of older hidden units. This localized constraint strategy maintains necessary model flexibility for practical problems while achieving sufficient learning efficiency.
Data Source
AI summary
A computer-implemented method, a computer program product, and a computer processing system are provided for online learning for a Dynamic Boltzmann Machine (DyBM) with hidden units. The method includes imposing, by a processor device, limited connections in the DyBM where (i) a current observation x[t] depends only on latest hidden units h[t-1/2] and all previous observations x[<t] and (ii) the latest hidden units h[t-1/2] depend on all the previous observations x[<t] while being independent of older hidden units h[t-1/2]. The method further includes computing, by the processor device, gradients of an objective function. The method also includes optimizing, by the processor device, the objective function in polynomial time using a stochastic Gradient Descent algorithm applied to the gradients.