Model Estimation Device Using Variational Inference

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing methods for estimating latent states and parameters in latent variable models, such as LDA, face challenges with high computational complexity, especially when the number of parameters increases with samples, leading to significant approximation errors.

Innovation Solution

A model estimation device and method that acquires observed data and sets initial values for variational probabilities and parameters, computes and optimizes these values to maximize the lower bound of the marginal model posterior probability, allowing for efficient estimation of latent states and parameters without losing theoretical validity.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If Laplace approximation is used to approximate the complete marginal likelihood function, then computational complexity is reduced, but approximation error increases significantly when the number of parameters increases with samples

Engineering Contradiction:

Improvecomputational speedVSAvoidapproximation accuracy

Core Design Contradiction:

ProductivityVSMeasurement precision

Solution Approach 1:

The patent changes the approach from approximating the complete marginal likelihood function to maximizing a lower bound of the marginal model posterior probability. This parameter change in the optimization target allows the method to maintain accuracy even when the number of parameters increases with samples, resolving the contradiction between computational speed and approximation accuracy.

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The patent replaces the mechanical system of Laplace approximation with a variational inference approach that maximizes a lower bound. This substitution allows the system to handle models where the number of parameters increases with samples without sacrificing accuracy, while maintaining computational efficiency.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Ease of manufacture

If the number of latent states is set beforehand, then estimation can be performed, but the model cannot adapt to data with varying numbers of latent states

Engineering Contradiction:

Improvemodel estimation feasibilityVSAvoidadaptability to varying latent state numbers

Core Design Contradiction:

Ease of manufactureVSAdaptability or versatility

Solution Approach 1:

The patent makes the number of latent states dynamic rather than fixed. By using a Dirichlet process prior, the model can automatically determine the number of latent states based on the data, allowing it to adapt to varying numbers of latent states while maintaining estimation feasibility through variational inference.

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The patent performs preliminary action by setting up a Dirichlet process prior that enables automatic determination of the number of latent states. This preliminary setup allows the model to adapt to data with varying numbers of latent states without requiring manual specification, resolving the contradiction between estimation feasibility and adaptability.

Inventive Principle:

Principle #10Preliminary action

3Adaptability or versatility

If a nonparametric Bayesian method using Dirichlet process is used, then the number of latent states can be estimated automatically, but computational complexity becomes extremely high

Engineering Contradiction:

Improveautomatic estimation of latent statesVSAvoidcomputational complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent replaces the computationally intensive nonparametric Bayesian method with a variational inference approach that maximizes a lower bound. This substitution maintains the ability to automatically estimate the number of latent states while dramatically reducing computational complexity through more efficient optimization.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The patent uses a cheaper, more efficient variational inference approach instead of the expensive nonparametric Bayesian method. This disposable approach provides sufficient accuracy for automatic latent state estimation without the computational burden of full Dirichlet process inference.

Inventive Principle:

Principle #27Cheap short-living objects (Disposable)

Data Source

PatentUS9489632B2Model estimation device, model estimation method, and information storage medium

Publication Date: 2016.11.08 NEC CORP

AI summary

A model estimation device includes: a data input unit; a state number setting unit; an initialization unit which sets initial values of a variational probability of a latent variable, a parameter, and the type of each component; a latent variable variational probability computation unit which computes the variational probability of the latent variable so as to maximize a lower bound of a marginal model posterior probability; a component optimization unit which estimates an optimal type of each component and a parameter thereof so as to maximize the lower bound of the marginal model posterior probability separated for each component of the latent variable model; an optimality determination unit which determines whether or not to continue the maximization of the lower bound of the marginal model posterior probability; and a result output unit which outputs a result.