Generative Probabilistic Model for Feature Clustering

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional techniques for discovering object categories in data lack accuracy and performance, particularly in unsupervised learning scenarios where semantic structures in image and video data are not effectively identified.

Innovation Solution

The use of a Deep Learning model with a Stochastic Gradient Descent based learning scheme to learn parameters of a generative probabilistic model, which maximizes log-likelihood and represents distributions through weights of a feed-forward network, enabling unsupervised clustering and categorization of features without supervision.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If conventional clustering techniques (k-means, agglomerative clustering) are used with cross-entropy loss, then the implementation is straightforward, but the accuracy and performance of feature discovery is insufficient

Engineering Contradiction:

Improveaccuracy of feature discoveryVSAvoidcomplexity of model architecture

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The patent changes the loss function from cross-entropy to negative log-likelihood, and transforms cluster centers from simple vectors to full covariance matrices. This parameter change enables the model to capture data distribution more accurately through Gaussian probability densities, resolving the contradiction by improving measurement precision through mathematical transformation rather than architectural complexity

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The patent replaces the mechanical iterative clustering process (k-means updates) with a probabilistic generative model that uses Stochastic Gradient Descent. This substitution transforms the optimization problem from iterative reassignment to gradient-based parameter learning, achieving better accuracy through a different computational paradigm while maintaining reasonable complexity

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Adaptability or versatility

If center-based clustering is implemented as part of the network, then the clustering process is integrated, but the ability to capture complex data distributions is limited

Engineering Contradiction:

Improveability to capture data distributionsVSAvoidnumber of parameters to learn

Core Design Contradiction:

Adaptability or versatilityVSQuantity of substance

Solution Approach 1:

The patent segments the covariance matrix into diagonal elements only, learning independent variance for each feature dimension rather than full covariance. This segmentation reduces the number of parameters from O(d²) to O(d) while maintaining the ability to capture feature-specific distributions, resolving the contradiction between adaptability and parameter quantity

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Instead of learning cluster assignments from fixed centers, the patent inverts the approach by learning Gaussian distributions (mean and variance) that naturally capture data variability. This inversion allows the model to adapt to complex distributions through probabilistic modeling rather than deterministic center-based assignment

Inventive Principle:

Principle #13The other way round (Inversion)

3Measurement precision

If supervised learning is used for category classification, then classification accuracy is high, but unsupervised discovery of semantic structures is lost

Engineering Contradiction:

Improveclassification accuracyVSAvoidunsupervised learning capability

Core Design Contradiction:

Measurement precisionVSExtent of automation

Solution Approach 1:

The patent creates a universal model that performs both unsupervised clustering and supervised classification through the same Gaussian mixture framework. The model can operate in unsupervised mode to discover semantic structures or in supervised mode with class labels provided, achieving multi-functionality that resolves the contradiction between classification accuracy and unsupervised capability

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

The patent introduces Gaussian mixture models as an intermediary layer between raw features and final classifications. This intermediary performs unsupervised distribution modeling that can be combined with supervised classification objectives, enabling both unsupervised discovery and supervised accuracy through a mediating probabilistic framework

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS11475313B2Unsupervised, semi-supervised, and supervised learning using deep learning based probabilistic generative models

Publication Date: 2022.10.18 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

Embodiments of the present systems and methods may provide techniques to discover features such as object categories that provide improved accuracy and performance. For example, in an embodiment, a method may comprise extracting, at the computer system, features from a dataset comprising a plurality of data samples using a backbone neural network to form a features vector for each data sample, training, at the computer system, using the features vectors for at least some of the plurality of data samples, an unsupervised generative probabilistic model to perform clustering of extracted features of the at least some of the plurality of data samples by minimizing a negative Log-Likelihood function, wherein clusters of extracted features form categories, and categorizing, at the computer system, at least some different data samples of the plurality of data samples, into the formed categories.