Variational Bayesian Network for Cold Entity Representation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing representation learning models struggle to accurately represent rare or 'cold' entities due to limited co-occurrence data, leading to poor performance in natural language understanding and other applications.

Innovation Solution

The proposed method utilizes a Variational Bayesian Network (VBN) that jointly models implicit co-occurrence relations, explicit relations, and hierarchical relations to learn entity representations, thereby improving the representation of cold entities.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If the training dataset size is increased to mitigate the cold entity problem, then the representation of rare entities improves, but the model complexity and data processing requirements increase

Engineering Contradiction:

Improverepresentation accuracy of rare entitiesVSAvoidmodel complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The patent introduces side information (explicit relations, hierarchical structures, knowledge graphs) as an intermediary to bridge the gap between limited co-occurrence data and accurate entity representation. This mediator provides additional structural constraints and semantic relationships that compensate for insufficient training data, enabling better representation of rare entities without proportionally increasing model complexity

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent changes the parameters of the learning problem by incorporating multiple types of information (co-occurrence data, explicit relations, hierarchical structures) rather than relying solely on increasing data volume. This parameter change allows the model to leverage structured knowledge and relational constraints to improve representation accuracy without linearly scaling data requirements

Inventive Principle:

Principle #35Parameter changes

2Device complexity

If only co-occurrence data is used for training, then the model remains simple, but rare entities are poorly represented

Engineering Contradiction:

Improvemodel simplicityVSAvoidrepresentation accuracy of rare entities

Core Design Contradiction:

Device complexityVSReliability

Solution Approach 1:

The patent merges multiple data sources and information types (co-occurrence relations, explicit relations, hierarchical structures) into a unified representation learning framework. This combination allows the model to leverage both the simplicity of co-occurrence-based learning and the rich semantic information from structured knowledge, improving representation accuracy while maintaining reasonable model complexity through integrated processing

Inventive Principle:

Principle #5Merging (Combining)

Data Source

PatentUS12223274B2Representation learning with side information

Publication Date: 2025.02.11 MICROSOFT TECHNOLOGY LICENSING LLC

AI summary

A relational similarity determination engine receives as input a dataset including a set of entities and co-occurrence data that defines co-occurrence relations for pairs of the entities. The relational similarity determination engine also receives as input side information defining explicit relations between the entities. The relational similarity determination engine jointly models the co-occurrence relations and the explicit relations for the entities to compute a similarity metric for each different pair of entities within the dataset. Based on the computed similarity metrics, the relational similarity determination engine identifies a most similar replacement entity from the dataset for each of the entities within the dataset. For a select entity received as an input, the relational similarity determination engine outputs the identified most similar replacement entity.