Hierarchical Transformer for Knowledge Graph Entity Embeddings

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current knowledge graph embedding methods primarily focus on local connectivity patterns, neglecting the structural formation in knowledge graphs, which hinders effective learning of entity representations for applications like content recommendation systems.

Innovation Solution

A hierarchical Transformer model is employed to learn entity embeddings, utilizing two Transformer blocks to generate relation-dependent embeddings for source entities and their neighbors, combined with a masked entity model to balance contextual and source entity information, enhancing link prediction accuracy.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If conventional knowledge graph embedding methods are used, then local connectivity patterns are captured, but structural formation in the graph context is ignored

Engineering Contradiction:

Improveentity representation accuracyVSAvoidstructural information

Core Design Contradiction:

Measurement precisionVSLoss of information

Solution Approach 1:

The patent transitions from local connectivity analysis to global structural analysis by introducing graph pooling operations that aggregate node features across multiple hops. This dimensional expansion allows the model to capture structural formation at different scales, from immediate neighbors to distant contextual entities, thereby resolving the contradiction between local pattern recognition and global structural understanding.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

Solution Approach 2:

The hierarchical graph pooling mechanism implements nested aggregation where nodes are pooled into super-nodes, which are then pooled into even higher-level representations. This nested structure enables the model to retain detailed local information while simultaneously capturing global structural patterns, effectively nesting multiple levels of abstraction to preserve both local connectivity and overall graph formation.

Inventive Principle:

Principle #7Nested doll (Nesting)

2Reliability

If hierarchical Transformer model is used, then structural information is captured, but model complexity increases

Engineering Contradiction:

Improvelink prediction accuracyVSAvoidmodel architecture

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The patent segments the graph processing task into distinct hierarchical levels with specialized pooling operations at each level. By dividing the complex graph transformation into manageable stages (node-level features, super-node aggregation, global structure formation), the model achieves high link prediction accuracy while maintaining architectural clarity and computational tractability through modular design.

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS20230267308A1Learning graph representations using hierarchical transformers for content recommendation

Publication Date: 2023.08.24 MICROSOFT TECHNOLOGY LICENSING LLC

AI summary

Knowledge graphs can greatly improve the quality of content recommendation systems. There is a broad variety of knowledge graphs in the domain including clicked user-ad graphs, clicked query-ad graphs, keyword-display URL graphs etc. A hierarchical Transformer model learns entity embeddings in knowledge graphs. The model consists of two different Transformer blocks where the bottom block generates relation-dependent embeddings for the source entity and its neighbors, and the top block aggregates the outputs from the bottom block to produce the target entity embedding. To balance the information from contextual entities and the source entity itself, a masked entity model (MEM) task is combined with a link prediction task in model training.