KB-QA System Using BERT Embeddings for Compositional Semantics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing knowledge base question answering (KB-QA) systems face limitations in capturing global compositional semantics and require large training datasets, which are often not available in practice.
Innovation Solution
The use of pre-trained language models, such as BERT, in an IR-based KB-QA framework, which encodes questions and candidate answer contexts to capture contextual representations and eliminate the need for recurrent neural network architectures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If semantic parsing based systems are used to convert natural language questions into structured expressions, then the system can query the knowledge base to obtain answers, but the system requires large training datasets and complex recurrent neural network architectures
Solution Approach 1:
The patent replaces the mechanical recurrent neural network architecture with a pre-trained language model (BERT) that uses transformer-based attention mechanisms. This substitution eliminates the need for complex RNN structures while maintaining or improving question answering accuracy through pre-trained contextual representations.
Solution Approach 2:
The patent applies pre-trained language models that have already been trained on large corpora before being deployed for knowledge base question answering. This preliminary training action provides the model with pre-existing linguistic knowledge and contextual understanding, eliminating the need for additional large training datasets specific to KB-QA tasks.
2Loss of information
If semantic parsing based systems are used to construct structured expressions from natural language, then the system can achieve better compositional semantics, but the system requires supervised training data that is often not available in practice
Solution Approach 1:
The patent utilizes pre-trained language models that have already acquired comprehensive linguistic knowledge and compositional semantics understanding from large corpora during pre-training. This preliminary action transfers general language understanding capabilities to the KB-QA task without requiring task-specific training data.
Solution Approach 2:
The patent employs a universal pre-trained language model that can handle multiple NLP tasks including question answering, semantic parsing, and knowledge base querying. This multi-functional approach allows the system to capture compositional semantics without needing separate training datasets for each function.
3Adaptability or versatility
If information retrieval based systems are used for KB-QA, then the system is more flexible and requires less supervision, but the system may struggle with capturing global compositional semantics
Solution Approach 1:
The patent replaces traditional information retrieval methods with a pre-trained transformer-based language model that inherently understands compositional semantics through attention mechanisms. This substitution maintains the flexibility of IR-based systems while improving semantic understanding capabilities.
Solution Approach 2:
The patent changes the fundamental parameter of how semantic understanding is achieved by using pre-trained contextual embeddings instead of traditional bag-of-words or shallow semantic representations. This parameter change enables the system to capture global compositional semantics while maintaining flexibility and requiring minimal supervision.
Data Source
AI summary
Aspects of the subject disclosure may include, for example, identifying an entity of a natural language question, locating a node of a knowledge graph corresponding to the entity, and generating a candidate answer set including a group of other entities located a predetermined proximity to the node. Contextual information for the group of other entities is determined from the knowledge graph, and the natural language question and contextual information are separately encoded to obtain separate encoded vectorial representations of the natural language question and members of the candidate answer set. The encoding uses pre-trained language model embeddings obtained via a bidirectional encoder representations from transformer encoding process. The encoded vectorial representations of the question under an influence of aspects of the contextual information are scored and a member of the candidate answer set selected according to the score to obtain an answer to the original question. Other embodiments are disclosed.