Commonsense Contextualizing Model for NLP Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing natural language models lack common sense, leading to failures on tasks considered trivial to humans, as they struggle to identify appropriate context based on commonsense knowledge.
Innovation Solution
A commonsense contextualizing model is generated to infer or identify commonsense context for sentences, outputting commonsense context in the form of a commonsense path that includes a sequence of entities connected via relations, determined based on a knowledge graph schema.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional natural language models are used, then the models can perform basic natural language processing tasks, but they lack common sense and fail on tasks considered trivial to humans
Solution Approach 1:
The patent introduces a commonsense knowledge graph as an intermediary component between the natural language model and the input text. This knowledge graph stores structured commonsense knowledge and relations, allowing the model to query and utilize commonsense context during inference without requiring extensive retraining. The intermediary enables the model to access external commonsense knowledge dynamically, resolving the contradiction between maintaining basic NLP capabilities and acquiring commonsense understanding.
Solution Approach 2:
The patent performs preliminary action by pre-processing the input text to identify entities and relations, then querying the commonsense knowledge graph in advance to retrieve relevant commonsense context before performing the main natural language processing task. This preliminary retrieval of commonsense knowledge allows the model to incorporate contextual information early in the processing pipeline, improving reliability on commonsense tasks without compromising versatility.
2Reliability
If natural language models are trained with more data to improve accuracy, then performance on trivial tasks improves, but computing resources and training time increase significantly
Solution Approach 1:
The patent extracts commonsense knowledge from large corpora and structures it into a dedicated knowledge graph, separating this knowledge from the main model training process. By extracting and organizing commonsense knowledge independently, the system avoids the need to train the entire model on extensive commonsense data, thereby reducing computing resources and training time while maintaining accuracy on natural language processing tasks.
Solution Approach 2:
The patent performs preliminary action by pre-processing and structuring commonsense knowledge into a queryable graph format before the actual NLP task execution. This preliminary organization of knowledge allows for efficient retrieval during inference without requiring repeated expensive training computations, thus improving reliability while minimizing the use of computing resources during operation.
Data Source
AI summary
Methods and systems are provided for facilitating generation and utilization of a commonsense contextualizing machine learning (ML) model, in accordance with embodiments described herein. In embodiments, a commonsense contextual ML model is trained by fine-tuning a pre-trained language model using a set of training path-sentence pairs. Each training path-sentence pair includes a commonsense path, identified via a commonsense knowledge graph, and a natural language sentence identified as contextually related to the commonsense path. The trained commonsense contextualizing ML model can then be used to generate a commonsense inference path for a text input. Such a commonsense inference path can include a sequence of entities and relations that provide commonsense context to the text input. Thereafter, the commonsense inference path can be provided to a natural language processing system for use in performing a natural language processing task.