Token Stratification for Domain-Adaptable Question Answering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current question answering systems face difficulties in determining the relevance of answers based on token relevance, as they struggle to differentiate between similar passages and accurately identify the most important information, leading to incorrect answers.
Innovation Solution
A method that selects and classifies nouns from questions, trains an AI system to associate relevancy with identified passages based on noun types, and uses a deep neural network to provide answers, adapting to new domains by learning the importance of token types through labeled question-answer pairs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If the system uses all tokens from questions to determine answer relevancy, then more information is considered, but the system cannot accurately differentiate between similar passages and identifies incorrect answers
Solution Approach 1:
The patent segments tokens from questions into distinct types (noun types) and selectively applies them to passage analysis. Instead of treating all tokens uniformly, the system categorizes them and determines which noun types are most predictive for answer relevancy, thereby improving precision while maintaining information completeness.
Solution Approach 2:
The patent applies different levels of importance to different noun types based on their predictive value for answer relevancy. By assigning varying weights or selection criteria to different noun types rather than treating all tokens equally, the system enhances the accuracy of relevancy determination while preserving essential information.
2Device complexity
If the system treats all tokens equally in passage analysis, then the process is simpler, but the system struggles to identify the most important information
Solution Approach 1:
The patent divides tokens into different noun types and selectively processes based on their predictive value. This segmentation allows the system to focus computational resources on the most important noun types rather than processing all tokens equally, thereby improving information identification accuracy while managing complexity through structured categorization.
Solution Approach 2:
The patent changes the parameter of token importance by determining which noun types are most predictive for answer relevancy. This parameter change allows the system to dynamically adjust which tokens receive full analytical attention, improving precision without requiring equally complex processing for all tokens.
3Adaptability or versatility
If the system uses a generic question answering approach, then it works across domains, but it cannot adapt to new domains or accurately determine relevancy
Solution Approach 1:
The patent performs preliminary classification of nouns into types before passage analysis. By pre-categorizing tokens and determining which noun types are most predictive for the specific domain, the system prepares the analysis framework in advance, enabling both domain adaptability and reliable answer determination through structured, domain-informed processing.
Data Source
AI summary
A method determines a relevancy of answers to questions based on token relevance in a system capable of answering questions. One or more processors receive a question that is composed of a set of tokens T (T1, T2, . . . , Tn). The processor(s) select tokens T′ (T′1, T′2, . . . , T′m) from the tokens T (T1, T2, . . . , Tn), where each T′j from T′ is a noun, and classify each T′j as a noun type. The processor(s) scan a corpus to identify passages with candidate answers to the question, and analyze the identified passages utilizing noun entries in the passages classified as the noun type. The processor(s) train an artificial intelligence (AI) system to associate a relevancy to the question for the identified passages based on noun types, and then utilize the trained AI system to provide an answer to the question based on an output of the trained AI system.