3D Tensor BERT Entity Relation Processing with Selective Masking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The introduction of extended information into the input data of BERT models affects the accuracy of processing results, and existing strategies fail to effectively reduce the adverse impact of this extended information on the original input data.
Innovation Solution
A natural language processing method that determines extended information for entities in a target sentence, adds it after the entity location, and tunes the attention score between extended information and other entities to zero, using a knowledge graph and relation recognition models to select relevant information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If extended information is added to BERT model input data, then the model can access more contextual knowledge, but the accuracy of processing results deteriorates due to adverse impact from irrelevant extended information
Solution Approach 1:
The patent extracts and removes irrelevant extended information from the BERT model input by using a mask mechanism. The mask identifies and excludes extended information tokens that do not have semantic relationships with the original input, thereby preventing them from adversely affecting processing accuracy while retaining useful contextual knowledge.
Solution Approach 2:
The patent applies different attention mechanisms to different parts of the input data. Original input tokens receive full attention for accurate processing, while extended information tokens are selectively masked based on their relevance to the original input. This local differentiation ensures that useful extended information is utilized while irrelevant information is excluded.
2Measurement precision
If extended information is selected using various strategies, then the accuracy of selecting extended information is improved, but the adverse impact on original input data cannot be effectively reduced
Solution Approach 1:
The patent employs a feedback mechanism where the model first processes the original input, then identifies relevant extended information based on semantic relationships, and finally applies masking to exclude irrelevant extended information. This iterative feedback process ensures that only truly relevant extended information influences the final processing results, thereby maintaining reliability.
Solution Approach 2:
The patent performs preliminary identification and masking of irrelevant extended information before the final processing step. By pre-filtering the extended information based on semantic relationships with the original input, the system prevents irrelevant information from interfering with the processing, thus ensuring result reliability.
3Loss of information
If all extended information is included in BERT input, then comprehensive knowledge is provided to the model, but processing efficiency deteriorates due to increased computational complexity
Solution Approach 1:
The patent extracts and removes irrelevant extended information tokens from the BERT model input sequence. By identifying and excluding tokens that do not have semantic relationships with the original input, the system reduces the total number of tokens processed, thereby improving processing efficiency while retaining comprehensive useful knowledge.
Solution Approach 2:
The patent applies partial masking to extended information tokens based on their relevance. Instead of masking all extended information or none, the system selectively masks only the irrelevant portion, achieving a balance between knowledge completeness and processing efficiency by processing only the necessary subset of extended information.
Data Source
AI summary
A natural language processing method and apparatus, a device, and a readable storage medium, where the method includes: obtaining a target sentence to be processed, and determining each entity in the target sentence (S101); for each entity in the target sentence, in response to the entity being present in a preset entity set, determining extended information for the entity, and adding the determined extended information after a location of the entity in the target sentence, to obtain an updated target sentence (S102); and inputting the updated target sentence to a bidirectional encoder representations from transformer (BERT) model, such that the BERT model performs a natural language processing task, where in a process in which the BERT model performs the natural language processing task, an attention score between extended information of any entity in the target sentence and another entity in the target sentence is adjusted to zero (S103).