Aggregating Named Entity Recognition Results via Weighted Voting

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing named entity recognition (NER) services struggle to consistently identify entities across different types of documents due to variations in entity type classification and performance characteristics, leading to limited applicability in real-world scenarios when combining extraction results from diverse NER services.

Innovation Solution

An aggregation service that considers differences in entity type schemas and performance characteristics across multiple NER services, transforming and ranking annotations to produce a standardized output, using techniques like extended majority vote, weighted majority vote, simple error path analysis, or pairwise error path analysis to improve the accuracy and reliability of extracted information.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Quantity of substance

If extraction results from multiple diverse NER services are combined, then the quantity of extracted information increases, but the reliability and accuracy of the extracted information deteriorates due to variations in entity type classification and performance characteristics

Engineering Contradiction:

Improvequantity of extracted informationVSAvoidreliability of extracted information

Core Design Contradiction:

Quantity of substanceVSReliability

Solution Approach 1:

The aggregation service dynamically adjusts weighting parameters for each NER service based on their performance characteristics and entity type classification accuracy. By changing these parameters adaptively, the system optimizes the combination of results to maintain high reliability while incorporating data from multiple services

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The aggregation service acts as an intermediary layer between multiple NER services and the final application. It standardizes entity type classifications from different services, resolves conflicts through weighted voting mechanisms, and produces a unified reliable output that combines information from diverse sources

Inventive Principle:

Principle #24Intermediary (Mediator)

2Adaptability or versatility

If NER services with different entity type schemas are integrated, then the versatility and applicability of the system improves, but the complexity of aggregating and standardizing entity type classifications increases

Engineering Contradiction:

Improveversatility of NER systemVSAvoidcomplexity of aggregation process

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The aggregation service implements a universal entity type schema that can accommodate and standardize classifications from multiple diverse NER services. This universal framework enables the system to handle various entity types and classification schemes while maintaining a consistent output format

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

The aggregation service serves as a mediating layer that translates and harmonizes different entity type schemas from various NER services into a standardized format. This intermediary function manages the complexity of schema alignment while preserving the versatility of supporting multiple service types

Inventive Principle:

Principle #24Intermediary (Mediator)

3Measurement precision

If performance characteristics are used to weight NER services in aggregation, then the accuracy of extracted information improves, but the complexity of evaluating and applying performance metrics increases

Engineering Contradiction:

Improveaccuracy of extracted informationVSAvoidcomplexity of performance evaluation

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The system performs preliminary evaluation of NER service performance characteristics before aggregation. By pre-assessing accuracy metrics, entity type classification quality, and reliability scores, the system establishes weighting factors in advance that simplify the aggregation process while maintaining high accuracy

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The aggregation service dynamically adjusts weighting parameters based on evaluated performance characteristics of each NER service. These parameter changes enable the system to prioritize more accurate services for specific entity types while managing the complexity through automated parameter optimization

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS10762117B2Aggregating results from named entity recognition services

Publication Date: 2020.09.01 SAP SE

AI summary

An aggregation service aggregates extraction results from diverse named entity recognition (“NER”) services, which can help improve the quality of extracted information. In some cases, the aggregation service considers differences in entity type classifications when aggregating extraction results from different NER services. The aggregation service can also consider performance characteristics (e.g., error rates) for the different NER services. For example, the aggregation service receives extraction results generated for a document corpus according to an entity type schema for each of multiple different NER services, then aggregates the extraction results based at least in part on relations between entity types for the NER services. For a given annotation area, the computing system can identify hypotheses and rank the hypotheses according to an aggregation approach. For some types of aggregation approach, the computing system uses weight values, error path values and/or other performance characteristics determined during training for the NER services.