Two-Tier Named Entity Recognition Engine Architecture

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current named entity recognition technologies face challenges such as low accuracy, high false positive and false negative rates, slow processing speed, and language dependency, making them inadequate for identifying and extracting pre-defined entities from texts in information leakage prevention and detection systems.

Innovation Solution

A two-tier architecture for named entity recognition, utilizing advanced regular expression techniques in the first tier and entity fingerprinting in the second tier, to efficiently identify and validate entities by marking entities with regular expressions and using hash-based matching for exact data matches.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If conventional named entity recognition technologies use linguistic grammar-based techniques and statistical models, then entity extraction capability is provided, but accuracy is poor with false positive rate more than 15% and high false negative rate

Engineering Contradiction:

Improveentity extraction accuracyVSAvoidfalse positive and false negative rate

Core Design Contradiction:

Measurement precisionVSReliability

Solution Approach 1:

The system divides entity extraction into two distinct tiers: first tier uses regular expression-based extraction for structured data patterns, second tier uses fingerprinting-based extraction for unstructured data patterns. This segmentation allows each tier to specialize in specific data types, improving overall accuracy while reducing false positives and false negatives compared to conventional single-approach systems.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces an intermediary component that bridges regular expression matching and fingerprinting algorithms. This intermediary manages the coordination between the two extraction methods, allowing the system to leverage the strengths of both approaches while mitigating their individual weaknesses, thereby improving entity extraction reliability.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Productivity

If conventional named entity recognition technologies process natural language text, then entity identification is performed, but processing speed is slow

Engineering Contradiction:

Improveprocessing speedVSAvoidtext processing time

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The system segments the text processing workload by applying regular expression-based extraction first for structured data patterns, which are more common and can be quickly identified. Then fingerprinting-based extraction is applied only to remaining unstructured patterns. This segmentation reduces overall processing time compared to applying complex natural language processing to all text.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system employs self-service mechanisms where regular expressions automatically identify and extract structured entities without requiring complex natural language processing. This self-service approach for structured data frees up processing resources and reduces time consumption for the more complex unstructured entity extraction tasks.

Inventive Principle:

Principle #25Self-service

3Adaptability or versatility

If conventional named entity recognition technologies are designed based on specific individual writing languages, then language-specific entity extraction is achieved, but flexibility and portability are limited

Engineering Contradiction:

Improvelanguage flexibility and portabilityVSAvoidlanguage-specific design complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent implements a universal extraction framework that can handle multiple languages and data formats. The regular expression engine and fingerprinting algorithm work together to extract entities from various language structures without requiring language-specific customization. This multi-functionality allows the same system to process different languages and data types, greatly improving flexibility and portability.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS8321434B1Two tiered architecture of named entity recognition engine

Publication Date: 2012.11.27 TREND MICRO INC

AI summary

A system (and a method) is disclosed to extract entity values from texts. The system receives, at a first tier entity recognition engine, an input data string having a plurality of entities. The first tier entity recognition engine marks entities of the plurality of entities that are regular expression and transmits the input data stream with the marked entities to a second tier entity recognition engine. The second tier entity recognition engine receives the input data stream and identifies unmarked entities in the input data stream received at the second tier entity recognition engine. The second tier entity recognition engine determines whether the unmarked entities comprise a predetermined data format, and if so, outputs those unmarked entities of the plurality of entities that comprise the predetermined data format.