Entity Detection in Unstructured Data via Probabilistic Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies face challenges in detecting entities within unstructured data, particularly when entities are referred to without explicit identifiers, and distinguishing between names and numbers, which becomes increasingly difficult as the amount of data grows, leading to confusion and inaccurate identification.
Innovation Solution
A probabilistic approach using a name probability source, a known entity database, and historical context information to calculate entity scores for terms in unstructured data, enabling the identification of known and unknown entities by weighing probabilities from these sources to determine if terms refer to entities of interest.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional entity detection methods are used, then known entities with explicit identifiers can be detected, but entities referred to without explicit identifiers cannot be detected
Solution Approach 1:
The patent replaces traditional mechanical matching systems with a probabilistic computational model. Instead of relying on explicit identifier matching, the system uses name probability sources and historical context to calculate entity scores for terms in unstructured data, enabling detection of both known and unknown entities through statistical inference rather than direct matching
Solution Approach 2:
The patent introduces an intermediary probabilistic scoring mechanism between the raw unstructured data and entity identification. The name probability source and historical context information serve as intermediaries that transform vague term references into structured entity detections, bridging the gap between unstructured data and reliable entity recognition
2Quantity of substance
If more data is processed to improve detection coverage, then more entities can be identified, but the difficulty of distinguishing between names and numbers increases
Solution Approach 1:
The patent changes the detection parameter from binary classification (name vs. number) to continuous probabilistic scoring. By calculating entity scores based on name probability sources and historical context, the system can distinguish between names and numbers on a spectrum rather than through rigid categories, making the distinction easier even as data volume increases
Solution Approach 2:
The patent incorporates historical context information as feedback that continuously refines the entity detection process. The system uses previously identified entities and their contextual relationships to adjust and improve the detection of current terms, making the distinction between names and numbers more accurate as more data is processed
3Adaptability or versatility
If probabilistic scoring is used to improve entity identification, then unknown entities can be detected, but computational complexity increases
Solution Approach 1:
The patent segments the complex entity detection problem into separate manageable components: name probability source processing, historical context analysis, and entity score calculation. Each component operates independently on specific aspects of the data, reducing overall computational complexity while maintaining the ability to detect unknown entities through the integration of these segmented processes
Data Source
AI summary
Examples herein involve detection of entities in unstructured data. Terms are extracted from unstructured data. Entities scores for the terms are calculated using information from a name probability source, a known entity database, and historical context information. The entity scores indicate a probability that the respective terms refer to entities. The presence of detected entities are indicated based on the entity scores.