Inverted Index Querying for Scalable Entity Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current entity extraction techniques, primarily based on machine learning and natural language processing, are resource-intensive and costly when processing large sets of documents, as they require scanning and processing all documents for each reference list, leading to inefficiencies and increased costs.
Innovation Solution
A system that filters documents using a covering set of token sets to reduce the number of queries on an inverted index, balancing query costs and matching efficiency, including a document identifier filter, an inverted index querier, and an entity string matcher to retrieve and filter documents efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If machine learning and natural language processing techniques are used for entity extraction, then entity recognition accuracy is improved, but processing cost and resource consumption increase significantly
Solution Approach 1:
The patent segments the entity extraction process into two distinct phases: a filtering phase that uses simple inverted index queries to identify candidate documents, and a recognition phase that applies expensive ML/NLP techniques only to those candidates. This segmentation allows the system to maintain high accuracy while dramatically reducing overall processing costs by avoiding unnecessary application of resource-intensive techniques to all documents.
Solution Approach 2:
The patent performs preliminary filtering using inverted indexes before applying machine learning techniques. By pre-identifying candidate documents that contain relevant entity strings or tokens, the system prepares a reduced subset of documents that require full ML/NLP processing, thereby reducing total resource consumption while maintaining extraction accuracy.
2Reliability
If all documents are processed for entity extraction, then complete entity coverage is achieved, but processing time and computational resources increase
Solution Approach 1:
The patent applies partial action by processing only a subset of documents through the complete entity extraction pipeline. Using inverted indexes, the system identifies and processes only those documents that are likely to contain target entities, rather than exhaustively processing all documents. This approach maintains reliability for relevant entities while significantly reducing processing time.
3Measurement precision
If traditional entity extraction methods process each document individually, then thorough entity identification is achieved, but scalability to large document sets deteriorates
Solution Approach 1:
The patent merges traditional inverted index technology with modern machine learning approaches. The inverted index component handles bulk document filtering efficiently at scale, while the ML component provides thorough entity identification for candidate documents. This combination enables the system to scale to large document sets while maintaining thorough entity identification through the coordinated operation of both technologies.
Data Source
AI summary
A set of documents is filtered for entity extraction. A list of entity strings is received. A set of token sets that covers the entity strings in the list is determined. An inverted index generated on a first set of documents is queried using the set of token sets to determine a set of document identifiers for a subset of the documents in the first set. A second set of documents identified by the set of document identifiers is retrieved from the first set of documents. The second set of documents is filtered to include one or more documents of the second set that each includes a match with at least one entity string of the list of entity strings. Entity recognition may be performed on the filtered second set of documents.