Similarity-Based Token Classification for Unstructured Data Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern systems face inaccuracies in decision-making and services due to the presence of extraneous data in unstructured data sets, which are not effectively distinguished from relevant data, leading to compromised quality.
Innovation Solution
Utilizing natural language processing (NLP) to tokenize unstructured data, determine similarity metrics like Levenshtein's distance, and classify tokens as 'similar' or 'dissimilar' to an identifier, extracting only relevant data for accurate decision-making and service provision.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If unstructured data is used directly for decision-making, then data processing is simple and fast, but decision accuracy deteriorates due to extraneous data
Solution Approach 1:
The patent applies segmentation by dividing unstructured data into discrete tokens and further segmenting the extraction process into multiple stages: tokenization, similarity metric calculation, threshold comparison, and classification. This systematic breakdown transforms the complex task of relevant data extraction into manageable sequential operations, resolving the contradiction between decision accuracy and processing complexity
Solution Approach 2:
The patent introduces an intermediary classification system that acts as a mediator between raw unstructured data and decision-making processes. By implementing a threshold-based classification mechanism that compares similarity metrics against predefined thresholds, the system filters extraneous data before it reaches decision-making, thereby improving accuracy without requiring direct complex processing of all unstructured data
2Reliability
If all data from a preferred data source is used, then data acquisition cost is low, but data quality deteriorates due to extraneous data
Solution Approach 1:
The patent applies the extraction principle by selectively removing extraneous data from the preferred data source while retaining relevant data. Through tokenization and similarity metric comparison, the system extracts only tokens that meet the relevance threshold, thereby maintaining data quality without requiring reduction of overall data volume from the preferred source
Solution Approach 2:
The patent applies local quality by applying different quality standards to different portions of the data. Instead of uniformly treating all data, the system evaluates each token individually against relevance criteria, assigning different quality classifications (relevant vs. extraneous) to different data portions based on their specific characteristics and similarity metrics
3Measurement precision
If relevant data is extracted using NLP, then decision accuracy is improved, but processing time increases
Solution Approach 1:
The patent applies preliminary action by performing tokenization and establishing similarity metrics before the actual classification and extraction process. By pre-processing the data into tokens and pre-defining relevance thresholds, the system prepares the data structure in advance, which accelerates the subsequent extraction process and reduces overall processing time while maintaining high data relevance accuracy
Data Source
AI summary
Systems, apparatuses, methods, and computer program products are disclosed for obtaining relevant data from an unstructured data source. An example method includes extracting relevant data that is intermixed with extraneous data using natural language processing. In order to do so, text from the unstructured data source may be tokenized and each token may be compared to an identifier associated with the relevant data. A similarity metric may be determined between each token and the identifier in order to classify tokens as similar or dissimilar to the identifier. All tokens classified as similar to the identifier may be aggregated in order to obtain relevant data.