Unstructured Data Classification Using Token Similarity Filtering

Solution Overview

Problem

Modern systems face inaccuracies in decision-making and services due to the inclusion of extraneous data in unstructured data sets, which are often obtained from preferred data sources due to low computational and security risks.

Innovation Solution

Implementing natural language processing (NLP) to tokenize unstructured data, determine similarity metrics like Levenshtein's distance, and classify tokens as 'similar' or 'dissimilar' to an identifier, extracting only relevant data for accurate decision-making and service provision.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of manufacture

If unstructured data is obtained from preferred data sources, then computational cost and security risk are reduced, but data accuracy deteriorates due to extraneous data

Engineering Contradiction:

Improvecomputational costVSAvoiddata accuracy

Core Design Contradiction:

Ease of manufactureVSMeasurement precision

Solution Approach 1:

The patent extracts relevant data from unstructured data by tokenizing the data, comparing tokens against identifiers using similarity metrics, and isolating only the relevant portions. This extraction process removes extraneous data while preserving relevant information, thereby improving data accuracy without requiring changes to the data source selection strategy

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent introduces an intermediary processing layer that includes tokenization, similarity metric calculation, and threshold-based filtering. This intermediary process acts as a mediator between the preferred data sources and the decision-making system, cleaning the data while maintaining the benefits of using preferred sources

Inventive Principle:

Principle #24Intermediary (Mediator)

2Object-affected harmful factors

If unstructured data is obtained from preferred data sources, then security risk is reduced, but data accuracy deteriorates due to extraneous data

Engineering Contradiction:

Improvesecurity riskVSAvoiddata accuracy

Core Design Contradiction:

Object-affected harmful factorsVSMeasurement precision

Solution Approach 1:

The patent extracts relevant data from unstructured data by tokenizing the data, comparing tokens against identifiers using similarity metrics, and isolating only the relevant portions. This extraction process removes extraneous data while preserving relevant information, thereby improving data accuracy without requiring changes to the data source selection strategy

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent introduces an intermediary processing layer that includes tokenization, similarity metric calculation, and threshold-based filtering. This intermediary process acts as a mediator between the preferred data sources and the decision-making system, cleaning the data while maintaining the benefits of using preferred sources

Inventive Principle:

Principle #24Intermediary (Mediator)

3Measurement precision

If natural language processing is implemented to extract relevant data, then data accuracy is improved, but device complexity increases

Engineering Contradiction:

Improvedata accuracyVSAvoidsystem complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The patent segments the NLP process into distinct, manageable components: tokenization of unstructured data, calculation of similarity metrics between tokens and identifiers, application of threshold comparisons, and extraction of relevant data. This segmentation reduces system complexity by making each component independent and well-defined

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent replaces complex semantic analysis with a simpler mechanical process based on string similarity metrics (such as Levenshtein distance) and threshold comparisons. This substitution maintains data extraction accuracy while significantly reducing the computational complexity and resource requirements of the system

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Data Source

PatentUS20260064989A1System and methods for classification of unstructured data using similarity metrics

Publication Date: 2026.03.05 WELLS FARGO BANK NA

AI summary

Systems, apparatuses, methods, and computer program products are disclosed for obtaining relevant data from an unstructured data source. An example method includes extracting relevant data that is intermixed with extraneous data using natural language processing. In order to do so, text from the unstructured data source may be tokenized and each token may be compared to an identifier associated with the relevant data. A similarity metric may be determined between each token and the identifier in order to classify tokens as similar or dissimilar to the identifier. All tokens classified as similar to the identifier may be aggregated in order to obtain relevant data.