Text Categorization via Vectorization and Deep Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Computer algorithms face challenges in accurately categorizing sequences of text from documents, especially when dealing with extraneous information, errors, or misspellings, which can lead to inaccurate analyses in downstream systems.
Innovation Solution
A categorization system that utilizes natural language processing with machine learning to identify and categorize sequences of text. This system includes a text analysis process to remove unique and generic data, a vectorization process to address misspellings, and a deep learning model for accurate categorization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If computer algorithms analyze document data including extraneous information and errors, then the analysis can be performed on complete data, but the categorization accuracy deteriorates due to misspellings and irrelevant content
Solution Approach 1:
The patent extracts and removes extraneous information from document data before analysis. The system identifies and eliminates irrelevant content, misspellings, and errors through preprocessing steps, keeping only the meaningful text for categorization. This extraction principle directly addresses the contradiction by separating useful information from harmful extraneous data, thereby improving categorization accuracy without losing essential content.
Solution Approach 2:
The patent applies preliminary cleaning and normalization actions to the document data before the main categorization process. By pre-processing the text to correct misspellings, remove irrelevant content, and standardize formatting beforehand, the system ensures that the subsequent analysis operates on high-quality data. This preliminary action prevents errors from propagating through the analysis pipeline, resolving the accuracy issue caused by extraneous information.
2Measurement precision
If text preprocessing removes unique strings and generic data, then categorization accuracy improves, but information loss occurs from removing potentially relevant content
Solution Approach 1:
The patent applies different treatment rules to different parts of the text based on their local characteristics. Rather than uniformly removing all unique strings or generic data, the system selectively processes text segments based on their context, position, and importance. For example, unique identifiers in specific fields may be preserved while generic headings are removed. This localized approach maintains categorization precision by targeting only the problematic elements while preserving potentially relevant information.
Solution Approach 2:
The patent changes the parameters of text representation rather than simply deleting data. Instead of removing unique strings and generic data outright, the system transforms them into standardized forms or aggregates them into category indicators. This parameter change allows the system to eliminate the harmful effects of extraneous information while retaining the essential informational content in a transformed state, thus improving precision without significant information loss.
3Adaptability or versatility
If the system processes multiple languages, then versatility improves, but system complexity increases due to language-agnostic processing requirements
Solution Approach 1:
The patent implements a universal text processing framework that handles multiple languages through a single system architecture. The preprocessing, cleaning, and categorization mechanisms are designed to be language-agnostic, working effectively with text from any language without requiring separate processing pipelines. This universality achieves multi-language support while avoiding the complexity of maintaining separate systems for each language, as the core algorithms naturally adapt to different linguistic structures.
Data Source
AI summary
Disclosed herein are system, method, and computer program product embodiments for categorizing sequences of text extracted from documents using natural language processing. In some embodiments, a categorization system may receive a first document file in a machine readable format. The categorization system may analyze a sequence of text from the first document file and identify a numeric text string in the sequence. The categorization system may also identify text data in the sequence matching text data from a second document file. The categorization system may remove the numeric text string and the matching data from the sequence to generate a trimmed version of the sequence. The categorization system may then apply a vectorization model to the trimmed version of the sequence as well as a trained deep learning model to the vector version to identify a corresponding category for the sequence of text.