Sliding Window Document Classification for Unstructured Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional techniques are inefficient in extracting information from unstructured textual documents, such as medical reports, which lack standard structures or fields, making it difficult for computer systems to analyze and classify data effectively, particularly in the medical community where formats vary widely.
Innovation Solution
A machine-learning based system that uses a relevance metric analysis with a sliding-window structure to automatically extract information from documents by training a model to learn associations between document data and classification data, allowing for efficient processing of large volumes of documents.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional language analysis techniques are used to extract information from unstructured documents, then the system can process documents without standard structures, but the extraction efficiency and accuracy deteriorate significantly
Solution Approach 1:
The patent divides the document into multiple window regions using a sliding window approach, processing each region independently to extract information. This segmentation allows the system to handle unstructured documents efficiently by breaking them down into manageable segments that can be processed through the trained classification model.
Solution Approach 2:
The patent replaces conventional language analysis techniques with a machine learning-based classification system. The system uses a trained model that has learned associations between document content and classification categories, enabling accurate information extraction from unstructured documents without relying on traditional parsing methods.
2Device complexity
If conventional techniques are used for document classification, then the system structure remains simple, but the classification accuracy and speed deteriorate when processing large volumes of documents
Solution Approach 1:
The patent employs a pre-trained classification model that has been trained in advance on labeled data. The model learns the associations between document content and classification categories during training, enabling accurate classification of new documents without requiring complex real-time analysis. This preliminary training action significantly improves classification accuracy and speed.
3Adaptability or versatility
If the system processes documents with varied formats and structures, then the adaptability improves, but the processing time and computational resources increase
Solution Approach 1:
The patent applies a sliding window approach that divides documents into overlapping window regions. This segmentation allows the system to process varied document formats efficiently by consistently applying the same classification model to each segment, reducing the need for format-specific processing and thereby decreasing overall processing time.
Data Source
AI summary
A method for extracting information from a dataset, e.g., a document, includes: receiving the dataset at an information handling device, optionally, extracting, via optical character recognition implemented by a processor of the information handling device, textual information associated with the dataset, and classifying the dataset into one of a plurality of classes. Classifying the dataset may include computing a similarity score for each of the plurality of classes for each of a plurality of window regions of the dataset, calculating a subset of highest similarity scores for each of the plurality of classes for each of the plurality of window regions, determining overall similarity scores for each of the plurality of classes, and classifying the dataset as corresponding to a class with a highest overall similarity score.