Document Classification via Late-Binding Schema Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current machine learning models for document classification, such as supervised and unsupervised models, are resource-intensive, rigid, and lack adaptability, requiring large amounts of labeled data and pre-defined classification schemas, which limits their ability to classify documents into a limited set of topics efficiently.
Innovation Solution
An event-based data intake and query system that uses a late-binding schema to extract and store data as events, allowing for flexible schema development and application at search time, enabling field-searchable and field-extractable data without relying on traditional machine learning models for classification.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If machine learning models (supervised or unsupervised) are used for document classification, then classification capability is achieved, but resource consumption and processing time increase significantly
Solution Approach 1:
The patent extracts and stores specific data items (fields) from generated data during pre-processing, creating a curated dataset that can be efficiently queried without requiring full machine learning model processing. This extraction approach separates the classification capability from the resource-intensive ML inference, allowing rapid querying of classified data.
Solution Approach 2:
The system performs classification-related data processing in advance by pre-processing and storing relevant data items in a structured format. This preliminary action creates a ready-to-query classification framework that eliminates the need for real-time ML model execution during data retrieval operations.
2Measurement precision
If supervised machine learning models are used for classification, then accurate classification into predefined topics is achieved, but the system becomes rigid and requires retraining when classification schemas change
Solution Approach 1:
The patent implements a dynamic classification framework where the data receiving system can store and retrieve data according to various classification schemas without requiring retraining. The system allows flexible modification of classification categories and schemas, enabling adaptation to organizational changes while maintaining classification accuracy through the pre-processed structured data.
3Adaptability or versatility
If unsupervised machine learning models are used for classification, then flexibility in handling unclassified data is achieved, but control over classification options is lost
Solution Approach 1:
The patent introduces an intermediary layer between raw data and classification output through the data receiving system that pre-processes and structures data with embedded classification information. This intermediary framework provides both the flexibility to handle diverse data types and the control to enforce specific classification options, as the pre-processed data can be configured with desired classification schemas before storage.
4Adaptability or versatility
If massive quantities of raw data are stored for later analysis, then analysis flexibility is improved, but search and analysis performance deteriorates
Solution Approach 1:
The patent segments massive raw data into structured components with pre-extracted relevant fields and classification information. By organizing data into manageable, pre-processed segments with metadata and classification tags, the system maintains analysis flexibility while enabling efficient search and retrieval operations through the structured organization.
Solution Approach 2:
The system performs preliminary data processing, extraction, and classification during data ingestion rather than during analysis. This preliminary action creates a structured, searchable framework that preserves all raw data while adding organizational layers that dramatically improve search and analysis performance when queries are executed.
Data Source
AI summary
A computer-implemented method is disclosed that includes operations of receiving document to be classified, performing pre-processing operations on the document resulting in generation of a tokenized document, performing word embedding operations on the tokenized document resulting in generation of a vectorized document, performing text similarity operations on the vectorized document and each of one or more vectorized topics resulting in a set of one or more similarity scores, wherein a first similarity score indicates a level of similarity between the vectorized document and a first vectorized topic, and wherein each vectorized topic represents one of a predetermined set of topics and classifying the document into one of the predetermined set of topics based on the set of one or more similarity scores. Performing the word embedding operations includes mapping each token of the remaining subset to a multi-dimensional vector, with each multi-dimensional vector representing a semantic meaning of a token.