Text Preprocessing for Statistical Machine Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current classification techniques for extracting attribute-value pairs from natural language documents are hindered by the presence of extraneous information, which degrades their reliability, and no single technique is universally suited for all types of input data, limiting their effectiveness in applications like product recommendation and demand forecasting.
Innovation Solution
A preprocessing method that segments and clusters text to identify and remove irrelevant information, using filters to preserve formatting and tokenize infrequent words, thereby enhancing the reliability of subsequent statistical machine learning techniques by focusing on relevant data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If classification techniques are applied to text containing extraneous information, then the processing can be performed on raw data, but the reliability of classifications deteriorates
Solution Approach 1:
The patent applies preliminary action by implementing a preprocessing stage that segments text into sentences, clusters them, and filters out extraneous information before the classification process. This preliminary cleaning of the data ensures that the subsequent classification algorithms operate on high-quality, relevant information, thereby maintaining reliability while still processing raw data sources.
Solution Approach 2:
The patent segments the text body into individual sentences and then groups them into clusters based on relevance to the target subject. This segmentation allows the system to distinguish between relevant and extraneous information, enabling reliable classification by processing only the relevant segments while maintaining the ability to handle raw text inputs.
2Device complexity
If no preprocessing is applied, then the processing pipeline remains simple, but extraneous information degrades classification accuracy
Solution Approach 1:
The patent implements preliminary action through a preprocessing module that performs segmentation, clustering, and filtering before the main classification process. This preliminary action removes extraneous information that would degrade accuracy, while the modular design keeps the added complexity manageable and organized.
Solution Approach 2:
The patent introduces an intermediary preprocessing layer between the raw text input and the classification algorithms. This intermediary layer acts as a mediator that cleans and prepares the data, ensuring high extraction accuracy while isolating the complexity of data cleaning from the main classification pipeline.
3Reliability
If manual enrichment of product databases is performed, then data quality can be controlled, but the process is time-consuming and labor-intensive
Solution Approach 1:
The patent applies self-service by implementing automated text processing that performs enrichment tasks independently without human intervention. The system automatically segments, clusters, and filters text to extract attribute-value pairs, eliminating the need for manual processing while maintaining data quality through algorithmic decision-making.
Solution Approach 2:
The patent replaces the mechanical manual process of reviewing and enriching product databases with an automated computational system. The mechanical action of human reviewers is substituted with automated algorithms that perform segmentation, clustering, and filtering, dramatically reducing time consumption while maintaining quality through structured processing.
4Device complexity
If a single classification technique is used, then the system remains simple, but it cannot be universally suited for all input data types
Solution Approach 1:
The patent implements universality by creating a preprocessing framework that can handle multiple types of input data (product descriptions, reviews, specifications) through a unified segmentation and clustering approach. This universal preprocessing layer adapts to different data types before they are processed by appropriate classification techniques, enabling the system to be suited for various applications.
Data Source
AI summary
Performance of statistical machine learning techniques, particularly classification techniques applied to the extraction of attributes and values concerning products, is improved by preprocessing a body of text to be analyzed to remove extraneous information. The body of text is split into a plurality of segments. In an embodiment, sentence identification criteria are applied to identify sentences as the plurality of segments. Thereafter, the plurality of segments are clustered to provide a plurality of clusters. One or more of the resulting clusters are then analyzed to identify segments having low relevance to their respective clusters. Such low relevance segments are then removed from their respective clusters and, consequently, from the body of text. As the resulting relevance-filtered body of text no longer includes portions of the body of text containing mostly extraneous information, the reliability of any subsequent statistical machine learning techniques may be improved.