Text Classification via Pause-Based Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing text classification systems are hindered by fixed form expressions, such as greetings, which introduce noise and reduce classification accuracy by increasing cosine similarity degrees and decreasing the term frequency value of feature words, making it difficult to extract appropriate features and classify documents effectively.
Innovation Solution
A method that involves receiving and analyzing text data, identifying pause parts, dividing the text at these parts, and generating new texts, which are then classified using vectorization and clustering techniques to separate and extract fixed form expressions, thereby improving classification accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If fixed form expressions are included in texts, then the texts can maintain their original structure and meaning, but the classification accuracy deteriorates due to noise introduction and feature extraction failure
Solution Approach 1:
The text is divided into multiple segments by detecting pause parts (commas, periods, question marks, exclamation marks) and splitting at these boundaries. This segmentation isolates fixed form expressions from the main content, allowing the classification system to process only the meaningful portions while maintaining the original text structure intact.
Solution Approach 2:
The system extracts and removes fixed form expressions (greetings, closing remarks) from the text by identifying pause parts that indicate these expressions. The extracted segments containing only meaningful content are then used for classification, eliminating the harmful noise while preserving the essential information.
2Loss of information
If fixed form expressions are present in documents, then the documents maintain their complete information, but the term frequency value of feature words decreases due to dilution
Solution Approach 1:
By segmenting the text at pause parts, the system separates fixed form expressions from feature words. This ensures that term frequency calculations are performed only on meaningful segments, preventing dilution of feature word frequencies while maintaining complete information about the document content.
Solution Approach 2:
The system applies different processing qualities to different parts of the text: fixed form expressions are excluded from feature extraction, while meaningful segments retain their full term frequency values. This local differentiation preserves information quality where needed while eliminating noise where harmful.
3Productivity
If texts are processed as a whole without division, then the processing is simple and fast, but the classification accuracy deteriorates due to inability to handle fixed form expressions
Solution Approach 1:
The text processing is divided into two simple stages: (1) detecting pause parts and splitting the text, and (2) performing classification on the segmented portions. This segmentation approach is computationally efficient and maintains processing speed while significantly improving classification accuracy by eliminating fixed form expression interference.
Data Source
AI summary
A text classification method is performed in a computer. The method includes: receiving a plurality of texts. When detecting that a text among the received plurality of texts includes a pause part satisfying a specific condition, dividing the text at the pause part and generating a new plurality of texts. The method may also include classifying texts, among the received plurality of texts, not including the pause part satisfying the specific condition, and the generated new plurality of texts into a plurality of clusters.