Sentence-Level Text Noise Removal for Analysis Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional text preprocessing techniques primarily focus on removing word-level noise and fail to effectively address sentence-level noise, which is necessary for improving the quality and performance of text analysis tasks.
Innovation Solution
A method and apparatus for inspecting text quality, correcting text, selecting noise candidates at the sentence level, and removing noise based on the purpose of the text analysis, utilizing a computing device with a processor and network interface to execute instructions for text preprocessing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If conventional word-level noise removal is applied, then word-level text cleaning is improved, but sentence-level noise remains unaddressed
Solution Approach 1:
The patent segments the text processing task into two distinct levels: word-level preprocessing (stop-word removal, special character removal) and sentence-level noise removal. By dividing the text into sentences and identifying sentence-level noise patterns separately, the system addresses both granularities of noise without conflating their removal strategies.
Solution Approach 2:
The patent introduces an intermediary sentence-level processing stage between word-level tokenization and final text analysis. This intermediary layer identifies and removes sentence-level noise (such as repeatedly recorded sentences) that would otherwise persist after word-level cleaning, acting as a bridge that catches noise at the appropriate linguistic level.
2Manufacturing precision
If comprehensive noise removal is applied to all texts, then text quality is improved, but adaptability to different text purposes is reduced
Solution Approach 1:
The patent implements dynamic noise removal by adjusting the noise removal level according to the specific purpose of the text analysis. The system can selectively apply different degrees of noise removal (from aggressive to conservative) based on whether the task requires high precision (e.g., academic analysis) or maintains more original content (e.g., sentiment analysis), making the preprocessing adaptable rather than static.
Solution Approach 2:
The patent changes the parameter of noise removal intensity based on the text analysis purpose. By allowing the noise removal level to be adjusted as a configurable parameter, the system can optimize text quality for different downstream tasks without being locked into a single removal strategy, thus maintaining versatility across applications.
3Productivity
If sentence-level noise removal is implemented, then text analysis performance is improved, but processing complexity increases
Solution Approach 1:
The patent segments the complex task of sentence-level noise removal into manageable sub-tasks: sentence tokenization, noise pattern identification, noise classification, and selective removal. This segmentation allows the system to handle complexity in discrete, modular steps rather than attempting to solve all noise issues simultaneously, making the processing more tractable and maintainable.
Data Source
AI summary
A method for removing text noise according to an embodiment of the present disclosure includes inspecting quality of the text, correcting the text based on a result of inspection; selecting a noise candidate based on each type of sentences included in the corrected text, wherein the noise candidate is selected for each sentence included in the text and removing at least some of the sentences included in the noise candidate based on the purpose of the text.