Automated Language Identifier Training Data Cleansing

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing automated language translation systems face reduced accuracy due to the presence of noise words and foreign words in training data, which are difficult to remove efficiently without manual intervention.

Innovation Solution

An automated method and system that calculates occurrence ratios of words in text data to identify and filter out noise words and foreign words by comparing their occurrence across multiple texts, generating clean training data for language identification systems.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If manual review is used to remove noise words and foreign words from training data, then cleaning accuracy is improved, but labor intensity and time consumption increase significantly

Engineering Contradiction:

Improvecleaning accuracyVSAvoidtime consumption

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The patent replaces manual mechanical review with an automated computational system that uses occurrence ratio analysis to identify and remove noise words and foreign words. The system calculates occurrence ratios of words across multiple texts and automatically filters out words with abnormal occurrence patterns, eliminating the need for human reviewers while maintaining cleaning effectiveness.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Productivity

If automated processing is used to clean training data, then productivity is improved, but cleaning precision may deteriorate without proper algorithms

Engineering Contradiction:

Improvedata processing speedVSAvoidcleaning precision

Core Design Contradiction:

ProductivityVSMeasurement precision

Solution Approach 1:

The patent transforms the cleaning task into a parameter-based problem by calculating occurrence ratios (a quantitative parameter) for each word across multiple texts. Words are identified for removal based on their occurrence ratio values compared against expected ranges, enabling automated processing with high precision through mathematical parameter analysis rather than qualitative manual judgment.

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The system uses occurrence ratio analysis as a feedback mechanism to automatically identify words that deviate from expected patterns. By comparing actual occurrence ratios against anticipated ranges derived from multiple reference texts, the system provides automated feedback for identifying noise words and foreign words, ensuring consistent and accurate cleaning without manual intervention.

Inventive Principle:

Principle #23Feedback

Data Source

PatentUS7729899B2Data cleansing system and method

Publication Date: 2010.06.01 BASIS TECHNOLOGY CORP

AI summary

An automated system and method is provided for debugging training data used to train an automated language identifier. The system and method collects texts written in a particular language, generates an occurrence count for words in each text by counting the number of times each of the words is found within the text, and generates an occurrence ratio (OR) of each of the words by dividing the occurrence count by the total number of words in each text. Words are then filtered from the texts in which their occurrence ratios are substantially higher than their occurrence ratios in at least one of the other texts, to generate a clean text.