Text Unnaturalness Evaluation Using Distribution Divergence
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for evaluating the unnaturalness of text data are time-consuming and require complex processes, such as using dedicated determination models.
Innovation Solution
An information processing apparatus with a reference distribution acquisition unit, a target distribution generation unit, and an evaluation value calculation unit is used to acquire a reference distribution, generate a target distribution, and calculate an evaluation value indicating the difference between the two distributions, thereby evaluating the unnaturalness of text data efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a dedicated determination model is used to accurately detect unnaturalness of text data, then measurement precision is improved, but processing time increases
Solution Approach 1:
The patent segments the text data processing into multiple independent components: n-gram extraction, frequency calculation, and divergence computation. Each component processes a specific aspect of the text independently, allowing for efficient parallel processing while maintaining detection accuracy. This segmentation enables the system to avoid using a complex dedicated determination model while still achieving accurate unnaturalness detection.
2Measurement precision
If a complicated cleansing process is used to evaluate text data, then measurement precision is improved, but device complexity increases
Solution Approach 1:
The patent extracts only the essential features needed for unnaturalness detection: n-gram frequencies and their divergence from reference distributions. By taking out only these critical elements and discarding unnecessary processing steps, the system achieves accurate text evaluation without requiring a complicated cleansing process or complex device architecture.
Solution Approach 2:
The patent changes the evaluation parameters from complex semantic analysis to simple statistical measures: n-gram frequencies and divergence values. This parameter transformation simplifies the evaluation process significantly while maintaining measurement precision, as the statistical parameters directly capture the unnaturalness characteristics of text data without requiring complex processing.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
An information processing apparatus (10) includes a reference distribution acquisition unit (105), a target distribution generation unit (104), and an evaluation value calculation unit (106). The reference distribution acquisition unit (105) acquires a reference distribution indicating a tendency of language information that is extracted from text data. The target distribution generation unit (104) generates a target distribution related to the language information. The evaluation value calculation unit (106) calculates an evaluation value indicating a difference between the reference distribution and the target distribution.