Mixed-Text Detection Using Synthetic Machine-Generated Calibration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing difficulty in distinguishing between human-generated and machine-generated text, particularly in mixed texts edited by humans, poses challenges in assessing student work, identifying false information, and detecting copyright infringement.
Innovation Solution
A method involving the generation of a synthetic mixed text dataset by replacing portions of human-generated text with machine-generated text, using a large language model, and applying a text discriminator score to identify the proportion of machine-generated text through kernel density estimation and Metropolis-Hastings algorithm sampling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If LLM-generated text is edited by humans, then the text becomes more human-like and harder to detect, but the difficulty in detecting machine-generated text increases
Solution Approach 1:
The system performs preliminary actions by creating a synthetic mixed text dataset with known proportions of machine-generated text before evaluating the detector. This allows the system to calibrate and validate its detection capabilities against controlled conditions where the ground truth is known, enabling accurate detection even in texts that have been edited to appear more human-like.
Solution Approach 2:
The patent introduces an intermediary approach by using a synthetic mixed text dataset as a mediator between pure human-generated text and pure machine-generated text. This intermediate representation with known machine-generated proportions serves as a calibration standard, allowing the detection system to accurately measure and identify machine-generated content even when it has been subsequently edited by humans.
2Reliability
If a detection system is developed to identify machine-generated text, then detection capability improves, but the complexity of the system increases
Solution Approach 1:
The system performs preliminary calibration by generating a synthetic mixed text dataset with known machine-generated text proportions before actual detection. This preliminary action establishes reference points and calibration curves that simplify the actual detection process, allowing the system to achieve high reliability without requiring overly complex real-time analysis during normal operation.
Solution Approach 2:
The patent creates a synthetic copy of the detection problem by generating artificial mixed text datasets that replicate the characteristics of real-world mixed text. This copying approach allows the system to develop and validate detection algorithms on controlled synthetic data, reducing the complexity of working with real-world variations while maintaining detection accuracy.
3Measurement precision
If synthetic mixed text dataset is generated by replacing portions of human-generated text with machine-generated text, then detection accuracy improves, but the time required for dataset generation increases
Solution Approach 1:
The system performs the time-consuming synthetic dataset generation as a preliminary action that is done once or infrequently, rather than continuously. By pre-generating and storing the synthetic mixed text datasets with various machine-generated text proportions, the system avoids the time cost of regenerating these datasets during each detection task, thereby maintaining high measurement precision without incurring continuous time penalties.
Solution Approach 2:
The patent uses synthetic copying to create representative samples of mixed text with known machine-generated proportions. These synthetic copies serve as pre-computed reference data that can be quickly queried and evaluated during detection tasks, avoiding the need to perform time-consuming generation operations repeatedly while maintaining accurate measurement capabilities.
Data Source
AI summary
Systems and methods for detecting human-generated text and machine-generated text in mixed text are disclosed. A computer program receives a dataset of human-generated text and generates a synthetic mixed text dataset by randomly replacing portions of the human-generated text with machine-generated text. The program evaluates the synthetic mixed text dataset at multiple levels, each representing a different fraction of machine-generated text, and computes a first text discriminator score for each level. Upon receiving a text sample, the program computes a second text discriminator score, fits a kernel density estimate using a Gaussian kernel to model a conditional probability distribution, samples from a posterior distribution, and returns an interval representing the amount of machine-generated text in the sample. The program can also split the text sample into segments, define concatenated subsequences, identify indices that maximize an objective function, and return identified machine-generated segments to a user electronic device.