Entity-Focused Natural Language Generation with Sentinel Tokens

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing natural language generation (NLG) techniques struggle to generate accurate and diverse synthetic data for personal identifiable information (PII) entities due to the scarcity of pre-labeled training data and the inability to capture interrelationships between entities and context, leading to inaccurate and contextually incoherent outputs.

Innovation Solution

A computer-implemented method involving populating fake entity values, inserting sentinel tokens, generating natural language sentences, and using text summarization and next sentence generation models to ensure accurate and diverse synthetic data generation, with pre-processing and post-processing steps to enhance grammar and context.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Manufacturing precision

If traditional NLG techniques are used to generate synthetic data for PII entities, then the process is simple, but the accuracy and diversity of the generated data are insufficient

Engineering Contradiction:

Improveaccuracy of synthetic dataVSAvoidcomplexity of data generation process

Core Design Contradiction:

Manufacturing precisionVSDevice complexity

Solution Approach 1:

The data generation process is divided into multiple stages: entity value generation, sentence generation, pre-processing, quality analysis, and post-processing. Each stage handles a specific aspect of data generation, allowing complex requirements to be met through coordinated simple operations across stages.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Fake entity values are generated and prepared in advance before sentence generation. The system pre-processes generated sentences by tokenization and context correction before final quality assessment, ensuring high accuracy outputs through preparatory processing.

Inventive Principle:

Principle #10Preliminary action

2Reliability

If more processing stages are added to improve data quality, then the accuracy and diversity improve, but the processing time increases

Engineering Contradiction:

Improvequality of synthetic dataVSAvoidprocessing time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system includes a quality analysis stage that evaluates generated sentences and provides feedback to the post-processing stage. This feedback mechanism allows the system to iteratively improve data quality while managing processing time through targeted corrections rather than complete regeneration.

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

Low-quality generated sentences are discarded and replaced through the post-processing stage, while high-quality sentences are recovered and used in the final dataset. This selective approach maintains quality standards while minimizing unnecessary processing time on already good outputs.

Inventive Principle:

Principle #34Discarding and recovering

3Loss of information

If existing NLG models are used, then the implementation is straightforward, but the models cannot capture interrelationships between entities and context

Engineering Contradiction:

Improvecontextual coherenceVSAvoidcomplexity of model architecture

Core Design Contradiction:

Loss of informationVSDevice complexity

Solution Approach 1:

A context correction model is introduced as an intermediary component between sentence generation and quality analysis. This intermediary specifically addresses contextual coherence by correcting sentences to ensure proper capture of entity relationships and context, bridging the gap between simple generation and complex requirements.

Inventive Principle:

Principle #24Intermediary (Mediator)

4Productivity

If synthetic data is generated without ensuring entity presence, then the generation process is faster, but the data does not meet training requirements

Engineering Contradiction:

Improvedata generation speedVSAvoidentity representation accuracy

Core Design Contradiction:

ProductivityVSManufacturing precision

Solution Approach 1:

Fake entity values are generated and inserted into sentences in advance during the entity value generation stage. This preliminary action ensures that required entities are present in the generated data before final quality assessment, maintaining both speed and precision by preparing entity placements ahead of time.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS20250291955A1Entity focused natural language generation

Publication Date: 2025.09.18 ORACLE INT CORP

AI summary

Method includes populating fake value for each of entities, to generate string of fake entity values that correspond to entities; inserting sentinel token between adjacent fake values included in the string to generate first input data; generating, by natural language generation model, natural language sentences based on first input data, natural language sentences including one or more fake values from the string; analyzing natural language sentences to determine whether any fake value from the string is missing; based on the fake value missing, summarizing, using text summarization model, natural language sentences to generate text summary; concatenating the text summary with the fake value, to generate second input data; and generating, by a next sentence generation model, additional natural language sentence using the second input data, the additional natural language sentence including the fake value. Additional natural language sentence is combined with natural language sentences to generate final natural language sentences.