Pre-trained Language Model Prompt Segmentation for Text Control

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing language models are difficult for users to manipulate and control, limiting their ability to generate desired text outputs.

Innovation Solution

A method and apparatus for pre-training a language model by acquiring sample natural language text, generating various types of prompt words, and using these to create sample input data for training an initial language model, resulting in a pre-trained model that can generate pseudo-natural language text.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If large-scale language models are used to achieve strong text generation capabilities, then the text generation quality is improved, but the controllability and ease of operation deteriorate

Engineering Contradiction:

Improvetext generation capabilityVSAvoiduser control over generation

Core Design Contradiction:

ReliabilityVSEase of operation

Solution Approach 1:

The patent segments the text generation control into multiple independent prompt word types (task type, topic type, style type, length type). Each type of prompt word independently controls a specific aspect of the generation process, allowing users to manipulate different dimensions of text output separately rather than as a monolithic control system.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent applies preliminary action by pre-defining multiple types of prompt words with specific functions before the text generation process. These prompt words are prepared in advance and can be selectively combined to control various aspects of generation, enabling users to plan and structure their desired output before actually generating text.

Inventive Principle:

Principle #10Preliminary action

2Device complexity

If traditional language models are used, then the model structure is simple, but the controllability and interpretability of text generation deteriorate

Engineering Contradiction:

Improvemodel structureVSAvoidcontrollability and interpretability

Core Design Contradiction:

Device complexityVSEase of operation

Solution Approach 1:

The patent introduces prompt words as intermediary elements between the user and the language model. These prompt words serve as mediators that translate user intentions into structured inputs for the model, making the interaction more interpretable and controllable without requiring changes to the underlying model architecture.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent segments the control mechanism into distinct prompt word types (task, topic, style, length), each handling a specific aspect of text generation control. This segmentation provides interpretability by clearly separating different control dimensions while maintaining a relatively simple model structure.

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS12314677B2Method for pre-training model, device, and storage medium

Publication Date: 2025.05.27 BEIJING BAIDU NETCOM SCI & TECH CO LTD

AI summary

A method and apparatus for pre-training a model, a device, a storage medium, and a program product. An embodiment of the method includes: acquiring a sample natural language text; generating N types of prompt words based on the sample natural language text, where N is a positive integer; generating sample input data based on the sample natural language text and the N types of prompt words; and training an initial language model based on the sample input data, to obtain a pre-trained language model.