Text Generation via Data Preprocessing and Token Sequences
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for generating text and audio files, such as automated journalism and computer-generated content, are limited by their ability to process only specialized and structured data, making them less effective with larger and more diverse datasets, and lack control over output quality and relevance.
Innovation Solution
A method involving data preprocessing, normalization, and training corpus creation using libraries like spacy and transformers, which filters and standardizes data, allows for the generation of diverse content by using APIs and integrating control tokens and knowledge graphs to influence output, enabling the processing of larger datasets and improving text quality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If template-based methods are used for text generation, then text can be generated from structured data, but the scope of application is limited and cannot process larger and more diverse data sets
Solution Approach 1:
The patent transforms structured data into token sequences that can be processed by language models, changing the parameter representation from tabular form to sequential token form. This enables the system to handle diverse and larger datasets while maintaining controlled output quality through the language model's inherent structure.
2Adaptability or versatility
If Language Models are used for text generation, then diverse content can be generated, but control over output quality and relevance is reduced
Solution Approach 1:
The system performs preliminary actions by transforming structured data into carefully constructed token sequences before feeding them to the language model. This pre-processing step ensures that the input contains all necessary contextual information and constraints, enabling the model to generate diverse content while maintaining quality and relevance control.
3Productivity
If structured data is used for automated text generation, then text can be generated efficiently, but the data must be in tabular form which is less straightforward for human readers
Solution Approach 1:
The patent introduces token sequences as an intermediary representation between structured data and the language model. This intermediary form maintains the efficiency of structured data processing while enabling the generation of readable natural language output that is straightforward for human readers to understand.
Data Source
AI summary
A method for generating files, in particular text and audio files as well as files for computer games or videos. The processing of very large quantities of texts, which differ in content and structure, is thereby ensured. For this purpose, existing data is filtered and processed (cleaned) in a first step, and subsequently a training corpus is generated from the processed data, which is adjusted to the desired results. The filtered and processed data is standardized.