Synthetic QA Generation Using LLMs and Entailment Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The creation of high-quality question-answer (QA) pairs tailored to specific domains is challenging due to the time-consuming and resource-intensive nature of manual or crowdsourced efforts, and existing techniques using single prompts generate low-quality QA pairs with similar questions and answers.
Innovation Solution
The use of large language models to generate synthetic QA pairs by chunking and partitioning textual data from a knowledge base repository, with question and answer generation models prompted to create context-based questions and answers, followed by filtering using textual entailment and human evaluations to remove low-quality pairs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If manual or crowdsourced efforts are used to generate domain-specific QA pairs, then data quality and domain expertise are improved, but time consumption and resource requirements increase significantly
Solution Approach 1:
The patent uses pre-trained language models to automatically generate QA pairs by copying and adapting patterns from existing domain-specific text, replacing manual creation while maintaining domain accuracy through the model's learned understanding of domain terminology and structures
Solution Approach 2:
The system enables self-service data generation where the language model autonomously creates QA pairs from domain text without human intervention, except for optional filtering stages, making the process automated and scalable while preserving quality through the model's training
2Productivity
If a single prompt with few-shot examples is used to generate QA pairs, then generation speed is improved, but question diversity and answer quality deteriorate
Solution Approach 1:
The patent segments the QA generation process into multiple independent prompting stages: first generating questions from domain text, then generating answers from the same text. This segmentation allows each stage to focus on one aspect, improving both diversity and quality while maintaining efficiency
Solution Approach 2:
The patent adds a new dimension to the generation process by using separate prompting for questions and answers rather than generating them simultaneously in a single prompt. This dimensional separation enables greater flexibility and diversity in the generated content
Data Source
AI summary
In various examples, synthetic question-answer (QA) pairs may be generated using question and answer generation models comprising corresponding language models (e.g., autoregressive LLMs). A repository of textual data representing a particular knowledge base may be used to source synthetic QA pairs by partitioning textual data from the repository into units of text (e.g., paragraphs) that represent context. For each unit of text, the question generation model may be prompted to generate a synthetic question from that unit of text, and the answer generation model may be prompted to generate a synthetic answer to the synthetic question. Textual entailment and/or human evaluations may be used to filter out low quality, incorrect, and/or non-productive QA pairs that may be a result of hallucinations. As such, the synthetic QA pairs may be used as, and/or may be used to generate, training data for one or more machine learning models.