Copy-Or-Generate Decoding for Faster, Secure Text Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large neural language models face issues with slow inference time, high computational cost, and security vulnerabilities due to prompt attacks, while smaller transformer models struggle with generalization on longer input lengths.
Innovation Solution
A copy-or-generate model architecture with a semi-sandboxed or fully sandboxed decoding configuration that uses context compression and embedding aggregation, enabling efficient text generation by determining whether to generate or copy tokens based on learnable weights, thus preventing data leakage and improving generalization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If large neural language models are used to improve text generation performance and generalization, then accuracy and functionality are improved, but inference time increases and computational cost increases
Solution Approach 1:
The patent segments the text generation process into two distinct pathways: a generation pathway that creates new tokens and a copying pathway that directly copies tokens from the input context. This segmentation allows the model to use a smaller architecture while achieving high accuracy by selectively copying relevant information rather than generating everything from scratch, thereby reducing inference time while maintaining text generation quality.
Solution Approach 2:
The patent introduces a copying mechanism where the model can directly copy tokens from the input context to the output, bypassing the need for complex generation processes. This copying capability allows smaller models to achieve performance comparable to large models by efficiently reusing input tokens, thus reducing computational cost and inference time while maintaining accuracy.
2Adaptability or versatility
If large neural language models are used to improve text generation performance, then functionality is improved, but computational cost increases
Solution Approach 1:
The patent implements a dynamic switching mechanism that adaptively chooses between generation and copying operations based on the specific context and requirements of each text generation task. This dynamic approach allows the model to optimize computational resource usage by selecting the most efficient operation for each token, maintaining high functionality while reducing overall computational cost compared to static large models.
Solution Approach 2:
The patent changes the operational parameters of the model by introducing a copy probability parameter that controls the likelihood of copying versus generating tokens. This parameter adjustment allows the model to adapt its behavior to different computational budgets and task requirements, achieving high functionality with variable computational costs depending on the specific generation needs.
3Reliability
If models are designed to handle prompt attacks and prevent data leakage, then security is improved, but model complexity increases
Solution Approach 1:
The patent extracts sensitive information from the input context and stores it in a separate memory structure, preventing the model from directly accessing or memorizing proprietary data during training. This extraction approach allows the model to utilize the information for generation tasks while maintaining security, as the actual sensitive data never becomes part of the model's internal representations, thus improving security without significantly increasing complexity.
Solution Approach 2:
The patent introduces an intermediary copying mechanism that acts as a mediator between the input context and the output generation. Instead of the model directly learning from and potentially leaking sensitive input data, the copying mechanism selectively reproduces necessary tokens without requiring the model to internalize the data, thereby preventing prompt injection attacks and data leakage while maintaining functional complexity at acceptable levels.
Data Source
AI summary
A copy-or-generate model architecture is provided that a generates generation distribution obtained from the outputs of the last decoder layer and a copy distribution built from the cross-attention scores of the last decoder layer. The model applies copy weights to the generation distribution and copy distribution to determine whether to generate a next token or to copy a token from the prompt. The model provides better security by ensuring that the input values from the prompt are directly copied to the output when appropriate, such that the model is blind to the original values to copy. In a semi-sandboxed configuration, additional information may be input to the model to help the model adapt the output based on the context of those input fields.