LLM Decoding With Approximate Gradient Logit Self-Correction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large Language Models (LLMs) tend to hallucinate or distort truth, leading to unreliable and inaccurate outputs despite extensive pre-training, with existing methods like retrieval techniques and fine-tuning failing to fully address this issue.
Innovation Solution
A Self-Evolution Decoding (SED) strategy that optimizes LLM outputs during inference by leveraging the latent knowledge within the model's hidden states, adjusting logits through an approximate gradient approach to align with real-world factuality without requiring external knowledge bases or retraining.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If LLMs are used to generate outputs based on pre-training, then the model can produce responses quickly, but the outputs may contain hallucinations or factual errors reducing reliability
Solution Approach 1:
The patent performs preliminary actions by computing logits from multiple layers in advance during the inference process. The method obtains logits from an evolution layer and multiple preceding layers, then processes these pre-computed logits to generate approximated gradients before final output generation. This preliminary computation of layer logits enables subsequent gradient-based adjustments without requiring additional forward passes, thus improving reliability while maintaining inference efficiency
Solution Approach 2:
The patent implements feedback by using the approximated gradient (computed from the difference between evolution layer logits and preceding layer logits) to adjust the evolution layer logits. The adjusted logits are then used to generate the final output. This feedback mechanism allows the model to self-correct potential hallucinations by leveraging the evolutionary information from earlier layers, improving output factuality without significantly increasing inference time
2Reliability
If retrieval techniques and fine-tuning are used to improve LLM factuality, then output accuracy may improve, but the complexity and resource requirements increase
Solution Approach 1:
The patent enables the LLM to self-correct its outputs by leveraging its own internal layer representations. The method computes approximated gradients using logits from the model's own evolution layer and preceding layers, then uses these gradients to adjust the logits and generate corrected outputs. This self-service mechanism eliminates the need for external knowledge bases, retrieval systems, or additional fine-tuned models, reducing system complexity while improving output accuracy
Solution Approach 2:
The patent changes the parameters (logits) of the evolution layer by adjusting them based on approximated gradients derived from the difference between evolution layer and preceding layer logits. This parameter adjustment allows the model to refine its output predictions dynamically during inference, improving accuracy without requiring external interventions or complex system architectures
Data Source
AI summary
Systems, methods, and apparatus for self-evolving decoding at inference. In an aspect, operations include processing, by a Large Language Model (LLM) of N layers, an input by an inference operation of the LLM; obtaining, from the LLM, logits of an evolution layer of the LLM, the evolution layer being subsequent to a first layer of the LLM; for a plurality of layers that occur before the evolution layer, processing the logits of the layer with the logits of the evolution layer to generate an approximated gradient; based on the approximated gradient and the logits of the evolution layer, generating adjusted logits for the evolution layer; and processing the adjusted logits for the evolution layer to generate an output for the LLM.