LLM Listwise Ranking With Permutation Self-Consistency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large language models (LLMs) exhibit positional bias, leading to inconsistent and potentially incorrect listwise rankings due to factors like prompt order and input length, causing them to 'get lost in the middle' of long contexts.
Innovation Solution
Generate multiple LLM input prompts with permuted list orders, aggregate and compare LLM outputs to minimize positional bias by determining a final output that minimizes distance between rankings using permutation self-consistency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If LLMs process lists in a single fixed order, then the processing is simple and fast, but positional bias causes inaccurate rankings especially for middle items
Solution Approach 1:
The system performs preliminary actions by generating multiple permuted versions of the input list before processing. Each permuted list is created by randomly shuffling the original list items, so that positional bias in the original ordering cannot systematically affect the final ranking. This preliminary permutation step ensures that items are not unfairly positioned due to their original order in the input.
Solution Approach 2:
The system merges multiple LLM outputs by aggregating the rankings from different permuted inputs. Instead of relying on a single LLM output that may be biased by positional factors, the system combines results from multiple independent LLM calls on different permutations and uses consensus or aggregation methods to produce a final ranking that is more robust and accurate.
2Productivity
If LLMs use positional information for processing, then processing is efficient, but the model becomes 'lost in the middle' of long contexts
Solution Approach 1:
The system performs preliminary permutation of the input list before processing by the LLM. By randomly shuffling the list order beforehand, the system ensures that positional information does not systematically bias the LLM's processing. This preliminary action distributes positional bias across different positions, preventing the model from becoming 'lost in the middle' of long contexts.
Solution Approach 2:
The system uses feedback from multiple LLM outputs to refine the final ranking. By comparing results from different permuted inputs and using this feedback to identify consistent patterns, the system can correct for positional bias and improve context accuracy without sacrificing processing efficiency.
Data Source
AI summary
Systems, apparatuses, and methods are described for minimizing prompt order bias in a large language model (LLM). Using an original input prompt for an LLM, that may include instructions and ordered list, a plurality of different LLM input prompts may be generated. A plurality of LLM outputs may be determined, for example, by providing the plurality of LLM input prompts comprising the original instructions but with the order of the list permutated. A positional bias of the LLM may appear differently in the plurality of LLM outputs, for example, based on the differing list orders of the plurality of LLMs. A final LLM output may be generated, for example, by aggregating the LLM outputs to minimize the effects of positional bias.