Method for selecting or recommending a language model for generating a text summary
The method addresses the issue of positional bias in LLMs by quantifying and minimizing it, ensuring a balanced and comprehensive text summary by selecting or recommending LLMs that evenly represent information from the beginning to the end.
Patent Information
- Application Number
- PCT/EP2026/061804
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-14
- Filing Date
- 2026-04-24
- Publication Date
- 2026-11-19
AI Technical Summary
Existing methods for selecting language models for text summarization fail to account for positional bias, leading to unbalanced summaries that omit crucial information in the middle of the text due to inherent biases in extended language models (LLMs), and lack a practical and objective way to quantify this bias.
A computer-implemented method to evaluate and compare candidate LLMs by calculating positional bias scores, including direction and magnitude, to select or recommend a model that minimizes positional bias, using semantic similarity and dynamic thresholds to identify ignored sentences and weight them based on length.
Ensures a more reliable and representative summary by reducing the likelihood of omitting important information, providing an objective and adaptable method for selecting LLMs that balance information across the entire text.
Smart Images

Figure EP2026061804_19112026_PF_FP_ABST
Abstract
Description
Method for selecting or recommending a language model for generating a text summary technical field
[0001] The present invention relates to the general field of natural language processing (NLP) and more particularly to the automatic generation of text summaries. It specifically targets methods and systems using language models, for example extended language models (LLMs), to create summaries, and aims at a method for selecting or recommending a language model from among several candidates based on a quantitative evaluation of their respective positional biases, in order to promote the generation of a more balanced summary that is representative of the source text in its entirety. Previous technique
[0002] The ability to concisely summarize essential information from large text documents is crucial in many fields. Whether it's transcripts of conversations (meetings, customer service calls), long articles, reports, or other types of text, automatically generating reliable summaries saves time and improves access to key information.
[0003] The emergence of extended language models (LLMs), capable of understanding and generating text with unprecedented fluency, has significantly advanced the field of abstract summarization. These models can produce relevant summaries of diverse texts, often without requiring any specific prior training ("zero-shot" mode).
[0004] However, the increasing use of these LLMs for summarizing has raised concerns about the potential biases introduced into the generated texts. One particularly detrimental bias in this context is positional bias. This is the tendency inherent in many models to give more weight to information located in specific positions in the source text—typically the beginning (lead bias) and / or the end (recency bias)—at the expense of information in other parts, especially the middle of the text. This phenomenon, sometimes described as a U-shaped attention pattern, can lead to the omission or underrepresentation of crucial elements (decisions, key arguments, important details) simply because of their position in the source text.The quality, completeness, and reliability of the summary are thus compromised, potentially providing a partial or misleading view of the original content.
[0005] Existing methods for evaluating the quality of summaries, such as lexical overlap metrics (e.g., ROUGE), do not measure this positional bias and provide no information on how the source text content was used and weighted by the model. Similarly, the choice of an LLM for a summarization task is often based on general criteria (average performance on various benchmarks, popularity, computational cost, energy efficiency) or is imposed by default, without specific and quantitative consideration of its positional bias behavior for the task and text in question.
[0006] Although the existence of positional bias is known, prior art lacks a practical and objective way to quantify this bias. Summary
[0007] This disclosure improves the situation.
[0008] To this end, a computer-implemented method is proposed for selecting or recommending a language model for generating a text summary, the method comprising: - access at least two candidate language models capable of generating a summary of said text; - for the two candidate language models, calculate respectively two positional bias scores quantifying a tendency of the two candidate language models to favour information located at least one position in said text when generating a summary; said two positional bias scores comprising at least a measure of the direction of the bias and a measure of the magnitude of the bias; - compare the two positional bias scores; and - select or recommend at least one language model from the two language models based on said comparison.
[0009] The term "computer-implemented" refers to the fact that the various stages of the process are executed by hardware for information processing, typically comprising one or more processors (or computers) and one or more associated memories. These resources can be centralized (e.g., a single server, a personal computer) or distributed (e.g., a cloud-based system, a computer network).
[0010] The term "language model" here refers to an artificial intelligence computer system, for example, one based on deep learning architectures such as Transformer neural networks. These models are pre-trained on large volumes of textual data, giving them the ability to establish semantic relationships, process, and generate natural language to perform various tasks. In the context of this invention, their exploited capability is that of "generating a text summary." This means producing a new text, shorter than the original source text (which could be, for example, a conversation transcript, a newspaper article, a technical report, etc.), but which concisely captures, for example, the essential points, main ideas, or salient semantic content.This is, for example, an abstractive generation, where the model reformulates the information, rather than a simple extraction of sentences.
[0011] The term "selection or recommendation" refers to the final stage of the process where, after bias analysis, one or more models are designated as preferable. "Selection" typically involves an automatic choice made by the system, for example, to directly use the chosen model to generate the final summary. "Recommendation," on the other hand, involves presenting this choice (or a bias-based ranking) to a human user or another system, which can then make the final decision or be informed of the bias characteristics of the various available models.
[0012] The term "candidate" refers to a language model that is one of the two or more models available and evaluated by the process. Before the bias score calculation and comparison stage, all accessible models are considered potential candidates for the summarization task.
[0013] The "positional bias score" refers to a numerical value or set of numerical values, calculated specifically for a given candidate language model and a particular source text. This score aims to objectively quantify the model's tendency to assign unequal weight to information based on its location (position) within the source text when generating the summary. For example, one model might consistently better represent the first and last sentences of a report (strong bias), while another might better balance information across the entire length of the text (weak bias).
[0014] The "bias direction" is a component of this score that indicates which position(s) the model tends to favor. Typically, this distinguishes between primacy bias (the model favors the beginning of the text) and recency bias (the model favors the end of the text). The direction can be represented by a sign (+ / -), a category (e.g., "Lead," "Recency," "Neutral"), or a value on a defined scale.
[0015] The "amplitude of bias" (or magnitude) is another component of the score that measures the intensity or strength of this positional bias, without necessarily taking into account its preferred direction. A high amplitude means that the model has a strong preference for certain positions (whether the beginning, the end, or both), resulting in a high risk of omitting information located elsewhere. Conversely, a low amplitude (close to zero) indicates that the model processes information more evenly and in a balanced way regardless of its position in the text, which is generally desirable for a representative summary.
[0016] The action of "comparing these scores" refers to the step where the positional bias scores (specifically the amplitudes, but potentially also the directions) calculated for the different candidate language models are compared. This comparison allows for a relative ranking of the models according to their level of positional bias for the text under consideration, or for the identification of those that meet a predefined criterion (for example, those whose amplitude is below a threshold).
[0017] The method according to the invention provides a technical solution to the unresolved problem of making an informed choice of a language model for summarizing, specifically taking into account positional bias. Prior art lacked a systematic and quantitative method for evaluating and comparing this bias before choosing the model and generating the final summary. The choice was often based on general criteria (particularly by performing global evaluations of language models and prompts on corpora) or made by default, with the risk of obtaining an unrepresentative summary that omits key information located in the middle of the source text.
[0018] The invention proposes a different approach: instead of being subject to the bias of a pre-selected model, it explicitly evaluates this bias for each available candidate model, specifically for the text to be summarized. The invention involves calculating a positional bias score comprising both a direction and a magnitude. This quantification allows for an objective measurement of each model's tendency. Comparing these scores between the different candidates then makes it possible to identify the model(s) that represent the best choice, that is, those with the lowest bias magnitude.
[0019] The advantage is obtaining a final summary that is potentially much more reliable and representative. By selecting or recommending the model least susceptible to positional bias (low amplitude), the likelihood that all important parts of the source text, including those in the middle, will be given a balanced consideration in the summary is significantly increased. This prevents the loss of important information solely due to its position.
[0020] Furthermore, the process provides an objective, quantifiable, and directly relevant selection (or recommendation) criterion for the positional quality of the abstract. It offers considerable flexibility as it can be applied to any type of text and any collection of candidate language models. The comparison result can be used to automate the selection of the appropriate LLM or to provide valuable information to a user, enabling them to make an informed choice of the model best suited to their needs for a balanced abstract.
[0021] The calculation of the positional bias score for a candidate language model may include prior identification of sentences ignored from said text by a summary generated by said candidate language model.
[0022] This step introduces a mechanism for assessing positional bias. Instead of analyzing the complex and often opaque internal workings of the language model, or relying on global metrics that ignore the position of information, the proposed approach examines the concrete result of applying the model—that is, the generated summary. More specifically, it aims to determine which portions of the original source text, such as sentences or semantic segments, are insufficiently or incorrectly represented in the summary produced by the evaluated candidate model. These portions are then designated as "ignored," not necessarily in a literal sense of total absence, but rather because their essential semantic content has not been adequately rendered in the summary.This identification can typically be based on a measure of semantic similarity calculated between each portion of the source text and all the portions of the summary, the source portions not reaching a threshold of. predefined or dynamic similarity with any part of the summary is then classified as "ignored".
[0023] Adopting this preliminary step of identifying ignored sentences allows for the direct measurement of the observable and tangible impact of the model's inherent positional bias on the actual synthesis output. This contrasts with the analysis of internal attention weights, which are complex, vary across models, and do not guarantee the final content, or with global metrics, which provide no information about the source of the summarized information. Furthermore, using a semantic similarity measure for this identification is particularly well-suited to the abstractive summaries produced by modern language models, which frequently reformulate and paraphrase the source content. Methods limited to simple lexical overlap would fail to accurately identify semantic correspondences and, consequently, the true omissions in such summaries.Another advantage is that the method does not require a reference summary. The process directly compares the summary generated by the model to the original source text, making it applicable in scenarios where creating references is prohibitively expensive, time-consuming, or simply impossible.
[0024] It should also be noted that an alternative or complementary approach could consist of analyzing not the ignored sentences, but on the contrary the sentences of the source text which are actually well represented in the summary, that is to say those whose semantic similarity score with at least part of the summary is high.
[0025] The aforementioned identification of ignored phrases in the text may include: - calculate, for each sentence C i (i=1 n) of said text, a maximum semantic similarity score s t max with at least one sentence from a summary generated by said candidate language model; - apply a dynamic threshold to said Sj scores max to identify a set of k phrases Sj(j = 0,...,fc - 1) as being the ignored sentences, said ignored sentences being those whose score s™ ax is below that threshold.
[0026] The technical implementation of the identification of ignored sentences relies on a detailed quantitative analysis of the semantic relationship between the source text and the generated summary. More precisely, the process assesses the extent to which each informative segment of the text, typically each sentence (where i ranges from 1 to n, n being the total number of sentences in the text), finds a semantic echo in the summary produced by the candidate language model. To do this, a score is calculated for each source sentence, representing its best possible semantic match with all the sentences composing the summary. This score, denoted s- n “ xThis is achieved by determining the highest semantic similarity between the source sentence and any of the sentences in the generated summary. Using maximum semantic similarity ensures that even if a source sentence is paraphrased or its content is spread across multiple sentences in the summary, its strongest potential contribution is captured. This approach is useful for efficiently processing abstractive summaries where direct lexical matches are rare.
[0027] Following this calculation, we have a distribution of scores s™ ax For each sentence in the source text, the score indicates the degree of semantic coverage of each sentence by the summary. In order to objectively identify sentences that are significantly less well represented than others in this specific context, a dynamic threshold is applied to this score distribution. sRather than using a fixed and arbitrary threshold value, which might not be suitable for the variable nature of texts and summaries (some of which may present a higher overall similarity than others), the threshold is determined based on the statistical characteristics of the distribution of scores. x obtained for this particular text-summary pair. One method consists of calculating the mean and standard deviation of these scores (possibly after a normalization step such as a quantile transformation to stabilize the distribution) and defining the threshold relative to these values. The source sentences whose score s 10 associated values that fall below this dynamic threshold are then considered "ignored" by the summary.
[0028] The advantage of this approach, combining maximum semantic similarity and a dynamic threshold, is twofold. Firstly, it provides a robust and adaptable method for quantifying the representation of each source sentence, even in the presence of abstractive reformulation. Secondly, it allows for an objective and contextual identification of the least well-covered sentences, based on their relative position within the similarity distribution specific to that particular summary, rather than on rigid external criteria. This resulting set of k sentences identified as "ignored," indexed, for example, as s7 (where j ranges from 0 to k - 1), then constitutes a reliable and relevant database for the subsequent analysis of the positional distribution, enabling the calculation of primacy, recency, and amplitude bias scores with greater precision and significance.
[0029] This dynamic threshold can be calculated based on the mean / z and the standard deviation <r desdits scores s i max , for example as threshold = / z - <r, éventuellement après une transformation quantile desdits scores si max The "quantile transformation" refers to a technique for preprocessing or normalizing numerical data. Its goal is to map the values of an original data distribution (here, the maximum semantic similarity scores s™). ax ,the quantiles of a predefined reference distribution). This reference distribution can be chosen to be the standard normal distribution (i.e., a Gaussian distribution with mean 0 and standard deviation 1).
[0030] The principle of this transformation is to replace each original score value with the corresponding quantile value in the target reference distribution, while preserving the relative ranking of the original scores. For example, if a score s™ ax is at the 80th percentile of the original score distribution, it will be replaced by the value corresponding to the 80th percentile of the standard normal distribution (if this is the target).
[0031] Applying a quantile transformation to the s-scores xCalculating the mean and standard deviation beforehand, which are used to determine the dynamic threshold, helps to make the distribution of scores more symmetrical and closer to normality (if a normal distribution is the target), regardless of the potentially asymmetrical or irregular distribution of the raw similarity scores. Crucially, it stabilizes variance and reduces sensitivity to outliers, thus making the scores and the resulting dynamic threshold more robust and comparable across conversations or summaries, even if the scales or ranges of the raw similarity scores vary considerably. This contributes to a more reliable and consistent identification of phrases that are effectively "ignored."
[0032] The calculation of the positional bias score can use a weight wj associated with each ignored sentence s7, said weight Wj being a function of the length l tof the said ignored sentence s7, for example according to the formula
[0033] To refine the calculation of the positional bias score and better reflect the relative importance of the different ignored sentences, the process can incorporate a weighting step. This step involves assigning a numerical weight Wj to each previously identified ignored sentence s7. This weight is not uniform but is determined based on an intrinsic characteristic of the ignored sentence itself, typically its length lj (measured, for example, in the number of words or tokens). The underlying intuition is that the omission or misrepresentation of a long, potentially information-rich sentence may be more indicative of significant bias than the omission of a very short and possibly less substantial sentence.
[0034] An efficient method for calculating these weights Wj, as envisaged in one embodiment, uses a Softmax-type function applied to the lengths lj of the k ignored sentences. Specifically, the weight w7 of the ignored sentence s7 can be calculated according to the aforementioned formula. In this formula, exponentiation of the length amplifies the importance of longer sentences in a non-linear fashion, while division by the sum of the exponentials of all lengths ensures normalization, guaranteeing that the weights are relative to each other and can be used in subsequent weighted average calculations to determine the bias scores Bl and Br.
[0035] Integrating this weighting based on the length of omitted sentences provides several technical benefits for the accuracy and robustness of positional bias assessment. By assigning greater weight to longer omissions, the method implicitly gives more influence to potentially more semantically rich segments in the final bias calculation. This allows the analysis to be better focused on the potentially most damaging informational gaps in the abstract.
[0036] Simultaneously, this weighting reduces the impact of very short phrases that might have been classified as ignored (e.g., interjections, brief responses like "Yes." or "Okay.") but whose omission is likely less significant for judging the overall balance of the summary. The calculation of the Bl and Br scores thus becomes less sensitive to the "noise" potentially introduced by these minor omissions and gains in robustness, more reliably reflecting the model's tendency to neglect substantial chunks of information located at specific positions. The use of the Softmax function, in particular, ensures a marked differentiation between lengths while maintaining consistent normalization of weights. Consequently, the final bias scores are more influenced by the position of the most significant omissions (the longest ignored sentences), leading to a more relevant assessment of the effective positional bias of the candidate language model.
[0037] The calculation of the positional bias score may include the calculation of a primacy bias score S ; representing the tendency to favor the beginning of the text, said score S ; being calculated as a weighted average by said weights Wj over the k ignored sentences, of the ratio between a normalized position P(c7) of the ignored sentence in the text and its normalized position P ignored^) in the list of ignored sentences.
[0038] To specifically quantify the tendency of a candidate language model to favor information located at the beginning of the source text, the process may include calculating a dedicated score, called the primacy bias score and denoted B t This score aims to measure the intensity of this preference for initial positions. The calculation of B L This is based on an analysis of the relative position of sentences previously identified as "ignored" by the summary. More specifically, B L is obtained by calculating an average, weighted by the weights wj associated with each ignored sentence s7 (where Wj typically reflects the length of s7), of a specific ratio calculated for each of these k ignored sentences.
[0039] This ratio compares two normalized position measures for each ignored sentence s7. The first measure, P(c7), represents the position of sentence s7 in the complete source text (containing n sentences), but normalized to give more weight to positions near the beginning. A logarithmic function can be used for this normalization, ensuring that the relative difference between early positions is more pronounced than between later positions. The second measure, P(c7), represents the position of the same sentence s7, but this time within the ordered list of k ignored sentences, also normalized similarly to emphasize initial positions in this specific list.
[0040] The ratio — — — therefore compare where an ignored sentence is located in the overall context (seen from the beginning) compared to where it is located among the other ignored sentences (always viewed from the beginning of this list of ignored sentences). If, on a weighted average, this ratio is significantly higher than a reference value (often close to 1), this indicates that the ignored sentences tend to appear proportionally later in the text than they appear in the sequence of ignored sentences. In other words, for the ignored sentences to be "pushed" towards the end of the text (relative to their order of appearance among the ignored sentences), the model must have strongly favored the very first sentences of the source text, thus exhibiting a pronounced primacy bias. A score S ; A high value is therefore directly interpreted as an indicator of a strong primacy bias.
[0041] The calculation of this score B tThis specific approach offers several technical advantages. It provides a targeted quantitative and directional measure, specifically isolating and measuring the "primacy" component of positional bias, thus enabling finer analysis than a simple global amplitude measurement would. The use of normalized positions (e.g., via a logarithmic transformation) allows for a fairer comparison between positions and ensures greater sensitivity to variations at the beginning of the text. Furthermore, the integration of the weighted average by the Wj weights makes the score more robust, by giving more weight to omissions of long and potentially more significant sentences, while mitigating the impact of minor omissions. This score is B t , calculated for each candidate model, thus constitutes valuable information for comparing their specific tendency to favor the beginning of the text.
[0042] The primacy bias score B t can be calculated using the formula: OR And n being the total number of sentences in the text and posj being the position index, starting at 0, of the ignored sentence s7 in the text.
[0043] The calculation of the positional bias score may include the calculation of a recency bias score B r representing the tendency to favor the end of the text, said score B r being calculated as a weighted average (by said weights wj) over the k ignored sentences, of the ratio between an inverted normalized position (Prev cj) of the ignored sentence in the text and its inverted normalized position (Pgn) Or e(P) in the list of ignored sentences.
[0044] In a manner symmetrical to primacy bias, the process may also include calculating a specific score to quantify the tendency of a candidate language model to favor information located towards the end of the text. This score, called the recency bias score and denoted B r This measures the intensity of this preference for final positions. Its calculation also relies on analyzing the relative positions of sentences previously identified as "ignored" by the generated summary. Similar to the calculation of B, the B score r is obtained by calculating an average, weighted by the weights Wj (typically reflecting the length of each ignored sentence s7), of a specific ratio established for each of the k ignored sentences.
[0045] However, to specifically measure the tendency to favor the end, this ratio compares inverse position measures. The first measure, P rev c represents the position of the sentence The first measure, `sj`, is ignored in the complete source text (containing n sentences), but this time normalized to give more weight to positions near the end of the text. This is typically achieved by counting positions from the end and applying a normalization function (e.g., logarithmic) to this reversed order. The second measure, `P^g^ed`, represents the position of the same sentence `Sj` within the list of k ignored sentences, but also considering this list in reverse order, and applying a similar normalization that emphasizes the last positions of this reversed list. P rev(c,-
[0046] The ratio rev v "Therefore, compare where an ignored sentence is located in the overall context (see ignored P from the end) compared to where it is located among the other ignored sentences (always viewed from the end of this list of ignored sentences). If, on a weighted average, this ratio is significantly higher than a reference value (often close to 1), it means that ignored sentences tend to appear proportionally earlier in the text (viewed from the end) than they appear in the reversed sequence of ignored sentences. In other words, for ignored sentences to be "pushed" towards the beginning of the text (relative to their order of appearance in the reversed list of ignored sentences), the model must have strongly favored the very last sentences of the text, thus exhibiting a pronounced recency bias. A score of B r Therefore, a high recency bias is directly interpreted as an indicator of a strong recency bias.
[0047] The calculation of this score B rspecific offers similar technical advantages to those of the B score t It provides a focused quantitative and directional measure, specifically isolating and measuring the recency component of positional bias, thus enabling a more nuanced analysis of model behavior. The use of inverted normalized positions allows for fair comparison and increased sensitivity to variations at the end of the text. The integration of the weighted mean by the Wj weights ensures the robustness of the B score. r by giving more weight to omissions of long sentences and mitigating the impact of minor omissions. This score is B r , calculated for each candidate model, thus provides specific information to compare their tendency to favour the end of the text when generating the summary.
[0048] Recency bias score B r can be calculated using the formula: Or And
[0049] Said measurement of the amplitude of bias B magnitude can be calculated on the basis of the absolute difference between said primacy bias score B l and said recency bias score B r .
[0050] To obtain an overall measure of the intensity of the positional bias, regardless of its preferred direction (beginning or end), the process can calculate a bias amplitude, denoted ^magnitude, representing the net difference between the tendency to favor the beginning and the tendency to favor the end of the text.
[0051] Calculating this amplitude B magnitude provides an easily interpretable metric that quantifies the overall strength of positional bias. A high value of B magnitude indicates a strong asymmetry in the treatment of positions (i.e., B l is much larger than B r(or the opposite), indicating a highly biased model. Conversely, a low value of B magnitude (close to zero) means that the primacy and recency trends are of similar intensity, suggesting a more balanced processing of information throughout the text. This amplitude measure is therefore particularly useful for directly comparing the intensity of bias between different candidate models and for selecting the one that exhibits the least biased overall behavior, a criterion often crucial for obtaining a representative summary. It complements the directional information obtained by simply comparing B l and B r indicating how pronounced the bias is.
[0052] The magnitude of the bias B magnitude can be calculated using the formula: ^magnitude \Bi - B r |.log(e + T) where T is the total number of tokens in said text.
[0053] To further refine the measurement of the magnitude of positional bias and allow for fairer comparisons between texts of different lengths, the method can thus calculate B magnitude taking into account the total length of the source text. Specifically, the amplitude can be calculated according to the aforementioned formula, where B l and B r are the primacy and recency bias scores respectively, T represents the total number of tokens in the analyzed source text, 'e' is the base of the natural logarithm (Euler's constant), and 'log' denotes the natural logarithm.
[0054] In this context, the term "tokens" refers to the fundamental units of text processed by language models. Depending on the tokenization method used by the model or system, a token can correspond to an entire word, a subword, a punctuation mark, or another symbol. Segmenting text into a sequence of tokens is a standard preprocessing step in natural language processing, and the total number of tokens, T, is a commonly used measure to quantify the effective length of a text as perceived by a language model.
[0055] The introduction of the multiplicative factor log(e + T) in the calculation of B magnitude implicitly recognizes that maintaining positional neutrality (i.e., obtaining a small difference |B l - B r|) is potentially more difficult for long texts than for short texts. By multiplying the raw difference |B l - B r By using a term that increases with the text length T, the formula gives more weight to bias deviations observed in long texts. However, the use of the logarithmic function ensures non-linear scaling: the impact of length increases, but in a mitigated way for extremely long texts. This prevents the amplitude measure from being excessively dominated by text length and allows for more effective normalization of B scores. magnitude across corpora containing texts of widely varying lengths. The addition of the constant 'e' ensures that the argument of the logarithm is always greater than 1, thus guaranteeing the positivity of the multiplicative factor for numerical stability. Ultimately, this formula provides a measure of the magnitude of the bias B magnitudewhich is not only sensitive to the gap between primacy and recency trends, but also appropriately normalized with respect to the difficulty induced by the length of the source text, making the scores more robust and more directly comparable across different contexts.
[0056] This measure of bias direction can be determined by comparing said primacy bias score B l and said recency bias score B r , for example by assigning a value indicating primacy bias if B l > B r and a value indicating a recency bias if B l < B r .
[0057] In addition to measuring the intensity of the bias (its magnitude), the process can also determine the preferred direction of this positional bias. This step aims to identify whether the candidate language model has a dominant tendency to favor the beginning or the end of the source text when generating the summary. Determining this direction is done simply by comparing the values of the two specific bias scores calculated previously: the primacy bias score B l and the recency bias score B r .
[0058] If the value of B t is greater than that of B r (S ; > S r ), this indicates that the model's tendency to overweight initial information is stronger than its tendency to overweight final information. In this case, the direction of the bias is determined to be a "lead bias". Conversely, if the value of Br is greater than that of B l (B l < B r This means that the tendency to favor the end of the text dominates, and the direction of the bias is determined to be a "recency bias." The result of this comparison can be represented by a symbolic value (e.g., +1 for primacy, -1 for recency) or a categorical label. In the case where B l and B r are very close (their difference being less than a small tolerance threshold), the direction can be considered "neutral" or "balanced", indicating the absence of a marked preference for one end or the other.
[0059] The direction of bias provides qualitative information that complements the quantitative measurement of magnitude. Knowing that a model has a high bias magnitude is useful, but knowing whether this bias favors the beginning or the end of the text allows for a much more nuanced understanding of its behavior. This directional information is easy to interpret and can be useful for diagnosing the specific type of limitation in the model. It allows models or abstracts to be categorized according to their dominant bias type (primacy, recency, balanced). Furthermore, this knowledge of the direction, obtained through a simple comparison of B l and B r and therefore very inexpensive in terms of computation, offers a more complete view which, combined with the amplitude, allows an exhaustive evaluation of the positional bias of the candidate model.
[0060] The language model(s) exhibiting the positional bias amplitude score B magnitudeThe weakest among said plurality may be selected or recommended.
[0061] The language model(s) exhibiting a positional bias amplitude score B magnitude below a predefined threshold can be selected or recommended.
[0062] If several language models satisfy the selection or recommendation criterion based on positional bias, at least one additional criterion, such as computational cost, inference speed, energy consumption, or model size, can be used to make a final selection or recommendation among these models.
[0063] As part of the selection or recommendation process, it is possible that the application of the criterion based on the positional bias score (typically, minimizing the amplitude Bmagnitude) may be used. ou I eAdherence to a threshold on this amplitude does not lead to the identification of a single optimal language model. Several candidate models could exhibit very similar and satisfactory levels of positional bias, or even identical levels depending on the precision of the calculations. In such cases, where the positional bias criterion alone does not allow for distinguishing a better candidate among a subset of equivalent models, the process can advantageously incorporate the use of at least one additional criterion to refine the selection or final recommendation within this subset.
[0064] These additional criteria are not related to positional bias but rather to other relevant characteristics of the language model or its use. These might include, for example, considerations related to resource efficiency, such as the computational cost associated with using the model (e.g., the cost of accessing an API or the necessary server resources), inference speed (the time required to generate the summary), the model's energy consumption during execution, or the model's size (e.g., the number of parameters or the memory space required). Other criteria, such as expected latency, known performance on specific ancillary tasks, measures related to social biases, measures of the intrinsic quality of the summary, or even security or data privacy considerations associated with each model, could also be considered.
[0065] Integrating these additional criteria as a tie-breaker or refinement mechanism allows for a robust and pragmatic approach to managing situations where several models offer similar performance with respect to the primary criterion of low positional bias. Rather than randomly or arbitrarily choosing among models with equivalent bias, this process optimizes the selection based on other objectives important to the deployment or the end user.
[0066] For example, by choosing the least biased model that is also the most "frugal" (lowest cost, lowest energy consumption) or the fastest, the process allows for more efficient use of computing resources and can improve the user experience (e.g., by reducing the waiting time to obtain the summary). This transforms the selection process into a multi-objective optimization, where the primary objective of minimizing positional bias is complemented by secondary considerations of efficiency or operational performance. This flexibility allows the final selection to be adapted to the specific constraints and priorities of the application context (e.g., prioritizing speed for a real-time application, cost for large-scale batch processing), while ensuring that the chosen model is among those with the lowest initially identified positional bias.
[0067] The process may include a step of receiving said text, for example a transcript of a conversation.
[0068] The process may further include a step of generating a final summary of said text using at least one selected or recommended language model.
[0069] The process may further include a step of presenting to a user, via a graphical interface, the summary made by said at least one selected or recommended language model and / or the positional bias scores calculated for at least some of the plurality of candidate language models.
[0070] The present invention also relates to a system for selecting or recommending a language model for generating a text summary, comprising a processor and a memory associated with said processor, the memory storing instructions which, when executed by the processor, configure the system to implement the aforementioned method.
[0071] The present invention also relates to a computer program product comprising computer-readable instructions which, when executed by a processor, cause the processor to execute the aforementioned process.
[0072] The present invention also relates to a computer-readable and non-transient storage medium storing the computer program product of the aforementioned type. Brief description of the drawings
[0073] Other features, details, and advantages will become apparent upon reading the detailed description below and analyzing the attached drawings, on which: Figure 1 schematically illustrates the steps of the process according to one embodiment of the invention. Figure 2 schematically illustrates a system according to one embodiment of the invention. Description of the implementation methods
[0074] The drawings and description can not only serve to better explain the proposed technique, but also contribute to its definition, where applicable. The detailed description below, with reference to the attached figures, illustrates non-limiting embodiments of the invention.
[0075] Figure 1 schematically illustrates one implementation of a computer-implemented process for selecting or recommending a language model for generating a text summary, the process comprising: - access (LLM_ACC) a plurality of candidate language models, each capable of generating a summary of said text; - for each candidate language model among said plurality, calculate (BIAS_CALC) a positional bias score quantifying a tendency of said candidate language model to favour information from specific positions, such as the beginning or the end, in said text when generating a summary; said positional bias score comprising at least a measure of the direction of the bias and a measure of the magnitude of the bias; - compare (COMP) said positional bias scores calculated for the plurality of candidate language models; and - select (SELEC) or recommend at least one language model from said plurality on the basis of said comparison.
[0076] More specifically, the process applies to various types of text, including, but not limited to, transcripts of multi-turn conversations (formal meetings, informal dialogues, customer service interactions), articles, reports, etc.
[0077] The method begins by accessing a source text to be summarized and a plurality of candidate language models capable of generating a summary of that text. The key step of the invention lies in calculating, for each candidate LLM, a positional bias score specific to that text. This score quantifies the LLM's tendency to favor or disregard information based on its position in the source text. The final score comprises two main components: a measure of the bias direction (indicating whether the model favors the beginning or the end) and a measure of the bias magnitude (indicating the intensity of this preference). Detailed Calculation of the Positional Bias Score
[0078] The calculation of this score relies on a methodology based on the semantic analysis of sentences in the source text that are poorly represented or "ignored" in the summary generated by the candidate's LLM. This approach is entirely independent of any reference summary. Identification of Ignored Phrases
[0079] Let the source text D be composed of n sentences C1, C2,..., C n Let S = {s1,s2,...,s m} the summary generated by an LLM candidate for text D.
[0080] For each source sentence C it We calculate its semantic similarity with each sentence S7 of the summary. This is typically done using sentence embeddings, for example those from models like Sentence-BERT, to represent each sentence as a vector in a semantic space, and then calculating a similarity measure (for example, cosine similarity) between the vectors of and 5). Let s it j = sim
[0081] For each source sentence, its maximum semantic similarity score with any sentence in the summary is determined: s-"“ x = max s,,j. This score s™ ax indicates how much the The sentence is well "covered" by the summary.
[0082] To ensure robustness and comparability between different texts and summaries, the scores s max p euven t undergoes a quantile transformation, denoted Q(.). This transformation maps the scores onto a reference distribution, typically a standard normal distribution: s t = Q(s™ ax ) this helps to stabilize the variance and reduce the impact of extreme values.
[0083] A dynamic threshold is then applied to the normalized scores. hat to identify significantly underrepresented sentences. This threshold is calculated from the mean |i and the standard deviation a of the scores s t for the text and summary considered: threshold = |i - a.
[0084] The sentences whose normalized score tsentences below this threshold are considered ignored. Let k be the number of these ignored sentences, denoted Sj(j = 0,...,k - 1). Weighting of Ignored Phrases
[0085] Each ignored phrase Sj is assigned a weight Wj that reflects its relative importance, typically based on its length lj (number of words or tokens). An efficient method uses the Softmax function:
[0087] This weighting gives more importance to the omission of long sentences, which are potentially richer in information. Calculation of the Leading Bias Score (B) t )
[0088] This score measures the tendency to favor the beginning of the text. It is calculated as the weighted average (by the w7 weights) of the ratios between the normalized position of each ignored sentence in the text and its normalized position in the list of ignored sentences.
[0089] The normalized position in the text (n sentences, posj index starting at 0) is:
[0090] The normalized position in the list of ignored phrases (fc phrases, index j starting at 0) is:
[0091] The score is then:
[0092] A high value of B l indicates a strong primacy bias. Calculating the Recency Bias Score Bias
[0093] This score measures the tendency to favor the end of the text. It is calculated symmetrically to Bi, using inverted normalized positions.
[0094] The normalized reversed position in the text is:
[0095] The inverted normalized position in the list of ignored items is: prev _ (_ln_(_ _fc - j _ _ + 1) _) ignore ^ n (fc + 1))
[0096] Score B r is then: fc-1 zp
[0097] A high value of B r indicates a strong recency bias. Calculation of ['Amplitude and Direction of the Bias]
[0098] The magnitude of the bias, B magnitude , measures the overall intensity of the bias, regardless of its direction. It is calculated by taking the absolute difference between B l and B r , and can be normalized by the total length T of the source text (in number of tokens, basic units such as words or subwords) for better comparability: ^magnitude \Bi - B r |.log(e + T)
[0099] A low value of B magnitude is desirable, indicating low bias and a balanced summary.
[0100] The direction of the bias indicates which tendency dominates. It is determined by a simple comparison of B t and B r :
[0101] Direction = +1 if B l > B r (primacy bias), or -1 if B l < B r (recency bias).
[0102] A direction close to 0 (if B l approx B r ) indicates a balanced or neutral bias. Comparison and Selection / Recommendation
[0103] Once the positional bias scores (B l , B r , B magnitude (Direction) calculated for each candidate LLM from the available plurality, the process performs a comparison of these scores. The main objective is generally to identify the model(s) exhibiting the lowest Bmagnitude bias amplitude, indicating the most balanced behavior for the text under consideration.
[0104] The comparison leads to a selection or recommendation stage. The system can automatically select the model with the B magnitude minimal (or below a predefined threshold) to then generate the final summary with this model. Alternatively, the system can recommend this model (or a ranking of models based on B magnitude and / or Management) to a user via an interface, allowing them to make an informed choice. If several models have similar and satisfactory bias scores, additional criteria (computational cost, speed, energy efficiency, etc.) can be used to differentiate and finalize the selection or recommendation. Experimental Validation
[0105] The effectiveness of this positional bias scoring methodology was experimentally validated on a dedicated benchmark (PoSum-Bench) comprising various corpora of English and French conversations of varying types and lengths (ranging from short dialogues of approximately 100 words to long meetings of over 10,000 words). Several candidate LLM families (e.g., Mistral-7B-lnstruct-v0.3, Gemma-3-1 B-it, Qwen2.5-14B-lnstruct, Llama-3.2-3B-lnstruct) were evaluated.
[0106] The experimental results demonstrated several key points, which are outlined below.
[0107] The methodology effectively quantifies positional bias for different models and types of texts.
[0108] It has been observed that many models exhibit significant positional bias, particularly for short texts (bias magnitudes typically between 1.3 and 1.6), while the bias tends to weaken and become more balanced for longer texts (magnitudes often below 0.1).
[0109] Differences in behavior were noted between languages (for example, a tendency towards a more pronounced recency bias in French for some models) and between model families.
[0110] Controlled experiments, where artificial biases (by extracting only the beginning, end or middle of a text) were introduced, confirmed that the B_l, B_r and Bjnagnitude metrics correctly and with high accuracy identify the direction and intensity of the induced bias, thus validating the relevance and reliability of the proposed calculation method.
[0111] In conclusion, the described process allows, through a detailed and validated calculation of the positional bias score based on the semantic analysis of ignored sentences, for the objective comparison of candidate language models and the selection or recommendation of the one best suited to generate a balanced and representative summary of the source text, thus significantly improving the reliability and usefulness of automatic summaries produced by LLMs.
[0112] A practical example of the application of this method involves a user interacting with a conversation analysis platform to obtain a summary of a long recorded meeting. Unable to take exhaustive notes, the user wants to ensure that the automatically generated summary accurately reflects all the discussions. By activating a feature based on the method of the invention, for example, via an option to "select the LLM with minimal positional bias," the system evaluates several available candidate language models. For each model, it calculates the positional bias score (direction and magnitude) specifically for the transcription of that meeting. Based on a comparison of these scores, the system can then either recommend to the user the model identified as the least biased (i.e., having the lowest bias magnitude) or automatically select that model to generate the final summary.
[0113] In a complementary application, the process can be integrated into a system designed to facilitate detailed exploration of the source text from a generated summary (potentially generated by a template selected or recommended according to the main process). A user, such as an analyst or employee in training reviewing the summary of a customer interaction or meeting, can identify a semantic segment of interest within the summary (for example, the mention of a specific technical problem and its resolution). The user can then initiate an action, such as selecting the segment in a user interface. The system then uses a semantic similarity measure to match the selected segment of the summary with the most relevant sentences or portions of sentences in the original source text (for example, the complete transcript of the interaction).The system can then present the user with the original source text, highlighting (for example, by underlining or focusing) the passages identified as semantically corresponding to the key information in the summary that piqued the user's interest. While distinct from bias calculation, this functionality benefits from the generation of a reliable and representative summary, potentially obtained through the bias-based selection process, as such a summary is more likely to point to the truly salient information in the source text.
[0114] Figure 2 illustrates a system (SYSM) for selecting or recommending a language model for generating a text summary, comprising a processor (PROC) and a memory (MEM) associated with said processor, the memory storing instructions which, when executed by the processor, configure the system to implement the aforementioned process.
Claims
DEMANDS
1. A computer-implemented method for selecting or recommending a language model for generating a text summary, the method comprising: - for two candidate language models capable of generating a summary of said text, calculate (BIAS_CALC) respectively two positional bias scores relative to the generated summary, said two positional bias scores comprising at least one measure of the direction of the bias and one measure of the magnitude of the bias; - compare (COMP) said two positional bias scores; and - select (SELEC) or recommend at least one language model from among the two language models based on said comparison.
2. A method according to the preceding claim, wherein the calculation of the positional bias score for a candidate language model includes a prior identification of ignored sentences of said text by a summary generated by said candidate language model.
3. A method according to the preceding claim, wherein said identification of ignored phrases from the text comprises: - calculate, for each sentence C i (i=1 n) of said text, a maximum semantic similarity score s t max with at least one sentence from a summary generated by said candidate language model; - apply a dynamic threshold to said scores s1 max to identify a set of k phrases Sj(j = 0,...,fc - 1) as being the ignored sentences, said ignored sentences being those whose score s i max is below that threshold.
4. A method according to the preceding claim, wherein said dynamic threshold is calculated on the basis of the mean / z and the standard deviation <r desdits scores s i max .
5. A method according to any one of claims 2 to 4, wherein the calculation of the positional bias score uses a weight Wj associated with each ignored sentence s7, said weight Wj being a function of the length l t of the said ignored sentence s7.
6. A method according to the preceding claim, wherein the calculation of the positional bias score includes the calculation of a primacy bias score S ; representing the tendency to favor the beginning of the text, said score S ; being calculated as a weighted average, using said weights Wj over the k ignored sentences, of the ratio between a normalized position P(c7) of the ignored sentence in the text and its normalized position Pt gnored ^ in the list of ignored phrases.
7. A method according to the preceding claim, wherein the primacy bias score is calculated according to the formula: Or And n being the total number of sentences in the text and posj being the position index, starting at 0, of the ignored sentence s7 in the text.
8. A method according to any one of claims 5 to 7, wherein the calculation of the positional bias score includes the calculation of a recency bias score B r representing the tendency to favor the end of the text, said score B r being calculated as a weighted average (by said weights Wj) over the k ignored sentences, of the ratio between a normalized reversed position (Prev cj') of the ignored sentence in the text and its normalized reversed position P ignores ( in the list of ignored phrases.
9. A method according to the preceding claim, wherein the recency bias score B r is calculated according to the formula: OR And prev rj > (ln(fc-j + l)) t ignore UJ (ln(fc+l)) '
10. A method according to any one of the preceding claims, wherein said measurement of the amplitude of the bias B magnitude is calculated on the basis of the absolute difference between said primacy bias score B t and said recency bias score B r .
11. A method according to the preceding claim, wherein the amplitude of the bias Bmagnitude is calculated according to the formula: ^magnitude \Bi - B r |.log(e + 7) where T is the total number of tokens in said text.
12. A method according to any one of the preceding claims, wherein said measure of bias direction is determined by comparing said primacy bias score B t and said recency bias score B r .
13. System for selecting or recommending a language model for generating a text summary, comprising a processor and associated memory for said processor, the memory storing instructions which, when executed by the processor, configure the system to implement the method according to any one of the preceding claims.
14. Product computer program comprising computer-readable instructions which, when executed by a processor, cause the processor to perform the process according to any one of claims 1 to 12.
15. Computer-readable and non-transient storage medium storing the computer program product according to the preceding claim.
